Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

# Supervising Sparse Attention Routing with Causal Evidence Sets
Source: [https://arxiv.org/html/2607.21692](https://arxiv.org/html/2607.21692)
## Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

###### Abstract

Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input\. These selectors are often trained by distilling the attention patterns of a dense teacher, assuming that attention reveals which context the teacher actually uses\. We test that assumption on retrieval tasks where the evidence for each answer is known exactly\. By masking parts of the context and measuring whether the answer changes, we find that attention and causal dependence often disagree, and distilled selectors inherit the mismatch\. Teachers attend to outdated facts they have learned to ignore, and their attention can vary across training runs even when they rely on the same evidence\. In a two\-step reference task, attention at the answer skips the intermediate step because it was resolved earlier in the forward pass: a selector trained on attention achieves 41% accuracy, while the same selector trained on causal evidence reaches 99% and matches the teacher\. These evidence sets require no annotation: recovered from a frozen teacher by masking alone, they train selectors to the same accuracy\. We find the same conflict in pretrained models: Qwen2\.5\-3B gives more attention to an outdated fact than the current one on 58% of conflicting\-fact examples despite answering correctly, while Gemma\-2\-9B rises from 56% to 99% accuracy when restricted to the two relevant sentences\. Attention shows where a model looks, not necessarily what its answer depends on; across the regimes we tested, that dependence matched or outperformed attention as a training target\.

## 1Introduction

A transformer reads its context through attention: each new token scores every earlier token and takes a weighted mixture of them\. The cost grows with the square of the context length, and over a long context most of it is spent on content that never affects the output\. Sparse attention cuts the cost by letting each query read only a small subset of the context\. The subset has to be chosen\. The strongest current selectors are trained by distillation\(Ahmad & Yun,[2026](https://arxiv.org/html/2607.21692#bib.bib2)\): a small network, the*selector*, learns to reproduce the attention of a full dense model, the*teacher*, so that at inference the model reads only what the teacher would have attended to\. The design assumes that a model’s attention identifies the context it needs\.

This paper tests that assumption\. The test needs a setting where the context a model needs is known independently of its attention, so we build one\. Our tasks are synthetic key\-value retrieval problems in which the context is a sequence of fixed width*blocks*, each holding one*record*, followed by a query\. A record is a single stored fact: a key and the value filed under it, written as one short line of tokens\. By construction we know which records determine the answer\. A set of blocks is*sufficient*for an example if restricting the model’s attention to those blocks and the query leaves its answer unchanged\. Sufficiency is verified by intervention, masking attention into a block at every layer with token positions unchanged\. The supervision target is the*causal evidence set*\(Section[2\.3](https://arxiv.org/html/2607.21692#S2.SS3)\): every block that alone preserves the answer when any such block exists, and otherwise every block whose masking flips the answer\. That label is not always a minimal sufficient set\. On these tasks we train dense teachers from several random seeds, freeze them, and compare two supervision targets for a selector, which we call a*router*: the teacher’s attention weights, and the causal evidence sets\. Deploying a router masks every block it did not select, so the model answers from the kept blocks alone\. The resulting accuracy is its*routed accuracy*\.

The clearest result comes from a task where answering requires following references\. A record may hold a pointer to another record in place of a value, so the queried record starts a chain of two pointers ending in a value, and the evidence is that three\-block chain\. We call this task*multi\-hop*retrieval, in contrast to the*single\-record*tasks, where one record determines the answer\. A dense teacher solves the task at 99%\. But the pointer steps were resolved in earlier layers at earlier positions, and by the time the answer is produced, attention covers the full chain on fewer than 10% of examples\. Routers inherit the omission\. With identical architecture, data, and budget, a router supervised on the annotated chains routes the task at 99%, while the same router distilled from the teacher’s attention reaches 41%\. Per\-element recall explains the gap\. It is 1\.00 on the first pointer, 0\.99 on the end value, and 0\.00 on the middle block\. Attention shows where the answer is read, not where it was computed, and imitating it prunes the middle of the computation away\. Nothing visible from outside says when this happens\. The same distillation swings between 0\.41 and 0\.98 across training runs, and a task variant that denies shortcuts makes it work again\. Attention is not so much a wrong target as one that cannot be vetted without interventions, and the interventions are the better label\.

The single\-record tasks give the same conclusion in a milder form\. On a task where a key is written several times and only the last write matters, every teacher, at every seed, spends attention on the obsolete writes on every example\. Attention is consistent across seeds and consistently includes evidence that does not matter\. It is also diffuse\. Covering 90% of the attention mass at the answer position takes about 14 of 32 blocks, while the causal evidence occupies about 1\.5\. Causal evidence sets, recovered by the same interventions, agree across seeds far more than attention does \(Jaccard 0\.79 to 0\.90, against 0\.38 to 0\.46 for size\-matched attention sets\)\. Planting redundant copies of the answering record breaks the single\-block interventions, because masking one copy changes nothing and so no single block looks necessary\. The family of interchangeable sufficient sets survives this, and it agrees across seeds better as redundancy grows, from 0\.81 with one copy to 0\.90 with four\.

These measurements matter for routing because supervision is a choice\. We train routers with identical architecture and inputs, one imitating block\-level attention distributions \(the imitation router\), one trained on causal evidence sets \(the causal router\), each deployed as a hard mask on the frozen teachers at a 10% block budget\. In distribution the supervisions tie and both track the dense teacher; at four times the training length the causal router leads by 9 points of accuracy on unique retrieval, on every seed, surviving a learning\-rate sweep and tripled training of the baseline\. Where the teacher’s attention and the annotated evidence largely agree, as on the last\-write task above, the supervisions tie, which is the pattern the measurements predict\.

The paper makes four contributions\.

- •A testbed with known evidence where interventions show attention and dependence come apart: attention covers obsolete writes, misses multi\-hop chains at the answer, and varies across seeds whose evidence stays fixed\.
- •Matched routing experiments where the label, not the loss, carries the effect\. Causal supervision reaches 0\.99 on multi\-hop against 0\.41 for attention distillation, which never selects the chain’s middle block\. The router must also be expressive enough: mean pooling fails on chains under every supervision\.
- •An annotation\-free estimator of the causal evidence set\. Routers trained on its labels match annotation\-trained ones, while every single\-signal alternative fails where the measurements predict\.
- •The same conflict and routing ordering on frozen pretrained models across three families, including a conflicting\-fact repair, exact label recovery, and the ordering preserved on counterfactual SQuAD\.

Prior work either imitates attention or questions it\. None tests it as a routing target against known evidence\. Learned selectors imitate teacher attention\(Ahmad & Yun,[2026](https://arxiv.org/html/2607.21692#bib.bib2); Roy et al\.,[2021](https://arxiv.org/html/2607.21692#bib.bib18)\)\. Work on end\-to\-end sparse training shows that the learned gates barely beat random ones, because the rest of the model co\-adapts to the mask\(Aquino\-Michaels,[2026](https://arxiv.org/html/2607.21692#bib.bib3)\); this is why we train our teachers dense and then freeze them\. Circuit\-finding methods recover causal subgraphs of a trained model after the fact\(Bhaskar et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib7)\), but they do not produce a per\-input predictor that runs before attention is computed\. Other related work is discussed in Section[9](https://arxiv.org/html/2607.21692#S9)\.

## 2Setup

### 2\.1Tasks

Every example is a sequence ofnnblocks ofwwtokens followed by a short query, withn=32n\{=\}32andw=8w\{=\}8unless stated\. Each block holds one record,WRITEffmmvv, padded with filler: a two\-token key \(familyff, membermm\) and a one\-token valuevv\. Two\-token keys give graded distractor similarity, since a distractor can share the family token without sharing the key\. The query isQUERYffmmvv, and the model is scored on predictingvvat its position\. Table[1](https://arxiv.org/html/2607.21692#S2.T1)lists the tasks\.

Table 1:Tasks\. The sufficient sets are known by construction and verified by intervention \(Section[2\.2](https://arxiv.org/html/2607.21692#S2.SS2)\)\.In the latest\-write task, the earlier writes carry different values, so answering with one is a scored error\. In duplicate evidence, each copy alone is sufficient, so the target is a collection of interchangeable sets; we call the set of minimal sufficient sets of an example its*family*\. In multi\-hop, a pointer record holds a second key in place of a value\. Aggregation asks for the number of records in a named family, so every marked block is necessary and only the full marked set is sufficient\. It is the intended boundary case\.

Two protocol details matter for reproduction: the value is predicted directly at the query key, since an answer marker between key and value prevents the retrieval circuit from forming at this scale, and training sequences carry eight queries each; evaluation is always single\-query, an exact prefix of the training format \(Appendix[B](https://arxiv.org/html/2607.21692#A2);Arora et al\.,[2023](https://arxiv.org/html/2607.21692#bib.bib4)\)\.

### 2\.2Teachers, interventions, and sanity checks

Teachers are decoder\-only transformers, 6 layers, 8 heads, model width 384, 10\.7M parameters, rotary position embeddings, trained with AdamW to high accuracy on a task mixture \(0\.90 to 0\.99 per task across seeds; per\-teacher finals in Appendix[G](https://arxiv.org/html/2607.21692#A7)\)\. All teachers are then frozen\. No sparsity is imposed during training, so the supervision labels cannot be absorbed into the weights\(Aquino\-Michaels,[2026](https://arxiv.org/html/2607.21692#bib.bib3)\)\.

Masking a block adds−104\-10^\{4\}to the attention logits of its positions at every layer for every query outside the block, the same fill value the causal mask uses; token positions do not move, so the intervention removes information, not positional structure\. Restricting to a set means masking its complement\. Both directions are checked: for solved examples, keeping only the annotated sufficient set preserves the answer, and masking the annotated block flips it on 98% of unique\-retrieval examples while masking a random other block flips it on 0%\. Masking a block and physically overwriting its record with filler produce the same answer on 100% of 200 checked retrieval examples\. On unique retrieval, the top attended block at the answer position is the annotated block on 99% of solved examples, so at least where one block matters and no distractor shares the key, attention and sufficiency coincide\. The disagreements below are not artifacts of a broken readout\.

### 2\.3Labels

Three objects need separate names, because our results show they come apart\.*Task evidence*is the annotated records that determine the answer by construction\. A block is*necessary*if masking it changes the full\-context answer\. A set is*sufficient*if restriction to that set preserves the answer, as defined above\. On solved single\-record examples these three coincide\. Redundancy breaks necessity \(Section[4](https://arxiv.org/html/2607.21692#S4)\), and restriction\-induced fallbacks can shrink a sufficient set below the task evidence \(Appendix[D](https://arxiv.org/html/2607.21692#A4)\), so each measurement states which object it uses\. For each example and each frozen teacher we compute three kinds of block label\. Attention labels take thekkblocks with the largest attention mass at the answer position, pooled over layers and max or mean over heads, plus an adaptive variant that takes the smallest set covering 90% of that mass over context blocks\. Ablation ranking takes thekkblocks whose individual masking most reduces the answer log probability\. We define the*causal evidence set*in two cases\. If some single block preserves the answer on its own, the set is every block that does so, recovering all copies under redundancy\. If no single block suffices, the set is every block whose masking flips the answer, recovering the necessary support under chains\. The name is neutral because the members are sufficient alternatives in one regime and necessary blocks in the other\. The set has no fixed size\. Interventions decide how many blocks it contains, example by example\. Comparisons against attention therefore use the adaptive attention variant, which also chooses its own size\.

## 3Attention includes evidence that does not matter

On the latest\-write task, where a key is written several times and only the final write is correct, the adaptive attention set includes at least one obsolete write on 100% of examples, for every one of five seeds\. The behavior is systematic, not noise\. Attention retrieves everything that matches the queried key and leaves the arbitration to later computation\. At a fixedk=3k\{=\}3budget, attention labels here agree across seeds about as well as causal\-evidence labels do \(Jaccard 0\.76\)\. Stability alone does not certify a target whose stable sets contain evidence that does not matter\.

Attention is also diffuse\. Covering 90% of the answer\-position mass takes 13 to 15 of 32 blocks depending on seed; the causal evidence set averages 1\.4 to 1\.6 blocks on the single\-record tasks\. A selector distilled from attention is being taught a nine\-times\-larger target than the evidence requires\.

## 4Causal evidence sets are stable

Table[2](https://arxiv.org/html/2607.21692#S4.T2)reports cross\-seed label agreement\. Two regularities stand out\. First, on the single\-record tasks the causal evidence set is about twice as consistent across seeds as size\-matched attention: different training runs solve these tasks with different attention and the same evidence\. Second, redundancy separates the estimators\. Withmmcopies planted, masking any one copy stops changing the answer: the flip rate of individual causal blocks falls from 0\.91 with one copy to 0\.06 with two and 0\.001 with four, and the ablation\-ranked label decays with it \(0\.25 to 0\.11\)\. The family label moves the opposite way, 0\.81 to 0\.90, because all copies are individually sufficient and all are recovered\. When several answers are equally good, which one a model uses is not identifiable, but the set of usable answers is, and it is the stable object\. Recovery is also accurate against construction: the recovered family equals the annotated one on 70 to 82% of examples, and covers a correct sufficient set on more than 99%\.

Table 2:Cross\-seed label agreement \(mean Jaccard over ten seed pairs, 500 examples per task\)\. Adaptive attention is the top\-p=0\.9p\{=\}0\.9set; ablation top\-kkranks blocks by individual masking effect; the family label is defined in Section[2\.3](https://arxiv.org/html/2607.21692#S2.SS3)\. Duplicate\-mmplantsmmidentical copies of the answering record\.
## 5Causal supervision improves routing

A router scores context blocks given the token sequence, with the supervised position removed, since the answer is never available at routing time; it never reads teacher activations, so one trained router runs unchanged on any teacher\. We use two architectures\. The*pooling router*embeds tokens with its own table, mean\-pools each block and the query, and scores each block independently from its summary, the query summary, and relative position\. The*chain router*replaces mean pooling with a projection per within\-block offset, so a pointer’s source and destination keys stay distinguishable, and adds two transformer layers over the summaries, so a block’s score may depend on other blocks\. At evaluation the router’s topk=⌈0\.1​n⌉k\{=\}\\lceil 0\.1n\\rceilblocks stay visible and everything else is masked, using the operator of Section[2\.2](https://arxiv.org/html/2607.21692#S2.SS2); the query is always visible\. Both routers are small: 23K and 111K parameters, under 0\.3 ms per 32\-sequence batch, flat from 32 to 128 blocks\.

All routers train on the same example stream from the same label teacher\. The imitation router distills the teacher’s attention at block granularity: it minimizes the Kullback–Leibler \(KL\) divergence between the teacher’s attention over blocks at the answer position and the router’s block distribution\(Ahmad & Yun,[2026](https://arxiv.org/html/2607.21692#bib.bib2)\)\. The KL divergence measures how far one probability distribution is from another\. The causal supervision trains on causal evidence sets with a coverage loss\. Per example, binary cross entropy is taken against the indicator of each acceptable set, then combined by a softmin over sets that anneals to the minimum\. Covering any one sufficient alternative is enough, so the router learns to commit to one copy rather than hedge across all of them\. A random\-gate control keepskkrandom blocks\(Aquino\-Michaels,[2026](https://arxiv.org/html/2607.21692#bib.bib3)\)\. Every routed accuracy is over 300 to 500 examples; 95% binomial intervals \(±0\.06\\pm 0\.06midrange,±0\.02\\pm 0\.02near the extremes\) are narrower than any accuracy difference we draw a conclusion from\.

Table[3](https://arxiv.org/html/2607.21692#S5.T3)contains the single\-record results, with pooling routers throughout\. In distribution the two supervisions tie within noise and both track the dense teacher; on unique retrieval both sit slightly above it \(0\.96 and 0\.98 against 0\.93\): masking distractors removes interference, a repair effect that returns at scale in Section[7](https://arxiv.org/html/2607.21692#S7)\. Under length shift they separate on unique retrieval: at four times the training length the causal router leads by 9 points of accuracy on average \(0\.72 against 0\.63\) and on every seed, and the lead survives retraining the imitation router at three learning rates and at three times the steps \(remaining gaps 8 to 13 points\)\. On the latest\-write task the two supervisions tie at every length\. There the teacher’s attention concentrates on the write records, so its top blocks and the annotated set largely agree, and the choice of supervision has little to separate\. The dense teachers themselves fail at 4×\\timeslength \(0\.32 and 0\.42\), and routing repairs much of the failure\. The 2×\\timesrows, where dense still functions, show the unique\-retrieval gap independently of that repair\.

Table 3:Single\-record routed accuracy at a 10% block budget: pooling routers trained against seed 0, means over five teacher seeds, 500 examples per condition\. 2×\\times/4×\\timesscale the block count at evaluation; stale\-4 adds obsolete writes; family distractors share the query’s family token\. Per\-seed values: Appendix[G](https://arxiv.org/html/2607.21692#A7)\.Multi\-hop is where the supervisions come fully apart, and it takes both the right target and a router that can express the selection \(Table[4](https://arxiv.org/html/2607.21692#S5.T4)\)\. Of five teachers trained on the five\-task mixture the multi\-hop circuit converged in four, and every multi\-hop number averages those four\. Routing on labels directly, with no learned router, separates the targets: the annotated chain and the ablation ranking score 0\.98 to 1\.00 on all four seeds, while the teacher’s top attended blocks range from 0\.41 to 0\.98 depending on seed and head pooling, and the best single layer’s attention never exceeds 0\.55\. Whether attention exposes the chain is a property of the particular solution a training run found\. Pooling routers fail under every supervision \(0\.08 to 0\.15\): mean pooling collapses a pointer’s source and destination keys into one summary, and no loss can teach what the encoding cannot represent\. The chain router removes that constraint, and the supervisions separate cleanly\. Trained on annotated chains it routes at 0\.99, keeps the full chain on 99 to 100% of examples at every depth and learning rate in our grid, and transfers unchanged to the other converged teachers at 0\.98 to 0\.99\. Trained on attention with everything else identical it reaches 0\.41, and its per\-element recall says why: 1\.00 on the first pointer, 0\.99 on the end value, 0\.00 on the second\. Answer\-position attention contains the readout locations and not the middle of the computation, so the middle is exactly what distillation deletes\.

Table 4:Multi\-hop routing atk=3k\{=\}3\(300 examples\)\. Oracles route on labels directly, as ranges over the four converged seeds \(attention: both head poolings\); recall lists selection rates for first pointer, second pointer, and end value\.Could something other than the label explain these gaps? Two alternatives are worth ruling out: that the gaps come from the loss rather than the labels, and that attention elsewhere in the model covers the chain even though the answer position does not\. Table[5](https://arxiv.org/html/2607.21692#S5.T5)tests both on pooling routers by crossing label type with loss form and adding an attention label pooled over every position and layer\. On the single\-record shift conditions the gap follows the label, not the loss, and the all\-position variant is the worst label in the table, at or below random\. Attention mass summed over positions is dominated by local structure rather than retrieval\. Attention does not hide the chain somewhere the answer position misses; averaging more of it buries the signal further\.

Table 5:Label type against loss form for pooling routers at the 10% budget, mean over teachers\. No pooling arm learns multi\-hop under any supervision; Section[5](https://arxiv.org/html/2607.21692#S5)locates the constraint in the block encoding\.The dependence on the teacher’s training problem cuts both ways\. We harden multi\-hop so that nothing short of resolving the chain answers the query: chain roles carry no positional order, and each context holds two distractor pointer chains\. A teacher trained on this variant must attend the chain, and imitation recovers: 0\.97 against 0\.99 to 1\.00 for causal supervision\. Attention can be a good target; whether it is depends on the teacher’s training problem\. Two more results come from this setting\. A second teacher learned everything except hardened multi\-hop \(0\.05 dense: it holds the chain circuit but cannot shield it from the distractors\), and the router trained on the first teacher transfers zero\-shot to route it at 0\.91, a repair of 86 points of accuracy from selection alone\. On three\-hop chains neither router nor teacher ever saw, selection stays essentially perfect \(full\-chain recall 0\.997 to 1\.0\) while routed accuracy reaches 0\.38 against 0\.05 dense: the teacher’s computation, not the selection, is the limit\.

## 6The labels can be recovered without annotations

The causal arm above trains on annotated causal evidence sets, which exist because the tasks are constructed\. To show the supervision needs no annotations, we hide them\. We label 8,000 training examples by the intervention readout alone, train routers on those labels, and return to the annotations only to score recovery\. The estimator matters, and each single\-signal choice fails where the measurements say it must \(Table[7](https://arxiv.org/html/2607.21692#A5.T7), appendix\)\. Estimating labels from sufficiency alone undershoots on chains, where a fallback keeps the restricted answer alive \(routers reach 0\.75\)\. Ranking blocks by masking effect at a fixed budget solves chains \(0\.97\) but pads one\-block tasks with noise \(latest write 0\.61\), and redundancy blinds the ranking entirely \(Section[4](https://arxiv.org/html/2607.21692#S4)\)\. The regimes separate in the interventions themselves\. Masking a block the computation runs through moves the answer log probability by a nat or more\. A nat is the natural\-logarithm unit of log probability, so one nat is a factor ofeein the probability itself\. Masking a redundant copy or a filler block moves it by near zero\. The estimator therefore keeps the ranked blocks that clear a 0\.25\-nat floor, capped at the routing budget, and falls back to sufficient singletons where nothing clears it\. Its labels match the annotated union on 87% of examples and contain a sufficient set on 92%\. Routers trained on them reach 0\.98 on multi\-hop \(full\-chain recall 0\.98 to 1\.00\), 0\.97 on unique retrieval, and 0\.91 on latest write, against 0\.99, 0\.98, and 0\.96 annotation\-trained\. The router also generalizes past its labels\. 82% of its chain labels are complete, yet routed chains exceed 98%\. An attribution baseline at the same budget, labeling the top blocks by input\-gradient saliency with one backward pass per batch, lands between the targets \(0\.79 on multi\-hop, 0\.92 on unique retrieval, 0\.87 on latest write\)\. Dependence signals beat attention weights even without interventions, and the intervention labels stay ahead everywhere\.

## 7The measurements transfer to frozen pretrained models

The synthetic teachers could be a special case, so we repeat the core measurements on frozen Qwen2\.5\-Instruct models\. The tasks become natural language: one record per sentence \(“The code for NARO is 4821\.”\), a question at the end, and, for the conflicting\-fact task, later sentences that update earlier ones\. Records’ token spans come from tokenizer offsets\. Ablation passes an additive attention mask with the synthetic operator’s semantics: the span is visible to itself, and nothing outside attends into it\. Scoring is forced choice among candidates, so nothing is generated under a mask\.

Table[6](https://arxiv.org/html/2607.21692#S7.T6)shows the same structure as the synthetic experiments\. The interventions are clean at every scale: masking the current record flips the forced choice to a stale value, masking a random record does not, and the current record alone is sufficient\. Dense on unique retrieval, every model scores 1\.00, and with the target record masked their choice among four candidates falls to the 0\.75 chance rate \(flip rates 0\.70 to 0\.79\); candidates come from outside the context, since in\-context candidates allow answering by elimination\. The attention finding persists where the task is solved: the 3B model answers 99% of conflicting\-fact queries correctly while giving the obsolete record more attention mass than the current one on 58% of them; the 7B model answers 96% and misattends on 43%\. Small models fail the task outright \(0\.23 and 0\.60\), preferring stale values; scale fixes the behavior before it fixes the attention\. A selector distilled from these attention maps would inherit exactly the stale evidence the model itself has learned to discount\.

Table 6:Frozen pretrained models on the conflicting\-fact task \(400 examples, 16 records, 2 obsolete writes; flip rates and sufficiency on solved examples\)\. Last column: fraction of solved examples where the obsolete record out\-attends the current one at the answer position\.The routing consequence follows\. We train a record\-level router against the frozen 3B model, the span analogue of Section[5](https://arxiv.org/html/2607.21692#S5), with pooled record and question summaries and a cosine record\-question similarity that carries name matching for rare tokens\. The causal router trains on the annotated record, the imitation router on the model’s answer\-position attention over records, both on 2,000 examples, both deployed by keeping the top two records\. The causal router matches the dense model on both tasks and stays there when the record count doubles past training \(0\.99 and 1\.00 on unique retrieval, 0\.99 and 0\.97 on conflicting facts, against dense 0\.99 to 1\.00\)\. The imitation router reaches 0\.45 and 0\.75 at the training count and falls to its random control at double length \(0\.31 and 0\.25, random 0\.29\)\. The 7B model repeats the pattern: causal 0\.93 to 1\.00 everywhere, imitation at its random control at the doubled count \(0\.27 and 0\.23\)\.

The pattern is not one family’s\. Yi\-1\.5\-9B solves both tasks dense, gives stale records more attention than current ones on 72% of examples, and routes the same way \(causal 0\.94 to 1\.00; imitation 0\.49 to 0\.85\)\. Imitation degrades there rather than collapsing: attention’s routing signal varies by family; its deficit does not\. Gemma\-2\-9B shows the opposite failure: dense, it fails conflicting facts at 0\.56, and with the causal router keeping two records it reaches 0\.99 and 0\.98, because masking the stale records removes a distraction the model cannot resolve\. Routing there is not a cheaper approximation of the dense model but a repair of it\. Anchors at Qwen2\.5\-14B match the smaller models \(Appendix[G](https://arxiv.org/html/2607.21692#A7)\)\. The result of Section[5](https://arxiv.org/html/2607.21692#S5)holds across three model families and three orders of magnitude of teacher scale: supervision copied from attention inherits these stale\-evidence failures, and supervision from causal evidence sets avoids them in these tests\.

These pretrained results use one operator, the attention mask; under a position\-preserving content overwrite that swaps each unselected record for a matched\-length nonce record, the conflicting\-fact repair and the causal\-over\-attention ordering both hold, narrowing to within noise only on Gemma conflicting facts, and on Yi\-1\.5\-9B, which solves that task, not at all \(Appendix[H](https://arxiv.org/html/2607.21692#A8)\)\.

Label recovery \(Section[6](https://arxiv.org/html/2607.21692#S6)\) transfers unchanged\. Hiding the annotations on the 3B model, one masked forward per record recovers them exactly: 1\.000 agreement over 2,000 examples, 34,000 forwards in 17 minutes on the same GPU\. The router trained purely on recovered labels routes at 0\.99 to 1\.00 across both tasks and both record counts, matching the annotation\-trained router\. Recovery holds as context grows: at 32, 64, and 128 records agreement stays 1\.000 and routing 0\.96 to 1\.00 \(Table[15](https://arxiv.org/html/2607.21692#A8.T15)\)\. The cost is about two forwards per record per example; examples label independently, and at the measured rate a million\-example corpus labels in under a week on the same card\. The procedure needs a scoreable answer, not task structure, so it extends to natural data\. We test it on SQuAD, with 16\-sentence contexts mixing the answer passage with distractor sentences\. Because these models answer many SQuAD questions from memory, the evidence sentence carries a counterfactual substitute answer, so only reading the kept evidence supports the correct choice\. Masking recovers the gold evidence sentence on 92% of examples\. At a two\-sentence budget the recovered\-label router scores 0\.47 and the attention\-imitation router 0\.35, against 0\.64 dense and 0\.19 random, which is below chance because missing evidence sends the model to its prior\. Both routers sit below dense, and the bottleneck is selection, not supervision or restriction: an oracle keeping the gold evidence scores 0\.77, above dense, while the routers keep it on only 59% and 41% of examples\. Natural text outgrows the small span router, not the labels\.

## 8Limits of sparse routing

The routing claims need a boundary, and the aggregation task provides it\. For each task, an oracle keeps the annotated sufficient blocks first and random filler after, giving accuracy as a function ofkkthat bounds any router from above\. Figure[1](https://arxiv.org/html/2607.21692#A6.F1)\(appendix\) shows the three regimes\. Unique retrieval is flat fromk=1k\{=\}1\. Multi\-hop is at floor throughk=2k\{=\}2and steps to 0\.99 exactly atk=3k\{=\}3, the chain size\. Aggregation stays at chance throughk=16k\{=\}16, including budgets that expose every marked block, and recovers only with the full context\. The aggregation regime is stronger than the task construction implies: the model’s counting does not survive masking even the blocks the count does not depend on, so its computation uses context beyond the task’s sufficient set, consistent with counting implemented against total attention mass\. Where computation is distributed, the sufficient set of the task and the sufficient set of the model diverge, and sparse routing at any useful budget is unavailable\. The mask and filler operators, which agree perfectly on retrieval, also diverge here, so operator choice must be validated per task family\.

Multi\-hop also cautions against reading a sufficient set as the exact route the computation used\. A greedy analysis \(Appendix[D](https://arxiv.org/html/2607.21692#A4)\) finds answer\-preserving sets smaller than the chain, because restriction can recruit fallbacks the full context never uses\. For routing that is harmless, since deployment is itself restriction, and the annotated chain is the conservative target\. The oracle curve above adds chain blocks in position order, which is why it steps at the chain size\.

## 9Related work

Sparse attention\.Fixed sparsity patterns predate learned selection\(Beltagy et al\.,[2020](https://arxiv.org/html/2607.21692#bib.bib6); Zaheer et al\.,[2020](https://arxiv.org/html/2607.21692#bib.bib25); Kitaev et al\.,[2020](https://arxiv.org/html/2607.21692#bib.bib12)\)\. Training\-free methods select or evict context using the model’s own attention or key statistics\(Zhang et al\.,[2023](https://arxiv.org/html/2607.21692#bib.bib26); Li et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib13); Tang et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib20); Jiang et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib11)\)\. Learned selection trains the signal instead\(Roy et al\.,[2021](https://arxiv.org/html/2607.21692#bib.bib18); Yuan et al\.,[2025](https://arxiv.org/html/2607.21692#bib.bib24); Lu et al\.,[2025](https://arxiv.org/html/2607.21692#bib.bib14); Gao et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib9); Ahmad & Yun,[2026](https://arxiv.org/html/2607.21692#bib.bib2)\)\. Across this line the importance signal is attention, copied or trained end to end; end\-to\-end training is confounded by co\-adaptation until learned gates barely beat random ones\(Aquino\-Michaels,[2026](https://arxiv.org/html/2607.21692#bib.bib3)\), and sparsity converts to wall\-clock speed only with kernel work\(Nawrot et al\.,[2025](https://arxiv.org/html/2607.21692#bib.bib16)\)\. Our design freezes dense teachers, supervises the selector externally, and checks the signal causally\.

Attention as importance\.That attention weights are unreliable importance estimates was argued for classification models\(Jain & Wallace,[2019](https://arxiv.org/html/2607.21692#bib.bib10); Serrano & Smith,[2019](https://arxiv.org/html/2607.21692#bib.bib19)\), disputed\(Wiegreffe & Pinter,[2019](https://arxiv.org/html/2607.21692#bib.bib22)\), and refined with flow\-based corrections\(Abnar & Zuidema,[2020](https://arxiv.org/html/2607.21692#bib.bib1)\)\. That debate lacked ground truth; the tasks here supply it, and Section[5](https://arxiv.org/html/2607.21692#S5)measures what the disagreement costs in routed accuracy\.

Causal analysis of transformers\.Our interventions follow the mediation and tracing lineage\(Vig et al\.,[2020](https://arxiv.org/html/2607.21692#bib.bib21); Meng et al\.,[2022](https://arxiv.org/html/2607.21692#bib.bib15)\)\. Automated circuit discovery recovers minimal causal subgraphs post hoc\(Conmy et al\.,[2023](https://arxiv.org/html/2607.21692#bib.bib8); Bhaskar et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib7)\), and retrieval\-head analysis identifies heads that move long\-context evidence\(Wu et al\.,[2024](https://arxiv.org/html/2607.21692#bib.bib23)\)\. None of these yields a per\-input selector that runs before attention is computed, and none measures whether the recovered evidence is consistent across training runs\. Head\-level stability studies find that heads differ across seeds\(Bali et al\.,[2026](https://arxiv.org/html/2607.21692#bib.bib5)\); our results locate the stable object in the evidence sets rather than the heads\. The training\-dynamics literature on associative recall\(Arora et al\.,[2023](https://arxiv.org/html/2607.21692#bib.bib4); Olsson et al\.,[2022](https://arxiv.org/html/2607.21692#bib.bib17)\)informed the task protocol\.

## 10Limitations

The tasks are synthetic, chosen so that ground truth exists; the teachers are small \(10\.7M parameters\) and use rotary position embeddings \(Appendix[B](https://arxiv.org/html/2607.21692#A2)\)\. Cost is accuracy at a block budget, not wall\-clock time; the paper studies supervision, not a deployable sparse runtime\. Router transfer covers seeds and training length, not tokenizers or architectures\. The floored estimator handles redundancy and chains but not their combination in one example; the greedy search covers that mixed regime at higher cost\. The pretrained experiments \(Section[7](https://arxiv.org/html/2607.21692#S7)\) cover three model families but only two tasks, forced\-choice scoring, and contexts of dozens of records; free\-form generation under routing and document\-scale contexts remain open\. Label recovery on natural QA needs a counterfactual control, and the floored estimator’s threshold is validated on one task mixture\.

## 11Conclusion

Sparse attention needs a target, and the teacher’s attention weights are the convenient one\. Where the needed context is known, that target fails in three measurable ways: it includes evidence whose removal changes nothing, misses evidence the computation ran through, and varies across training runs whose evidence is identical, with only interventions to tell a good instance from a poor one\. Causal evidence sets, recoverable by intervention, are stable where attention is not, and wherever sparse selection stayed viable a router trained on them or their recovered estimates matched or outperformed attention imitation, most on conflicting evidence and multi\-step retrieval\.

## Reproducibility statement

The released repository contains the task generators with their sufficient\-set annotations, the teacher training scripts, the intervention and label code, both router implementations, and every analysis script\.reproduce\.shreruns the stages in order, and Appendix[A](https://arxiv.org/html/2607.21692#A1)describes them\. The results files behind every number in this paper, including the appendix, are tracked in the repository as JSON, and all figures and tables are generated from those files by scripts rather than transcribed\. Seed ranges for training streams, evaluation sets, and routers are fixed in code and disjoint; each run directory stores its full configuration next to its metrics\. The model components come from a small open library the repository pins as a dependency\. Synthetic\-arm experiments total roughly two GPU\-days on one consumer card; the pretrained experiments add a few GPU\-hours per model using publicly available checkpoints\.

## Use of large language models

The author used an LLM assistant in the course of this work, including for parts of the implementation, for running and monitoring experiments, and in preparing parts of the manuscript\. All such use was directed and reviewed by the author, who set the research questions and decision criteria, checked the experimental designs and results, and takes responsibility for the claims\. Reported numbers are regenerated from the tracked results files by scripts in the repository\.

## References

- Abnar & Zuidema \(2020\)Samira Abnar and Willem Zuidema\.Quantifying attention flow in transformers\.In*Proceedings of ACL*, 2020\.
- Ahmad & Yun \(2026\)Huzama Ahmad and Se\-Young Yun\.Spotattention: Plug\-in block\-sparse routing for pretrained long\-context transformers\.*arXiv preprint arXiv:2606\.22874*, 2026\.
- Aquino\-Michaels \(2026\)Keston Aquino\-Michaels\.Routing absorption in sparse attention: Why random gates are hard to beat\.*arXiv preprint arXiv:2603\.02227*, 2026\.
- Arora et al\. \(2023\)Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré\.Zoology: Measuring and improving recall in efficient language models\.*arXiv preprint arXiv:2312\.04927*, 2023\.
- Bali et al\. \(2026\)Karan Bali, Jack Stanley, Praneet Suresh, and Danilo Bzdok\.Quantifying llm attention\-head stability: Implications for circuit universality\.*arXiv preprint arXiv:2602\.16740*, 2026\.
- Beltagy et al\. \(2020\)Iz Beltagy, Matthew E\. Peters, and Arman Cohan\.Longformer: The long\-document transformer\.*arXiv preprint arXiv:2004\.05150*, 2020\.
- Bhaskar et al\. \(2024\)Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen\.Finding transformer circuits with edge pruning\.*Advances in Neural Information Processing Systems*, 2024\.
- Conmy et al\. \(2023\)Arthur Conmy, Augustine N\. Mavor\-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga\-Alonso\.Towards automated circuit discovery for mechanistic interpretability\.In*Advances in Neural Information Processing Systems*, 2023\.
- Gao et al\. \(2024\)Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Hayden Kwok\-Hay So, Ting Cao, Fan Yang, and Mao Yang\.Seerattention: Learning intrinsic sparse attention in your llms\.*arXiv preprint arXiv:2410\.13276*, 2024\.
- Jain & Wallace \(2019\)Sarthak Jain and Byron C\. Wallace\.Attention is not explanation\.In*Proceedings of NAACL\-HLT*, 2019\.
- Jiang et al\. \(2024\)Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H\. Abdi, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\.Minference 1\.0: Accelerating pre\-filling for long\-context llms via dynamic sparse attention\.In*Advances in Neural Information Processing Systems*, 2024\.
- Kitaev et al\. \(2020\)Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya\.Reformer: The efficient transformer\.In*Proceedings of ICLR*, 2020\.
- Li et al\. \(2024\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\.Snapkv: Llm knows what you are looking for before generation\.In*Advances in Neural Information Processing Systems*, 2024\.
- Lu et al\. \(2025\)Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al\.Moba: Mixture of block attention for long\-context llms\.*arXiv preprint arXiv:2502\.13189*, 2025\.
- Meng et al\. \(2022\)Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\.Locating and editing factual associations in gpt\.In*Advances in Neural Information Processing Systems*, 2022\.
- Nawrot et al\. \(2025\)Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M\. Ponti\.The sparse frontier: Sparse attention trade\-offs in transformer llms\.*arXiv preprint arXiv:2504\.17768*, 2025\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, et al\.In\-context learning and induction heads\.*Transformer Circuits Thread*, 2022\.
- Roy et al\. \(2021\)Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier\.Efficient content\-based sparse attention with routing transformers\.*Transactions of the Association for Computational Linguistics*, 9:53–68, 2021\.
- Serrano & Smith \(2019\)Sofia Serrano and Noah A\. Smith\.Is attention interpretable?In*Proceedings of ACL*, 2019\.
- Tang et al\. \(2024\)Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han\.Quest: Query\-aware sparsity for efficient long\-context llm inference\.In*Proceedings of ICML*, 2024\.
- Vig et al\. \(2020\)Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber\.Investigating gender bias in language models using causal mediation analysis\.In*Advances in Neural Information Processing Systems*, 2020\.
- Wiegreffe & Pinter \(2019\)Sarah Wiegreffe and Yuval Pinter\.Attention is not not explanation\.In*Proceedings of EMNLP\-IJCNLP*, 2019\.
- Wu et al\. \(2024\)Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu\.Retrieval head mechanistically explains long\-context factuality\.*arXiv preprint arXiv:2404\.15574*, 2024\.
- Yuan et al\. \(2025\)Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y\. X\. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng\.Native sparse attention: Hardware\-aligned and natively trainable sparse attention\.In*Proceedings of ACL*, 2025\.
- Zaheer et al\. \(2020\)Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed\.Big bird: Transformers for longer sequences\.In*Advances in Neural Information Processing Systems*, 2020\.
- Zhang et al\. \(2023\)Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\.H2o: Heavy\-hitter oracle for efficient generative inference of large language models\.In*Advances in Neural Information Processing Systems*, 2023\.

## Appendix AReproduction

reproduce\.shruns the stages in order: unit and property tests \(the task generators are property\-tested against a brute\-force oracle\), teacher training, sanity anchors, the agreement report, router training with the ablation arms, oracle budget curves, the chain\-router and hardened arms, label recovery with its estimator ablation, the pretrained probes, routing, recovery, and QA, and figure and table generation\. The label pipeline costs two forward passes per block per example plus one per annotated set\.

## Appendix BTask and protocol details

Records are four tokens,WRITEfamily member value, padded with filler to the block width; queries areQUERYfamily member value with the value supervised directly at the member token\. Two protocol findings are worth recording because both failure modes look like model incapacity\. First, placing a marker token between the query key and the supervised value prevents retrieval circuits from forming at any model size or learning rate we tried; the loss settles at the guess\-a\-context\-value level indefinitely\. Second, the workable learning rate depends on the record count: 8\-record grids train at10−310^\{\-3\}, 16 records and up sit at the same plateau until the rate drops to3×10−43\\times 10^\{\-4\}; staged curricula over record counts saturate below half accuracy with a loss spike at each stage switch, and per\-batch mixtures of record counts never leave the plateau\. Training sequences carry eight queries; evaluation is single\-query, an exact prefix of the training format\. The aggregation answer is a count, not a parity: parity never trains at this scale \(teachers sit at the label base rate for tens of thousands of steps\), and a control the dense model cannot solve bounds nothing\. Finally, the positional encoding is load\-bearing: teachers with learned absolute positions never leave the same plateau at any learning rate we tried \(10−410^\{\-4\},3×10−43\\times 10^\{\-4\},10−310^\{\-3\}; up to 45,000 steps\), while rotary embeddings train cleanly, so every teacher in the paper uses rotary embeddings and the encoding axis is untested rather than tested and passed\.

## Appendix CHyperparameters

Teachers: decoder\-only, 6 layers, 8 heads, width 384, SwiGLU width 1024, RoPEθ=104\\theta\{=\}10^\{4\}, positional table sized for the 4×\\timesevaluations, AdamW with weight decay 0\.01, cosine schedule from3×10−43\\times 10^\{\-4\}to3×10−53\\times 10^\{\-5\}with 200 warmup steps, gradient clip 1\.0, batch 64, 30,000 steps \(45,000 for the five\-task mixture\), fresh examples every step\. Routers: token embedding width 64, mean\-pooled block and query summaries, a two\-layer scorer, Adam at10−310^\{\-3\}, 4,000 steps at batch 64 \(32 on shared\-GPU runs\); the causal arm anneals its softmin temperature linearly to zero over the first 80% of training\. The span router for the pretrained arm adds a cosine interaction feature between record and query summaries and trains for 15 epochs over 2,000 examples\. Evaluation budgets are⌈0\.1​n⌉\\lceil 0\.1n\\rceilblocks for the synthetic arm and 2 records for the pretrained arm\.

## Appendix DMinimality of behaviorally sufficient sets

The greedy search \(add blocks in order of masking effect until the restricted answer is preserved, then prune\) finds behaviorally sufficient sets of 1\.4 to 1\.6 blocks on solved multi\-hop examples, strict subsets of the three\-block chain on 86 to 100% of them: restricted to the final value record alone, the model often answers with the only value it can see\. The chain blocks remain individually necessary, with flip rates of 0\.64 to 0\.95 across seeds whose variation tracks the robustness of the fallback rather than different routes; reading sufficient sets as the computation’s information route is sound only up to this bound\.

## Appendix ERecovery estimator ablation

Table[7](https://arxiv.org/html/2607.21692#A5.T7)reports every estimator behind Section[6](https://arxiv.org/html/2607.21692#S6)\. Two null results sharpen the design\. Extending the sufficiency family with the smallest answer\-preserving prefix of the masking\-effect ranking changes nothing: behavioral answer preservation is the wrong size criterion wherever a fallback exists \(Appendix[D](https://arxiv.org/html/2607.21692#A4)\)\. And branching on singleton sufficiency before consulting effect sizes sends chain examples with a strong fallback down the wrong branch, handing the router a shortcut label\. Sweeping the floor from 0\.25 to 4\.0 nats moves overall exact match from 0\.87 to 0\.75 with the loss concentrated on chains, so the choice is not delicate downward\.

Table 7:Recovery estimators on the synthetic mixture: label agreement with the hidden annotations over the 8,000 training examples, and routed accuracy of routers trained only on the recovered labels \(mean of three router seeds, 300 examples per condition\)\. Cover: the recovered set contains an annotated sufficient set\. Each single\-signal estimator fails in the regime Section[2\.3](https://arxiv.org/html/2607.21692#S2.SS3)predicts; the floored estimator, whose every size decision is an intervention, is the one the main text reports\. The last row is an attribution baseline at the same budget, labeling the top blocks by input\-gradient saliency\.
## Appendix FBudget curves

Figure[1](https://arxiv.org/html/2607.21692#A6.F1)gives the oracle accuracy\-against\-budget curve behind the routing boundary of Section[8](https://arxiv.org/html/2607.21692#S8), with the annotated sufficient blocks kept first in position order, so it upper\-bounds any router\.

![Refer to caption](https://arxiv.org/html/2607.21692v1/x1.png)Figure 1:Accuracy against block budgetkkwhen the selection keeps the annotated sufficient blocks first, in position order \(mean of two teachers trained on the five\-task mixture\)\. Unique retrieval is flat fromk=1k\{=\}1; multi\-hop steps up at the chain size under this selection order \(see the minimality caution in the text\); aggregation stays at chance until the full context\. The aggregation count varies per example, so a constant answer scores 0\.2\.
## Appendix GPer\-seed results

Tables[8](https://arxiv.org/html/2607.21692#A7.T8)through[13](https://arxiv.org/html/2607.21692#A7.T13)report per\-seed teacher accuracy, per\-seed routing for every condition and arm, the label\-by\-loss ablation per seed, per\-pair label agreement, the tuning guard, and the pretrained unique\-retrieval numbers\. Figures[2](https://arxiv.org/html/2607.21692#A7.F2)through[4](https://arxiv.org/html/2607.21692#A7.F4)accompany them: how redundancy separates the estimator families, routed accuracy across supervisions, and the scale\-before\-attention pattern on the frozen models\.

![Refer to caption](https://arxiv.org/html/2607.21692v1/x2.png)Figure 2:Redundancy separates the estimators\. As planted copies of the answering record increase, the rate at which masking a single causal block flips the answer collapses, and labels ranked by single\-block ablation decay with it; the recovered family of sufficient sets grows more consistent across seeds\. Attention sets sit in between throughout\.![Refer to caption](https://arxiv.org/html/2607.21692v1/x3.png)Figure 3:Routed accuracy at the 10% block budget, means over teacher seeds\. Single\-record rows use pooling routers \(Table[3](https://arxiv.org/html/2607.21692#S5.T3)\); the multi\-hop row uses the chain router \(Table[4](https://arxiv.org/html/2607.21692#S5.T4)\)\. The supervisions tie in distribution, separate under length shift on unique retrieval, and separate completely on multi\-hop\.![Refer to caption](https://arxiv.org/html/2607.21692v1/x4.png)Figure 4:Frozen Qwen2\.5\-Instruct on the conflicting\-fact task\. Scale fixes the behavior \(accuracy\) before it fixes the attention: at 3B and 7B the task is solved while the obsolete record still out\-attends the current one on 61% and 40% of examples\.Table 8:Final teacher accuracy per task and seed\.Table 9:Per\-seed routed accuracy at the 10% budget \(single\-hop teachers, routers trained against seed 0\)\.Table 10:Per\-seed results for the label\-by\-loss ablation \(five\-task teachers\)\. Seed 3 never learned multi\-hop \(dense 0\.04\) and is reported for completeness\.Table 11:Cross\-seed agreement per seed pair: causal\-evidence labels \(left of slash\) against adaptive attention \(right\)\.Table 12:Tuning guard: the imitation router retrained at other learning rates and at triple steps \(means over the three\-seed teachers of that phase\)\.Table 13:Frozen pretrained models on unique retrieval \(400 examples per model; Qwen2\.5\-14B is a 60\-example anchor; candidates from outside the context; chance flip under full removal is 0\.75\)\.
## Appendix HOperator robustness and record\-count scaling

Table[14](https://arxiv.org/html/2607.21692#A8.T14)re\-scores the pretrained routing selection under two operators: the additive attention mask used throughout, and a position\-preserving content overwrite that replaces each unselected record with a matched\-length nonce record carrying a fresh name and value\. In this latter case the token count, sentence boundaries, and the query position are unchanged and the ablated content is physically absent rather than masked\. Overwrite figures average three nonce fillers, and the routers and keep\-sets are held fixed, so only the operator varies\. Removing the current record flips the answer under both operators; removing a random one does not\. The causal\-over\-attention ordering and the conflicting\-fact repair survive the operator change: routing to the annotated current record lifts Gemma from 0\.55 stale\-biased dense to 1\.00 under both\. The trained routers narrow in only one place: Gemma conflicting facts under overwrite\. There the visible nonce records distract a model that already answers stale on 45% of these queries dense, and a random\-record overwrite flips 8% of its answers, against 0% on Qwen\. The mask and overwrite operators diverge here, as they do on aggregation \(Section[8](https://arxiv.org/html/2607.21692#S8)\), but the repair holds under both\. A third family settles which reading of the narrowing is right: Yi\-1\.5\-9B, which unlike Gemma answers conflicting facts correctly dense, holds the causal\-over\-attention ordering and near\-zero random\-record flip under overwrite\. Thus the narrowing is specific to Gemma’s stale\-answering computation rather than a property of 9B scale or of the overwrite operator\. Table[15](https://arxiv.org/html/2607.21692#A8.T15)extends label recovery to longer contexts: at 16, 32, 64, and 128 records the recovered labels match annotations exactly and the recovered\-label router stays at 0\.96 to 1\.00, at a cost of one masked forward per record\.

Table 14:Operator robustness \(150 examples; overwrite averaged over three nonce fillers\)\. The two operators are defined in the text\. Causal and Attention give the causal\-supervision and attention\-imitation routers’ accuracy; Repair keeps only the annotated current record; Flip cur\. and Flip rnd\. are the rates at which removing the current or a random record changes the answer\.Table 15:Annotation\-free recovery and recovered\-label routing on Qwen2\.5\-3B as the record count grows\. Agreement is exact match of recovered labels against annotations; routing is the recovered\-label router evaluated at its training record count\. Recovery cost is one masked forward per record\.