Through the Looking Glass: Directly Reading and Writing Transformers

# Through the Looking Glass:Directly Reading and Writing Transformers
Source: [https://arxiv.org/html/2609.10210](https://arxiv.org/html/2609.10210)
Mark OskinAffiliation:ProfessorAffiliation:School of Computer Science and EngineeringAffiliation:University of WashingtonEmail:[mhoskin@uw\.edu](mailto:)

September 2026

###### Abstract

How many of a transformer’s components decide a token? Counted the standard way — feed\-forward units and attention channels, scored by the absolute value of what each contributes to the logit — one prediction rests on three thousand to two hundred thousand of them\. But contributions are signed, and across the eighteen models here the mass pushing away from the predicted token is a median of seven times the mass carrying it\. Divide by the net instead and the same attribution on the same predictions returns dozens\. On the model this paper is built on, fifty\-three components carry ninety percent of a prediction, thirteen it cannot survive losing, and eight suffice to produce it with everything else at that position zeroed\. Across twelve models trained by other people,124124million to seven billion parameters, the sufficient set runs from two components to sixteen, and what a prediction draws on, followed all the way back, is one to three percent of the model, a share that does not grow with size\. The rest of the model is neither idle nor holding work for later tokens: three quarters of what a layer adds to the residual is a fixed linear map of the state it received\.

Everything here is read from the model’s own parameters and activations, with nothing trained and nothing fitted, and it names a component on both sides\. What a component writes is named from the predictions it drives, reaching close to half of every model, and the name is a share of the drive that predicts what deleting the component costs\. What it reads is decoded from its weights in the frame of its own layer, at58\.958\.9percent above chance over its eight strongest inputs and below chance once enough of the list is asked for; sorting the remainder by which upstream component supplies it separates it into grammatical categories the embedding cannot see\.

A name can be acted on\. An association the model does not hold installs into one spare unit, key and value read from the weights, moving the target from rank578578to rank11for a quarter of one percent of held\-out loss — a fortieth of what a rank\-one update costs to reach the top ten\. Where an edit lands is set by the architecture: a write before the second layer keeps a tenth of its direction at the readout, one in the last third up to eight tenths\. An attention head installed to mark a token and a unit installed two layers above it to read the mark make an edit fire only where that token occurred earlier in the context, and a unit the model trained for itself is driven from two layers upstream with8686percent of the effect passing through it\. Replacing the activation with an order\-preserving one makes a unit’s inputs readable at the instrument’s ceiling, flat across the list rather than decaying, at the price of turning an install into a two\-part edit\.

## 1Introduction

A transformer produces a token\. How, specifically, did it do that? Are there identifiable parts of the model that matter to that particular prediction, and can they be traced — a “circuit” used*for that specific token*? They can, and the circuit is small\. A handful up to a few dozen components do the work behind any one prediction, and most of what a transformer computes at a position is a side effect of the architecture: moving data between layers and keeping its own geometry organized\.

That is not what the literature would lead one to expect\. In the basis a transformer actually has, individual feed\-forward units and attention channels, published counts run to thousands or tens of thousands of components for a single behavior\[[33](https://arxiv.org/html/2609.10210#bib.bib31)\], and the accounting in common use reproduces that here: measured the standard way, one prediction in the models below rests on three thousand to two hundred thousand components\. A number that size is not an explanation of anything\.

It is also wrong\. Contributions to a logit are signed, so they cancel, and the mass pushing away from the prediction is a median of seven times the net across the models here and twenty times it on the model this paper is built on\. The standard count divides by the absolute total, and so counts that opposing mass as part of the answer\. Divide by the net instead and the same attribution on the same predictions returns dozens\. The opposing mass is not a rival circuit the net hides\. A model writes its prediction into the token frame directly rather than assembling it elsewhere and turning it on, and it does not use that frame as scratch space\[[41](https://arxiv.org/html/2609.10210#bib.bib14)\]; so the net contribution is the model writing the output, and the rest is the residual stream being carried between layers and kept organized so that many things can share it\[[17](https://arxiv.org/html/2609.10210#bib.bib5)\]\. Our lens separates that management from the prediction, and what remains is a small legible circuit\.

This paper concerns the legibility and editability of transformers\. What exactly does that mean? The intuitive versions are easy enough to state\.

- •A component is legibleif what it does can be stated\. Two things have to be readable: what drives it, and what it produces\. A name for either has to meet two conditions\. It must be*coherent*: the tokens that drive the component share something, and so do the tokens it promotes\. And it must be*verified*against the model’s behavior rather than against whether the list looks sensible\. What a component is credited with writing depends about equally on its weights and on when it fires, so a column read on its own narrows the answer to a seventh of the vocabulary and no further \(Appendix[C\.11](https://arxiv.org/html/2609.10210#A3.SS11)\)\. What does supply them is behavior: the predictions a component drives name what it writes, and the positions where it fires name what it reads\. Where no token covers the answer, a component can still be described by the circuit it fits within\.
- •A model is legibleif the connections between its components can be followed\. When it generates a token there must be a trace of why it produced*that*one, and the trace must explain the choice to someone reading it\. The path must be traceable end to end and land in the token embedding space where it meets the input and the output, even when what it passes through in between is conceptual\. Ideally a trace is small enough for a person to read; it must in any case be complete enough to include what actually decided the token\.
- •A component is editableif its behavior can be changed, the model respects the change, and everything unrelated is left alone\. Install an intended behavior into one named component, measure how strongly the model takes it, and measure how much else moved\. Neither number means anything alone, since any edit is respected if pushed hard enough and any edit is harmless if pushed gently\. Editability is therefore a trade\-off rather than a threshold\.

Our aim was a lens derived from the transformer’s own parameters, with no training or fitting of its own\. Good fitted explainers exist: a sparse autoencoder trained on a model’s activations recovers features a person can name, and we reproduce that result in Appendix[C](https://arxiv.org/html/2609.10210#A3)\. The reason we did not build on one is that a fitted explainer is a second model, and understanding it becomes a prerequisite to understanding the transformer it was meant to explain\. Simple is good if it works\.

On the model this paper is built on, the lens returns three numbers answering three questions about one prediction\. Fifty\-three components carry ninety percent of it\. Thirteen it cannot survive losing: removing them changes the answer, where removing the same number at random leaves it alone\. And eight suffice to produce it — keep those eight at a position, zero every other feed\-forward unit and attention channel there, let the survivors recompute, and the model emits the same token\. Eight random components in their place never do, on any of115115predictions we tested\.

The rest of the model is neither idle nor holding work for later tokens\. Keep only the circuit at a position and the cost at the token being predicted is a quarter of what an equally sized random set costs, and on each of the next five hundred and twelve tokens the two cost the same\. What the remainder is doing is managing the workspace: three quarters of what a layer adds to the residual is a fixed linear map of the state it received, applied whatever the model is about to say, and that work is spread across every component rather than concentrated in a few\. Cancellation is far more severe arriving at a component than leaving the model — about a dozen upstream components reach ninety percent of a unit’s incoming drive counted signed and many thousands counted absolute, a factor of roughly a thousand against about twenty at the readout\.

Most of this paper’s effort goes into the gap between finding a component and reading it\. A component is named on both sides\. What it writes is named from the predictions it drives, which reaches close to half of every model, and the name is a share of the drive that predicts what deleting the component costs\. What it reads is named from its own weights, decoded in the frame of its own layer rather than through the vocabulary: asked for a unit’s eight strongest drivers, the parameters recover the true set at58\.958\.9percent, thirty\-seven times chance, and that agreement is confined to the head of the list\. Somewhat over a quarter of what drives a unit falls on tokens the embedding groups into a category; the remaining three quarters have no name in the vocabulary at all, being concepts the model builds for itself, and sorting them by which upstream component supplies them separates them into grammatical categories the embedding cannot see\. The same reading extends to attention, where a channel’s value projection is evaluated at the position it attends to, and the drop between what a layer’s attention reads and what its feed\-forward block reads is the context that layer delivered\.

Naming a component is one half of the problem; acting on it is the other\. An association the model did not hold installs into a single spare component, with the key and the value both derived from the weights without fitting, moving the target from rank578578to rank11for a quarter of one percent of held\-out loss — a fortieth of what the rank\-one update the editing literature established costs to reach the top ten\. Where such a write can be made turns out to be a property of the architecture rather than of the component chosen\. The bottom two layers erase the input token and stash it off the token axis, so a write made there loses most of its direction by the readout\. The top layer answers in the token frame, which is the frame the readout reads and the only one a write acts on directly, so the last hop of a circuit belongs in that frame and the earlier hops do not — though not in that layer, which consumes readout\-frame content rather than carrying it, and is the one place preservation falls instead of rising\. Between the two, the residual carries a write in the coordinates it was written in, and the baseline uses about a quarter of its width, the same quarter at every depth, so a direction drawn at random carries a signal from one installed component to another with nothing else in the model oriented to read it\. That is enough to build a circuit across two layers from components that did not exist before, and to drive a unit the model trained for itself from two layers upstream, with8686percent of the effect passing through it\.

Two further sections take the lens beyond the model it was built on\. Section[5](https://arxiv.org/html/2609.10210#S5)replaces the activation function with an order\-preserving one, sigmoid or softplus, which makes a unit’s inputs readable at the instrument’s ceiling, flat across the whole list where the conventional unit decays below chance; the price is that such a unit is never silent, so an association installed into it alone arrives everywhere at once, and the install becomes a two\-part edit\. ReLU buys legibility with accuracy\. Section[6](https://arxiv.org/html/2609.10210#S6)applies the lens unchanged to twelve models trained by other people, from124124million to77billion parameters across seven architecture families\. There the sufficient set runs from two components to sixteen with the largest models at its small end, and on gated feed\-forward models nearly half the components carrying a prediction fire negative, so a scope read without that sign lands on the wrong tokens\.

Section[2](https://arxiv.org/html/2609.10210#S2)builds the lens and applies it to the baseline: the accounting, what is necessary, what is sufficient, and what the rest of the model is doing\. Section[3](https://arxiv.org/html/2609.10210#S3)names components, on the write side from behavior and on the read side from weights, and follows the part the vocabulary cannot name back to its sources\. Section[4](https://arxiv.org/html/2609.10210#S4)acts on the names: a gain, an install, where an edit can be placed, a circuit across two layers, a tap into a trained unit, and a comparison against a rank\-one editor\. Sections[5](https://arxiv.org/html/2609.10210#S5)and[6](https://arxiv.org/html/2609.10210#S6)are the two above\. Section[7](https://arxiv.org/html/2609.10210#S7)places the work, Section[8](https://arxiv.org/html/2609.10210#S8)takes up what the measurements ask of the architecture, and Section[9](https://arxiv.org/html/2609.10210#S9)closes\. Appendix[A](https://arxiv.org/html/2609.10210#A1)is a gallery of traces drawn mechanically rather than chosen, two per model, where a reader can see what a trace actually looks like; Appendix[B](https://arxiv.org/html/2609.10210#A2)is the set\-operator model of Section[5](https://arxiv.org/html/2609.10210#S5)and how it is trained; Appendix[C](https://arxiv.org/html/2609.10210#A3)is the measurements that failed, the conventions that flatter and are therefore not used here, and the ways each number here could have been made to look better by choosing a different null\. Appendix[D](https://arxiv.org/html/2609.10210#A4)is the dependency graph: the edge relation between components, the closure it builds, and why that closure is a necessity object rather than a sufficient one\.

## 2Reading a prediction: The Looking Glass

This section builds the instrument necessary to read a transformer\. We begin with the baseline model used for every measurement here, and with the further models used later in the paper\. We then introduce the lens, and show how it identifies the parts of a transformer doing real work for a prediction\. Next we ask which parts are*necessary*for a correct prediction and which are*sufficient*to produce it, and then what the rest of the model is doing while that small set carries the answer\. We close with the limitations of the lens\.

### 2\.1Models

The baseline is a model trained here on the GPT\-2 small architecture\[[45](https://arxiv.org/html/2609.10210#bib.bib23)\]: twelve layers of widthd=768d=768, twelve attention heads, learned positional embeddings, LayerNorm, tied input and output embeddings, and no bias term anywhere\. It is trained at context lengthT=2048T=2048rather than the released model’s10241024, and the released GPT\-2 appears separately among the models of Section[6](https://arxiv.org/html/2609.10210#S6)\. Writingx\(l\)x^\{\(l\)\}for the residual stream entering layerll,

x\(l\+1\)=x\(l\)\+Attn\(l\)​\(LN⁡\(x\(l\)\)\)\+FFN\(l\)​\(LN⁡\(h\(l\)\)\),x^\{\(l\+1\)\}\\;=\\;x^\{\(l\)\}\\;\+\\;\\mathrm\{Attn\}^\{\(l\)\}\\\!\\big\(\\mathrm\{LN\}\(x^\{\(l\)\}\)\\big\)\\;\+\\;\\mathrm\{FFN\}^\{\(l\)\}\\\!\\big\(\\mathrm\{LN\}\(h^\{\(l\)\}\)\\big\),\(1\)withh\(l\)h^\{\(l\)\}the stream after the attention write\. The feed\-forward block is the object this paper is about:

FFN\(l\)​\(z\)=Wout\(l\)​ϕ​\(Win\(l\)​z\),Win\(l\)∈ℝm×d,m=4​d=3072\.\\mathrm\{FFN\}^\{\(l\)\}\(z\)\\;=\\;W\_\{\\text\{out\}\}^\{\(l\)\}\\,\\phi\\big\(W\_\{\\text\{in\}\}^\{\(l\)\}z\\big\),\\qquad W\_\{\\text\{in\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\},\\;m=4d=3072\.\(2\)The block decomposes intommindependent*units*\. Unituuowns one row of the up\-projection and one column of the down\-projection,

wu=Win\(l\)\[u,:\]∈ℝd,cu=Wout\(l\)\[:,u\]∈ℝd,w\_\{u\}=W\_\{\\text\{in\}\}^\{\(l\)\}\[u,:\]\\in\\mathbb\{R\}^\{d\},\\qquad c\_\{u\}=W\_\{\\text\{out\}\}^\{\(l\)\}\[:,u\]\\in\\mathbb\{R\}^\{d\},\(3\)and contributesau​\(z\)​cua\_\{u\}\(z\)\\,c\_\{u\}to the block output, where its activation is the scalar

au​\(z\)=ϕ⁡\(wu⋅z\)\.a\_\{u\}\(z\)\\;=\\;\\phi\\big\(w\_\{u\}\\cdot z\\big\)\.\(4\)The read rowwuw\_\{u\}decides which inputs drive the unit, and the write columncuc\_\{u\}decides what it contributes once driven\. Neither projection carries a bias, which is worth noting here because Section[4](https://arxiv.org/html/2609.10210#S4)has to build one out of the layer’s own geometry when an installed unit needs an off state\. A component in what follows is one feed\-forward unit or one \(head, channel\) pair of attention, in both cases a scalar multiplying a write column, so the two are counted on the same footing\. For the baseline,

ϕ⁡\(ζ\)=GELU⁡\(ζ\)=ζ​Φ​\(ζ\),\\phi\(\\zeta\)\\;=\\;\\mathrm\{GELU\}\(\\zeta\)\\;=\\;\\zeta\\,\\Phi\(\\zeta\),\(5\)withΦ\\Phithe standard normal CDF\. Two properties of Equation[5](https://arxiv.org/html/2609.10210#S2.E5)are recorded here because they matter later: GELU is unbounded above, and it is not monotone, having a minimum nearζ=−0\.7517\\zeta=\-0\.7517and rising on both sides of it\.

#### 2\.1\.1Training and seeds

Training is one epoch of OpenWebText at batch size 16, peak learning rate3×10−43\\times 10^\{\-4\}with 2000 warmup steps and cosine decay to a tenth of peak, for 272,687 steps\. Nothing about the recipe is unusual\.

Eight seeds were trained, differing only in initialization and data order\. Five converged and three did not, by a margin wide enough that nothing turns on where the line is drawn: the failures sit at perplexity 21\.6, 24\.4 and 39\.5 against a converged band of 19\.0 to 19\.7\. Table[1](https://arxiv.org/html/2609.10210#S2.T1)shows all eight rather than the five, because broken models can be deceptively legible, as Appendix[C](https://arxiv.org/html/2609.10210#A3)shows\. The seeds are therefore separated on quality alone, fixed before any legibility number is computed\.

Table 1:The eight baseline seeds\. Only initialization and data order differ\. The excluded seeds are shown rather than dropped because Appendix[C](https://arxiv.org/html/2609.10210#A3)needs them\. Two columns are defined properly in Table[14](https://arxiv.org/html/2609.10210#S3.T14)and appear here only to show that the excluded seeds differ on more than perplexity:ov​@​8\\mathrm\{ov\}@8is how well a unit’s parameters name the inputs that drive it, andPR\\mathrm\{PR\}is the participation ratio of the early layers, a measure of how much representational variety the model retained\. A collapsed model reads as more legible, which is why the two are always quoted together\.ov​@​8\\mathrm\{ov\}@8here is taken under the ranked summary of Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4), raw rather than chance\-corrected\.We use LAMBADA\[[44](https://arxiv.org/html/2609.10210#bib.bib24)\]to assess model quality in addition to perplexity\. The five converged seeds are of similar quality, spanning0\.2480\.248to0\.2740\.274, a spread we read as seed and evaluation noise; later comparisons are held against that range, so two models whose ranges overlap are called equal\.

#### 2\.1\.2Additional models used

Two further groups of models appear from Section[3](https://arxiv.org/html/2609.10210#S3)onward\. The first is five more models trained here, listed in Table[2](https://arxiv.org/html/2609.10210#S2.T2); each changes only what the feed\-forward unit does with its pre\-activation, so any difference between them is attributable to that change\.

Table 2:The models trained for this paper\. The five others differ from the baseline only in the feed\-forward unit, sharing its depth, width, head count, parameter budget, data and schedule\. The case for choosing an activation function on these grounds is Section[5](https://arxiv.org/html/2609.10210#S5), where the set\-operator row is taken up as well and Appendix[B](https://arxiv.org/html/2609.10210#A2)defines it; this table is a reference for the rows that appear in the tables that follow\.The second is twelve models trained by other people, none of them modified for this work: GPT\-2, OPT, SmolLM2, three of Qwen2\.5, Gemma\-3, OLMo\-2, two of Llama\-3\.2, TinyLlama and Mistral, spanning124124million to77billion parameters across seven architecture families and three activation functions\. Section[6](https://arxiv.org/html/2609.10210#S6)is where they are the subject, and Table[35](https://arxiv.org/html/2609.10210#S6.T35)lists them with their sizes and activations\.

One family is deliberately absent from that set: the GPT\-NeoX line, of which the Pythia suite is the part we tested\. Those models compute their two sublayers in parallel from the same input,x\+Attn⁡\(LN1​\(x\)\)\+FFN⁡\(LN2​\(x\)\)x\+\\mathrm\{Attn\}\(\\mathrm\{LN\}\_\{1\}\(x\)\)\+\\mathrm\{FFN\}\(\\mathrm\{LN\}\_\{2\}\(x\)\), rather than letting the feed\-forward block read what attention has just written, and every reading in this paper assumes the sequential arrangement\. Extending the work to that design is left for later\.

Results in Sections[2](https://arxiv.org/html/2609.10210#S2)through[4](https://arxiv.org/html/2609.10210#S4)are measured on the baseline\. The exceptions are noted where another model behaves differently\.

### 2\.2The lens

The lens is built from three pieces: an accounting of the mass each component writes toward the prediction, a decision about which of that mass counts as explaining it \(Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)\), and a frame in which anything read below the readout is scored\.

Write the logit of tokenttas an exact sum over components\. Each componentuucontributes its activationaua\_\{u\}times the projection of its write columncuc\_\{u\}\(Equation[3](https://arxiv.org/html/2609.10210#S2.E3)\) onto the readout direction,

κu=au​⟨cu,γ⊙\(Ut−U¯\)/ρ⟩,∑uκu=N,\\kappa\_\{u\}\\;=\\;a\_\{u\}\\,\\big\\langle c\_\{u\},\\;\\gamma\\odot\(U\_\{t\}\-\\bar\{U\}\)/\\rho\\big\\rangle,\\qquad\\sum\_\{u\}\\kappa\_\{u\}\\;=\\;N,\(6\)withγ\\gammathe final\-norm gain andρ\\rhothe norm of the final residual state\. The sum runs over feed\-forward units and attention channels, soNNis the net those components deliver\. The token and positional embeddings reach the readout along the residual path and are not among the terms; their share is the floor that Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)measures and sets aside\.

The logit lens and the tuned lens are*projections*: they choose what to read a hidden state through\[[38](https://arxiv.org/html/2609.10210#bib.bib1),[6](https://arxiv.org/html/2609.10210#bib.bib2)\]\. The projection here is the ordinary one\. What this lens changes is the*accounting*applied afterward, which of the terms in Equation[6](https://arxiv.org/html/2609.10210#S2.E6)are counted as explaining the prediction and which are set aside as the architecture’s own traffic\.

##### The embedding frame\.

A transformer’s residual stream does not arrive in token coordinates until very late, and models mostly operate off the token axis\. The prediction reaches that axis in the last few layers by being written onto it, rather than by the accumulated content turning onto it\[[41](https://arxiv.org/html/2609.10210#bib.bib14)\]\. Equation[6](https://arxiv.org/html/2609.10210#S2.E6)is read in the right basis by construction: it is evaluated*at*the readout, and the unembedding is the frame the readout is written in\. Anything read earlier is not\. Ask what a mid\-stack component passes to the component that reads it\. Score the writer through the unembedding and the reader through its own layer’s basis and the answer is nothing, a correlation of−0\.02\-0\.02against a random control, which reads as*the graph does not link*\. Score both sides in the reader’s own frame and the same edges correlate\+0\.28\+0\.28and beat a size\-matched random control on7676percent of cases \(Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)\)\. The two halves had been scored in different bases, and a null produced by the wrong frame looks exactly like a null\.

The lens we construct uses the*layer\-native token table*\. Every reading taken between layers depends on it\. Fix a positionppand a set of contexts𝒞\\mathcal\{C\}drawn from held\-out text\. A context is one sequence at the model’s training length, andppis a single position inside it — the same position in every context — so a context contributes one measurement rather than one per token\. For a candidate tokentt, letz\(l\)​\(c,t\)z^\{\(l\)\}\(c,t\)be the input to layerll’s feed\-forward block at positionppwhenttis substituted there in contextcc\. The table is the average of that state over contexts,

E~​\[l,t\]=1\|𝒞\|​∑c∈𝒞z\(l\)​\(c,t\)∈ℝd,\\tilde\{E\}\[l,t\]\\;=\\;\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}z^\{\(l\)\}\(c,t\)\\;\\in\\;\\mathbb\{R\}^\{d\},\(7\)which is the layer’s own picture of the token, in the layer’s own frame\. One row per token per layer, computed once in a single pass, with nothing fitted and no per\-input work\.

That it is measured rather than fitted is the point\. A tuned lens\[[6](https://arxiv.org/html/2609.10210#bib.bib2)\]would also supply a per\-depth frame, but it is trained, and then understanding the explainer becomes part of understanding the model\.E~\\tilde\{E\}is read off the model’s own forward passes, so a reading taken through it is a derivation rather than a second model’s opinion\. Where this paper decodes anything below the readout — what a component fires on, what an edge carries, what a mid\-stack write means to the layer that consumes it — it is scored againstE~\\tilde\{E\}at that layer\.

### 2\.3Identifying components that matter

The ninety percent used throughout is a bar on contribution mass\. A set clears it when its members account for nine tenths of the signed total arriving at the readout\. How much of a prediction such a set actually carries is a separate measurement, made in Section[2\.4](https://arxiv.org/html/2609.10210#S2.SS4)\.

Ranking components byκu\\kappa\_\{u\}and asking how many are needed to reach ninety percent of the total requires choosing what the total is\. Three choices are available:

n90net:∑i≤nκ\(i\)≥0\.9​N,n90pos:…≥0\.9​∑umax⁡\(κu,0\),n90abs:…≥0\.9​∑u\|κu\|\.n\_\{90\}^\{\\text\{net\}\}:\\textstyle\\sum\_\{i\\leq n\}\\kappa\_\{\(i\)\}\\geq 0\.9N,\\qquad n\_\{90\}^\{\\text\{pos\}\}:\\ldots\\geq 0\.9\\textstyle\\sum\_\{u\}\\max\(\\kappa\_\{u\},0\),\\qquad n\_\{90\}^\{\\text\{abs\}\}:\\ldots\\geq 0\.9\\textstyle\\sum\_\{u\}\|\\kappa\_\{u\}\|\.\(8\)Contributions cancel heavily, so the three denominators differ by orders of magnitude\. Table[3](https://arxiv.org/html/2609.10210#S2.T3)measures the consequence on the same predictions with the same attribution: keeping the sign and dividing by the net returns tens of components, while discarding the sign returns tens of thousands\. The opposing mass is twenty times the net at the median on this model, seven across the eighteen measured here, and it is spread thin: reaching ninety percent of the prediction takes a quarter of every component in the model under the positives\-only denominator, and half of them under the absolute one\. It is not a rival circuit that the net denominator hides; Appendix[C\.8](https://arxiv.org/html/2609.10210#A3.SS8)tests that against its floor\.

Table 3:Components needed to reach ninety percent of a prediction, under the three denominators of Equation[8](https://arxiv.org/html/2609.10210#S2.E8)\. Only the denominator changes\. Median over 115 predictions, each the model’s own top\-1 token with probability at least 0\.3\. The two rightmost conventions are the ones in common use; the same comparison across twelve models trained by other people is Table[35](https://arxiv.org/html/2609.10210#S6.T35)\.Figure 1:Left: the same counts on a log axis\. The choice of denominator moves the answer by two to three orders of magnitude, which is larger than any difference between the models being compared\. Right: circuit size against how confident the prediction is\. Confident predictions rest on fewer components, so a circuit size quoted without the confidence of the predictions it was measured on is not comparable across papers\. Every other number in this section is measured at probability≥0\.3\\geq 0\.3\.The count is insensitive to where the bar is set\. On the same ranking the median count at eighty percent is0\.750\.75to0\.820\.82of the count at ninety, and at ninety\-nine percent it is1\.161\.16to1\.251\.25, across the five models trained here it was run on\. Nearly the whole usable range of the bar moves the answer by less than a factor of two\. The tail is enormous counted absolutely and nets to almost nothing, so raising the bar finds few components left to add\.

### 2\.4What part of a transformer is necessary

The accounting names a set\. We call it the one\-hop set, because every member is credited for writing to the output directly rather than through anything else\. Whether that set is the reason for the prediction is settled by removing components and re\-running the model\. Remove the topkkat the position being predicted; Figure[2](https://arxiv.org/html/2609.10210#S2.F2)reports how often the prediction changes; the control removes the same*number*of components, chosen at random\.

Figure 2:Removing thekkhighest\-contributing components changes the prediction, where removingkkrandom components leaves it alone\. Bands are bootstrap 95 percent intervals over predictions\. The random control stays at or below 5 percent across the whole range, while byk=128k=128the identified components have changed nearly every prediction\. The baseline is plotted here; Figure[29](https://arxiv.org/html/2609.10210#S6.F29)shows the same curves for the twelve models trained by other people\.Few components are needed in any model we measured, which is a statement about transformers rather than about anything built here; Section[6](https://arxiv.org/html/2609.10210#S6)makes that case across twelve models trained by other people\.

Removing a set is not the same as adding up its contributions\. Equation[6](https://arxiv.org/html/2609.10210#S2.E6)is linear, so it predicts exactly what a removal should cost: the sum of the contributions taken away\. Perform the removal and the*target logit*falls by only4242to5050percent of that \(Table[4](https://arxiv.org/html/2609.10210#S2.T4)\)\. Both quantities are logit mass, and neither is a rate at which the prediction changes; that is Figure[2](https://arxiv.org/html/2609.10210#S2.F2), and a removal can take half the predicted logit away and still leave the argmax where it was\. The shortfall is made up by the components left behind, which compute something different once the removed ones are gone — the self\-repair reported by[McGrath et al\. \[35\]](https://arxiv.org/html/2609.10210#bib.bib27)and[Rushing and Nanda \[46\]](https://arxiv.org/html/2609.10210#bib.bib28)\. Ablation is therefore a second measurement rather than arithmetic on the first\.

Table 4:Removal cost against predicted cost, both measured in logits\. Each entry is the realized drop in the target logit divided by the sum of the contributions of the components removed, so1\.001\.00would mean the model absorbed the removal exactly as the linear attribution predicts and the removal test would add nothing\. About half the predicted loss is instead recovered by the components left behind\. This is a statement about logit mass rather than accuracy, which is Figure[2](https://arxiv.org/html/2609.10210#S2.F2)\.##### What to measure when a component is removed\.

Removing components shrinks the residual stream\. The final norm then divides by a smaller quantity, so every logit inflates together, and a drop in the target’s logit mixes two effects: what happened to that token, and a shift applied to the whole vocabulary\. Only the first is about the prediction, so damage has to be read off something a uniform shift cannot move\.

We use the*margin*: the target logit minus its strongest competitor\. A shift common to every logit cancels from that difference, and the margin falls monotonically as more components are removed, which any measure of damage has to do\. It also joins two readings that could otherwise disagree — whether the prediction flipped, and how much damage was done — because a flip*is*the margin reaching zero\. The change in log probability is equally invariant, and is used where a likelihood is the natural reading\.

Anything read off the argmax was never exposed to this, because a constant added to every logit cannot change which entry is largest: the flip curves, the minimality search, the sufficiency numbers andn90n\_\{90\}itself are all unaffected\. On this model the choice barely matters, and Table[5](https://arxiv.org/html/2609.10210#S2.T5)shows the raw drop and the margin disagreeing about the direction of the effect in three percent of cases at the mediankk\. It matters elsewhere\. Measured in raw logits, the ratio of realized to predicted effect comes out*negative*on GPT\-2, which is not a possible value\. Section[6\.5](https://arxiv.org/html/2609.10210#S6.SS5)reports raw drops disagreeing with the margin up to forty\-five percent of the time and falling askkgrows\.

Table 5:The two quantities on the baseline\. “Sign disagreement” is how often the raw logit drop and the margin disagree about the direction of the effect, at the median and worstkk; “non\-monotone steps” counts the times the median drop*falls*askkgrows, which a measure of damage should never do\. Here they agree, with no non\-monotone steps\. Table[43](https://arxiv.org/html/2609.10210#S6.T43)is the same table on models where the raw quantity fails\.
##### What the removal costs elsewhere\.

Any sufficiently large perturbation lowers a logit, so the effect on the target has to be read against the collateral\. Define specificity as the drop in the target logit divided by the mean absolute change across the rest of the vocabulary\. Table[6](https://arxiv.org/html/2609.10210#S2.T6)compares three rankings\.

Table 6:Specificity: target\-logit drop divided by mean absolute change elsewhere\. Ranking by absolute contribution selects large negative contributors, and removing them can*raise*the target logit, which is why that ranking scores worst\. Ranking by activation magnitude selects high\-norm components whose removal disturbs the whole distribution\. The net ranking is the most specific at everykk, by a margin over the alternatives that widens askkgrows\.
##### The required graph\.

The required graph is the smallest prefix of the one\-hop set, ordered by contribution, whose removal changes the prediction\. Every required\-graph number in this paper is found by scanning prefix lengths upward and stopping at the first that flips\. Appendix[C\.6](https://arxiv.org/html/2609.10210#A3.SS6)covers the flaws in the more attractive\-looking alternatives\. On some predictions no prefix flips at all, so the quantity is undefined there; the proportion is reported with each table, and medians are taken over the predictions where it exists\.

### 2\.5What part of a transformer is sufficient

Necessity asks what a prediction cannot survive losing\. The opposite question is what it takes to*build*it: keep a set of components, zero every other feed\-forward unit and attention channel at that position, let the survivors recompute from what is left, and ask whether the model still emits the same token\. Nothing is spliced in\. A component that is kept fires from the residual that remains, which is the token embedding plus whatever the other survivors have written\.

##### The choice of replacement value\.

A transformer has no null value for a component’s output, so an ablation must write something in its place, and three conventions are in common use: zero, the component’s mean output over a sample of inputs, and a value resampled from a different input\. Nothing in the architecture prefers one\. For the necessity measurement above the choice barely matters — removing the same components under all three changes the same predictions to within seven points of each other, because a large signed contribution is gone whichever value is written where it was\. It matters a great deal for what follows, and zero is the choice made here; the alternatives are tested below and in Appendix[D\.5](https://arxiv.org/html/2609.10210#A4.SS5)\.

Finding such a set is a search, and its shape follows from why a kept set fails: some member is starved, its drivers among the components that were zeroed, and the repair is to put those drivers back\. Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3)gives the procedure — seed with the required graph, repair starved members from their suppliers until the model emits the token, then cut every member the set can do without — with no free parameters and a stopping rule the model supplies\.

Figure[4](https://arxiv.org/html/2609.10210#S2.F4)is one such set drawn out, with what each member fires on and what it writes toward\.

Algorithm 1The smallest set that builds a prediction, at positionppfor tokentt\.

Figure 3:Every line is a forward pass of the model scored against its own answer, with nothing fitted and nothing spliced in\. Step 2 is what separates this from a search over the contribution ranking: it uses the dependency structure to say*which*components are missing rather than trying prefixes until one works, and it stops when the model emits the token rather than when a budget runs out\. Step 3 scores many candidates in one pass\.Table 7:The smallest set that reproduces a prediction, and what the same number of components does when chosen differently\. “Converged” is how many of the sampled predictions the procedure verified\. A prediction is dropped when the repair step of Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3)reaches its round limit without the kept set becoming sufficient, which is a limit of the procedure rather than a fact about the model and, since the repair has typically built hundreds of components by then, evidence that the criterion is not one that volume satisfies\. “Floor” is the share of predictions on which keeping*nothing*at the position — so that the residual is the token embedding alone — already returns the same token\. No set kept on top of that can be credited with producing it, and the search returns a set of size one on every one of them, so those predictions are counted here and then set aside\. The size and the two controls are medians over what remains after both exclusions\. The last column is the median set as a fraction of the model’s components\.same size, other setsmodelpredsfloorconvergedsizetop\-kkrandom% of modelbaseline \(GELU\)15012%11587%0%0\.017%Eight components reproduce the prediction, out of46,08046\{,\}080\(Table[7](https://arxiv.org/html/2609.10210#S2.T7)\)\. That is under two parts in ten thousand of the model, and it is not an artifact of the test\. Zeroing everything else also deletes the competition, so the floor has to be measured first and then excluded; on this model it is1212percent of predictions\. On the predictions that remain, eight components drawn at random reproduce the token on*none*of them — not one of115115— and the eight highest\-contributing on77percent, against the search’s hundred\.

Two checks say a set is not right for the wrong reasons\. Adding components does not satisfy the criterion: where the repair never converges it has already built a mean of489489components, at most962962or two percent of the model, and the token still does not come out\. And the sets are conditioned on their context rather than carrying their token — kept at a position in a different context whose intact answer is a different token, seven percent still emit the original and eighty\-nine percent emit neither it nor the new one\.

The set does not depend on where the search starts, and it is not the only one\. Starting the search from a random draw of components the size of the required graph, instead of from the required graph itself, and running the same repair and cut reaches a sufficient set on8585percent of predictions against8989, at the same median size of six on that draw, for about half as much work again\. Where both starts succeed, the set found from the random start and the set found from the required graph overlap at a Jaccard of0\.310\.31, against0\.00040\.0004for two random sets of the same sizes, and roughly half the pairs share at least half their members\. A fifth share nothing\. That is what an overdetermined prediction should give, and the repair reports the same thing on its own on half the predictions it succeeds on: the model holds more than one set of components that can carry the answer alone\. Section[6\.3](https://arxiv.org/html/2609.10210#S6.SS3)reports the same measurement on the models trained elsewhere\.

The search uses the answer: every step asks whether the*known*token survives, so what it establishes is that such a set exists\. It is not a procedure that predicts one in advance, which is worth pursuing and not attempted here\. And the set is sufficient at one position, with the rest of the sequence computing normally, so any attention channels among its members read ordinary values from the positions around them\. What the measurement licenses is that eight of the model’s components need to be active*here*for this token, not that eight components are all that are needed for the entire transformer to function\.

The result does not depend on zeroing\. Zero is one choice of what to write where a removed component was, and the objection it invites is that a model with most of a position blanked is nowhere it has ever been\. Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)supplies a replacement that keeps it on\-distribution: the layer\-to\-layer map, which is what the removed components would have delivered if they were doing only carriage\. Run under both backgrounds, the prefix\-and\-prune search of Figure[4](https://arxiv.org/html/2609.10210#S2.F4)returns a median of1818components against the map and1616against zeros\. The pair is to be read against each other rather than against the88of Table[7](https://arxiv.org/html/2609.10210#S2.T7), which is a different search on a draw excluding a different floor\. The carriage \(described below\) alone predicts the token on a quarter of predictions, and those are excluded here for the same reason the token embedding’s own successes are: a prediction the background alone already makes is not evidence about any set\. A small set suffices whether the space around it is emptied or filled with the traffic the architecture would normally have put there\.

Every size reported here is an upper bound returned by a heuristic search, leaving the minimal sufficient circuit open\. What the sizes establish is that sufficient circuits orders of magnitude smaller than the published counts can be located at all\. The cost of that search grows with the model\. On the twelve\-layer models it converges in a few dozen forward passes\. On a twenty\-four\-layer model most predictions still converge, and a minority exhaust a budget an order of magnitude larger and return a bound rather than an answer\. Finding the circuits in a very large model is a search problem, and scaling that search is left for later work\.

Figure 4:One prediction’s sufficient set\. Keeping these eighteen components and zeroing every other feed\-forward unit and attention channel at this position leaves the model predicting the same token;n90n\_\{90\}is4242of the model’s46,08046\{,\}080components, and seven of the eighteen are attention channels\. Bold marks the one component whose*removal*from the intact model changes the prediction: an attention channel at layer one, feeding a unit at layer ten, feeding the unit at layer eleven that promotes the predicted token first among the whole vocabulary\. Rows are the members nearest the readout first, with what each draws on indented beneath it and its share of that member’s incoming drive; the last column is where the predicted token sits in what the component promotes\. In the graph, a member that writes to the answer has an edge to it, labeled with the rank the predicted token holds in what that member promotes\. Appendix[A](https://arxiv.org/html/2609.10210#A1)has the full key\.
##### What the surviving sets look like\.

They are small graphs rather than bags, as Figure[4](https://arxiv.org/html/2609.10210#S2.F4)shows: its members sit at five depths, several draw on one another, and the seven attention channels among them are the only route by which anything outside the position reaches it\. Between two thirds and three quarters of the members are feed\-forward units and the rest are attention channels, spanning about five layers, and most feed\-forward members draw part of their drive from other members — though only about a tenth of it at the median, the rest arriving from components outside the set entirely\.

##### Do models say “not this token” in a circuit?

A prediction can be reached by pushing the answer up or by pushing its rivals down, and the accounting so far counts only the first\. Asking about the second needs positions where the model faced two candidates\. Pairs of positions that share their last four tokens and end in different confident predictions,A→XA\\rightarrow XandB→YB\\rightarrow Y, are such positions, since what separates them is what attention carried in from earlier context\. There are a hundred per model, from held\-out text at the training length\. Where a token wins, its net\-denominatorn90n\_\{90\}is tens of components; where it loses, it still carries a circuit of the same order and keeps5858to7373percent of its push\. Its promoters overlap across the two contexts at a Jaccard of0\.280\.28to0\.450\.45, and the winner’s and the loser’s promoters overlap at0\.140\.14to0\.330\.33\(Table[8](https://arxiv.org/html/2609.10210#S2.T8)\)\. The loser is not pushed down: its net push stays positive on at least9595percent of pairs\. It is out\-pushed, by the winner’s circuit firing harder in the winner’s context, by a median of three to five logits\. We looked for circuits that say “not this,” components whose removal would hand the prediction to the token they hold down, and they are not there: the mass pushing the loser down is no different from any other token’s \(Appendix[C\.8](https://arxiv.org/html/2609.10210#A3.SS8)\)\. Removing eight of the winner’s promoters moves the prediction on3030to7272percent of pairs; removing eight of the loser’s suppressors moves it to the loser on88to2626percent, and would do the same for any token\. The baseline’s other converged seeds fall within the ranges quoted, with one exception taken up in Appendix[C](https://arxiv.org/html/2609.10210#A3)\.

Table 8:Two answers at one position\. On a hundred contrastive pairs per model, two positions sharing their last four tokens and ending in different confident predictions: the net\-denominatorn90n\_\{90\}of the winner where it wins and of the loser both where it loses and where, at the other position, it wins; the share of the loser’s winning\-context push its top sixteen promoters still deliver where it loses; and the Jaccard overlap of the loser’s top sixteen promoters across the two contexts, and of the winner’s with the loser’s at the position the winner takes\. Medians over pairs\.Three numbers now sit on top of one another, answering three questions about the same prediction\. The signed accounting says which components*carry*it: fifty\-three at the median, read off the parameters and activations with nothing removed\. Ablation says which of those it*cannot lose*: thirteen\. And the search says which set*builds*it: eight\. Fewer than a sixth of the components carrying ninety percent of the logit are needed to reproduce the token\.

### 2\.6What the rest of the model is doing

A prediction resting on a few dozen components leaves an obvious question: what is everything else doing? “Nothing” is not credible in a trained network\. Three candidates are tested in turn: staging for later tokens, effort on rival answers, and the architecture’s own overhead\. If the remainder were staging information for*later*tokens, then removing everything at a position except that position’s circuit would spare the token being predicted and damage the ones that follow, which attend back to it\. Table[9](https://arxiv.org/html/2609.10210#S2.T9)tests exactly that and finds no such staging\. Keeping only the circuit costs a quarter as much at the token itself as keeping an equally sized random set, which is the necessity result restated\. On the tokens that follow, it costs the same as the random set, and it goes on costing the same however far out one looks: the median per\-prediction ratio of the two costs runs0\.980\.98to0\.990\.99from sixteen tokens to five hundred and twelve, with no trend in distance \(Table[10](https://arxiv.org/html/2609.10210#S2.T10)\)\. Whatever the rest of the model is doing at this position, it is not principally holding things for the future, and the window it is not holding them for is the whole context\. Section[6\.6](https://arxiv.org/html/2609.10210#S6.SS6)reports the same sweep on nine further models\.

Table 9:Everything at a position is removed except the set named\. Cost at this token is the fall in the margin; cost on the next sixteen is the rise in their cross\-entropy\. Keeping the circuit beats keeping a random set for the token being predicted alone, matching it on the tokens after\. The two later\-token medians are taken separately, so their ratio differs from the median per\-prediction ratio quoted in the text\.Table 10:The same test at six horizons, nested on one set of positions so the horizon is the only thing varying\. Each entry is the median over predictions of one prediction’s circuit cost divided by its random\-set cost, so1\.001\.00means the circuit is worth no more than an equally sized random set for the tokens that follow\. “At it” is the same ratio at the predicted token, where a low number is the necessity result\.tokens after the predictionmodelpredsat it163264128256512baseline \(GELU\)2060\.250\.980\.980\.990\.990\.990\.98A second candidate is that the remainder is spent on near misses\. Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)found that where a token wins, the token that lost at that position still carries a circuit of tens of components and keeps most of its push, and the same records say how much of the model’s pushing goes to candidates that do not win\. At the position where a token wins with net pushNN, the token that lost there carries0\.630\.63to0\.840\.84ofNNacross the six models of Table[8](https://arxiv.org/html/2609.10210#S2.T8), a competitor drawn from the model’s top fifty carries0\.430\.43to0\.790\.79, and a random vocabulary token carries none\. Counted as positive mass rather than net, the runner\-up draws on0\.940\.94to0\.990\.99of what the winner draws on and a top\-fifty competitor on0\.880\.88to0\.980\.98\. The model promotes its near misses nearly as hard as its answer, each with a circuit of the winner’s order:2626to4848components on the four smaller models, one to two hundred on Gemma\-3 and Qwen\. That is effort of the same kind as the answer’s, and it accounts for a few dozen components per candidate, not for the tens of thousands whose pushes cancel\.

The remaining candidate is that the remainder is the architecture’s own overhead: moving information between layers, and keeping a superposed residual stream organized so that many concepts can share one vector space\[[17](https://arxiv.org/html/2609.10210#bib.bib5)\]\. Work of that kind has to happen somewhere, and it happens*everywhere*\. It has no reason to point at the token being predicted, and shows up as an enormous signed mass, spread over tens of thousands of components, that nets to little at the readout\. That overhead has a known shape\.[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14)report a transformer’s residual stream turning as a near\-rigid body from depth to depth, with the prediction arriving on the readout in the last few layers by being written onto it\. What follows is an independent check on that picture, made with a different instrument and for a different purpose — not the geometry of the stream, but how much of a layer’s work a fixed map of the incoming state can account for\. It confirms the published result and adds two things to it: a size for the reshaping, and a test of whether the map can stand in for the components\.

Writex\(ℓ\)x^\{\(\\ell\)\}for the residual entering layerℓ\\ellat the position being read, so that the layer’s whole contribution there is

Δ\(ℓ\)=x\(ℓ\+1\)−x\(ℓ\),\\Delta^\{\(\\ell\)\}\\;=\\;x^\{\(\\ell\+1\)\}\-x^\{\(\\ell\)\},\(9\)everything the attention and feed\-forward blocks write into the stream at that depth\. Carriage means the part of that update which is the same transformation applied to whatever happens to be there, so it is what a*fixed*map of the incoming state can account for\. Fit that map over a sample of positions𝒫\\mathcal\{P\}and score it on a disjoint sample𝒫′\\mathcal\{P\}^\{\\prime\}:

Aℓ=arg⁡min⁡∑p∈𝒫A⁡‖xp\(ℓ\)​A−xp\(ℓ\+1\)‖2,Δ^\(ℓ\)=x\(ℓ\)​\(Aℓ−I\),A\_\{\\ell\}\\;=\\;\\arg\\min\_\{A\}\\sum\_\{p\\in\\mathcal\{P\}\}\\big\\\|x^\{\(\\ell\)\}\_\{p\}A\-x^\{\(\\ell\+1\)\}\_\{p\}\\big\\\|^\{2\},\\qquad\\hat\{\\Delta\}^\{\(\\ell\)\}\\;=\\;x^\{\(\\ell\)\}\\big\(A\_\{\\ell\}\-I\\big\),\(10\)with a ridge term for conditioning\. The share of the update the map accounts for is the part ofΔ\\Deltalying*along*Δ^\\hat\{\\Delta\}, which is bounded by one however large the prediction is:

sℓ=\(∑p∈𝒫′⟨Δp,Δ^p/‖Δ^p‖⟩2∑p∈𝒫′‖Δp‖2\)1/2\.s\_\{\\ell\}\\;=\\;\\Bigg\(\\frac\{\\sum\_\{p\\in\\mathcal\{P\}^\{\\prime\}\}\\big\\langle\\Delta\_\{p\},\\;\\hat\{\\Delta\}\_\{p\}/\\\|\\hat\{\\Delta\}\_\{p\}\\\|\\big\\rangle^\{2\}\}\{\\sum\_\{p\\in\\mathcal\{P\}^\{\\prime\}\}\\\|\\Delta\_\{p\}\\\|^\{2\}\}\\Bigg\)^\{1/2\}\.\(11\)A ratio of norms is not so bounded and will exceed one when the map’s prediction is large and misaligned\. RestrictingAAto a rotation gives the rigid variant, by orthogonal Procrustes on the same samples:Qℓ=U​V⊤Q\_\{\\ell\}=UV^\{\\\!\\top\}whereX⊤​Y=U​Σ​V⊤X^\{\\\!\\top\}Y=U\\Sigma V^\{\\\!\\top\}\. A rotation preserves every length and angle, sosℓ​\(Aℓ\)−sℓ​\(Qℓ\)s\_\{\\ell\}\(A\_\{\\ell\}\)\-s\_\{\\ell\}\(Q\_\{\\ell\}\)is the shear — the part of the map that reshapes the stream rather than turning it\.

Measured this way, most of what a layer writes is carriage\. A fixed map of the incoming state accounts for7474percent of the update on the baseline and7979on the sigmoid model, with a positive held\-outR2R^\{2\}at every depth, and a rotation alone accounts for the bulk of that \(Table[11](https://arxiv.org/html/2609.10210#S2.T11)\)\. That much is the near\-rigid body of[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14), recovered here without appeal to any frame\.

What is left over is where this measurement goes past the published one\. The stream is reshaped between depths and not merely reoriented, and the reshaping is concentrated at the entrance: the first layer shears the stream three to five times as much as any layer after it\.

Table 11:How much of a layer’s update a fixed map of the incoming state accounts for, as a median over layers\. “Rigid part” restricts that map to a rotation, which preserves every length and angle; the shear is what the full map explains beyond it, and is the median of the per\-layer difference rather than the difference of the two medians beside it, so those columns are not meant to subtract\. The shear is concentrated in the first layer in both models\.Whether the map can stand in for the components is tested by substitution\. We run the model with a layer’s own output replaced by what the map predicts, and with the actual writes of a chosen setSSadded back on top:

x~\(ℓ\+1\)=x\(ℓ\)​Aℓ\+∑u∈S∩ℓau​cu,\\tilde\{x\}^\{\(\\ell\+1\)\}\\;=\\;x^\{\(\\ell\)\}A\_\{\\ell\}\\;\+\\;\\sum\_\{u\\in S\\cap\\ell\}a\_\{u\}c\_\{u\},\(12\)applied at the position being read while the rest of the sequence computes normally, exactly as in the ablations above\. WithSSempty and one layer substituted, the model still produces the same token on a median of9292percent of predictions\.

One part of what the accounting sets aside is not carriage but supply\. A member of a circuit has inputs of its own, written by components upstream, and those are causally necessary to it: remove a node’s attributed sources and its activation falls by two thirds, where removing as many random upstream components leaves it unchanged \(Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)\)\. The one\-hop accounting cannot see them, because it credits a component only for what its own write lands on the readout and never for what a later component reads from it\. The sufficiency search of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)recovers them by repair, and the closure of Appendix[D](https://arxiv.org/html/2609.10210#A4)recovers all of them, at one to three percent of the model\. The cancellation that governs the readout governs a component’s input too, and more severely: reaching ninety percent of a node’s incoming drive takes about a dozen sources counted with sign and many thousands counted without, a ratio near a thousand against about twenty at the readout, which is why the branching figures of Appendix[D](https://arxiv.org/html/2609.10210#A4)are computed signed\. The bar of ninety percent is a choice, and Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)tests it: at the readout the count moves by less than a factor of two between eighty and ninety\-nine percent, because the tail is enormous counted absolutely and nets to almost nothing\. A bar at one hundred percent, the first count at which the signed sum reaches the net, is barely larger, and what it never does is draw in the thousands of sources that cancel, since a signed sum reaches the net without them\. Raising the bar buys a somewhat larger closure, still a graph of overlapping backward trees at a few percent of the model, not the model\.

### 2\.7Limitations

Five qualifications travel with every number in this section\.

The first is that credit is direct\. A component is scored by what its own write places on the readout, so one that acts only by changing what a later component computes never enters the ranking, however much the prediction turns on it\. This reaches further than the accounting: the sufficient sets of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)are grown from candidates the same equation nominates, so a component that acts only through others cannot enter one of those either, however much it matters\. The recursion of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)recovers such components as sources, which is much of why a closure is larger than the set it closes over\.

The second is that the unit is a choice\. An attention head is sixty\-four channels, and counting the head rather than the channel cutsn90n\_\{90\}by more than half while lifting attention’s apparent share by two thirds \(Table[12](https://arxiv.org/html/2609.10210#S2.T12)\)\. Channels are used throughout because they put attention and feed\-forward components on the same footing; the per\-head figures are given so the comparison with work that counts heads is available\.

Table 12:The one\-hop accounting with attention counted per channel and per head\. The result survives the coarser unit, though the numbers do not transfer between the two conventions\.The third is that late layers are over\-credited\. A component is a neuron activation times its write column, and late layers project more directly onto the readout, so direct contribution flatters them\. Comparing attributed mass against causal effect on this model, the later half of the stack is credited with about twice the direct contribution of the earlier half while removing its topkkchanges barely more predictions — atk=16k=16, thirty\-two percent against twenty\-nine\. The attribution over\-credits late layers by a factor of0\.530\.53atk=16k=16and0\.650\.65atk=64k=64\. Section[6](https://arxiv.org/html/2609.10210#S6)reports the same measurement on models trained by other people, where it does not always come out this way\.

The fourth is that the prediction is a choice\. Circuit size falls with the model’s confidence in its own output \(Figure[1](https://arxiv.org/html/2609.10210#S2.F1), right\), so these figures describe confident predictions, and a number quoted at a different confidence floor describes a different population\.

The fifth is that a collapsed model reads as maximally legible\. The worst excluded baseline seed sits far below its siblings on LAMBADA \(Table[1](https://arxiv.org/html/2609.10210#S2.T1)\) and reads as perfectly legible:n90=1n\_\{90\}=1, a single component flips the prediction, and attention accounts for all of it\. Seeds that merely miss the quality bar read normally: across the eight seeds of the sigmoid construction of Section[5](https://arxiv.org/html/2609.10210#S5), the three that failed the screen span the samen90n\_\{90\}, the same attention share and the same depth as the five that passed\. The quality screen therefore travels beside every figure here\.

## 3What’s in a Name?

Section[2](https://arxiv.org/html/2609.10210#S2)showed that a prediction rests on a few dozen components and that they can be found without training anything\. That does not say what any of them*is*\. This section asks the second question, of a unit and of a channel, and of what each writes and what each reads\. The two sides are named by different means: what a component writes is read off the predictions it drives, and what it reads is decoded from its parameters in the frame of its own layer\.

### 3\.1What a unit writes

Section[2](https://arxiv.org/html/2609.10210#S2)already answers which components carry a prediction: the accounting set is the components whose direct contribution reaches ninety percent of the net, and the shares that define it are the same shares used here\. Those components wrote that token, by construction rather than by inference, since they are selected by their contribution to it\. So a unit can be named without reading its column weights at all\. Take a prediction, take its strongest contributors, and repeat until the model is covered\.

Doing that costs little\. A million and a half predictions take minutes, and each one supplies a few dozen memberships\. What comes back has to be reported in three parts \(Table[13](https://arxiv.org/html/2609.10210#S3.T13)\)\. A component is*covered*when it appears among the strongest contributors to any prediction at all; the loosest statement, reaching most of the model\. It is*characterized*when it appears there twenty times or more, which is when there is enough to say what it writes\. And its set of tokens is*coherent*when the set holds together against a frequency\-matched null, judged in a distributional space built from the training corpus, so the yardstick is the data the model was trained on rather than another model’s geometry\. Taken together, close to half of the components in a model have a nameable write\.

Table 13:Naming components by the predictions they drive, on the baseline over1\.51\.5million predictions\.*Covered*is appearing among the top\-kkcontributors to any prediction,*characterized*is appearing there twenty times or more, and*coherent*is the share of those whose token set beats a frequency\-matched null in a distributional space built from the training corpus\. Every share is of all46,08046\{,\}080components\. Wideningkktrades a little coherence for a great deal of coverage, which is why the last row and not the first is the operating point\.Components write a closely related set of tokens rather than one exact token, with the relation within a set grammatical rather than semantic\. A unit promotes negated auxiliaries, or first names, or units of measurement, or the tokens that open a sentence — categories a reader can state in a phrase, and not the topical groupings that the word*concept*usually suggests\. Only a tenth of one percent of components write a single token\. Weighted by how much each token receives rather than counted, a third of components put half their contribution on a single token, so the two ways of asking give different answers\. The breadth is visible on the page\. Every node in Appendix[A](https://arxiv.org/html/2609.10210#A1)carries the list of tokens it writes toward, and the token being predicted is one entry among several — so a component that helps deliver an answer is seen promoting it alongside its neighbors rather than singling it out\.

A component supplies a stable share: across the contexts where a component and a token appear together, its coefficient of variation has a median of0\.240\.24, and two thirds of pairs sit below0\.30\.3\. So the amount belongs to the component rather than to the context\. That number can be checked by deleting the component and watching what the prediction loses, which is a different measurement from the attribution that produced it\. The two agree atr=0\.60r=0\.60and the relation is monotone: components credited with under five percent of the drive cost about a third of a logit when removed, those credited with five to fifteen percent cost four fifths of one, and those above fifteen percent cost more than two\. A name of the form*this supplies a seventh of the drive toward*‘dogs’ is therefore a claim about the model\.

Components nameable this way are late in the model\. In the baseline’s feed\-forward blocks the characterized share runs at one or two percent through the first layers, reaches a quarter by layer six, half by layer eight, and ninety\-five percent at the last; attention lags by about two layers and then catches up\. Coherence does not move with depth, sitting between eighty and ninety\-nine percent everywhere, so depth changes how many components can be named rather than how well\. What does change is breadth: two tokens carry half a component’s contribution in the middle of the stack and nine do at the last layer, which is the same distributed commitment seen in Figure[5](https://arxiv.org/html/2609.10210#S3.F5), arriving here as a property of the names\.

Three other measurements put the same event at the same depth\. The residual first carries the token the model will*predict*at layer eight: the token*at*the position starts at\+0\.45\+0\.45and is gone within a single layer, while the one it emits sits below zero through the body of the stack\. Write columns first clear chance there\. And a component’s own ranking of the token it helps predict lifts there — score a component by whether that token lands in the top hundred of what it promotes, and the last quarter of the stack clears its own floor by five to eleven times against the middle’s one and a third to five and a half, higher in the last quarter than in the middle in every model measured \(Figure[6](https://arxiv.org/html/2609.10210#S3.F6)\)\.111The floor of this measurement is set by the readout’s geometry — tokens with large centered rows sit near the top of almost any column — and runs from half a percent in one model to thirteen in another, which is why the figure reports multiples of each model’s own floor\. Appendix[C\.11](https://arxiv.org/html/2609.10210#A3.SS11)takes the readout apart; the depth result survives that correction, the floor halving and the margin over it widening\.

A name says which tokens a component writes toward, leaving the sum to pick one of them\. Components write toward the result and rarely at it\. The predicted token sits inside the top hundred of two thirds of the weighted contribution, and inside the top thousand of ninety\-three percent of it, so no member selects the answer on its own\. Individually each is an aimed near miss\. Score a contributing component’s own write, scaled by how it fired, against the direction the readout would need: a typical member sits at0\.070\.07, which is far too small for a maximum over the vocabulary to register and still carries aim, because the same vectors score about zero against a different prediction’s token\. Adding them in order of contribution takes the alignment to0\.450\.45, three times the best single member \(Figure[5](https://arxiv.org/html/2609.10210#S3.F5)\)\. The climb is slower than independent averaging would give, and not because the misses share a direction — measured directly they are close to orthogonal, at a mean pairwise cosine of0\.0040\.004\. It is that the members are unequal: the strongest contribution is twice as well aimed as a typical one,0\.1560\.156against0\.0710\.071, so the sum is carried by its largest terms and each additional weak member buys less than the last\.

Figure 5:How a prediction is assembled\. The cosine here is taken against a*known*target rather than searched for over the vocabulary, which is what makes it answerable where the column measures of Appendix[C\.11](https://arxiv.org/html/2609.10210#A3.SS11)are not\. Left: the cosine between the target’s readout direction and the running sum of what its components write, added strongest contribution first; the dashed line is the climb independent misses would give, anchored on each model’s own first member\. Every model falls below it, so the sum is carried by its largest terms rather than by their number\. Right: the control, the same partial sums scored against a different prediction’s token\. Flat and near zero for every model plotted, so what the left panel shows is aim rather than undirected drift toward the token cone\.Figure 6:A component*names its own token*when the token the model predicts lands inside the top hundred of its write column’s ordering over the vocabulary, signed by how the component fired\. Left: the share by relative depth, one line per model, with each model’s own measured floor dotted\. Right: the middle half of the stack and the last quarter, as multiples of that floor, which is the only form in which models can be set beside one another\. Rank exists only for components appearing in a trace, so this is conditioned on a component mattering to some prediction\.
### 3\.2What a unit reads

A unit’s parameters name the tokens that drive it, and the name can be checked against the model rather than against whether the list looks sensible\. Asked for a unit’s eight strongest drivers, the baseline’s parameters read through the layer basis described below recover the true set at58\.958\.9percent, thirty\-seven times chance — the strongest agreement between a component’s weights and its behavior anywhere in this paper\. How far down the list that holds is answered below as well\.

A read row is a covector rather than a vector in the residual stream: it consumes states and returns a number, expressed in the frame of the layer that hosts it rather than the frame the output head reads\. Projecting it through the vocabulary, as the logit lens reads activations\[[38](https://arxiv.org/html/2609.10210#bib.bib1)\], asks it what it would mean somewhere it is not, and recovers77percent against a chance rate of1\.61\.6\. The repair needs no fitting\. The layer\-native token tableE~\\tilde\{E\}of Equation[7](https://arxiv.org/html/2609.10210#S2.E7)is the layer’s own picture of each token, in the layer’s own frame, and a read row decodes against it directly, both sides centered across the candidate vocabularyVV:

su\(t\)=\(wu−w¯u\)⋅\(E~\[l,t\]−E~¯\[l\]\),𝒫uK=arg​top−Kt∈Vsu\(t\)\.s\_\{u\}\(t\)=\\big\(w\_\{u\}\-\\bar\{w\}\_\{u\}\\big\)\\cdot\\big\(\\tilde\{E\}\[l,t\]\-\\bar\{\\tilde\{E\}\}\[l\]\\big\),\\qquad\\mathcal\{P\}\_\{u\}^\{K\}=\\operatorname\*\{arg\\,top\-\}K\_\{\\,t\\in V\}\\;s\_\{u\}\(t\)\.\(13\)The architecture’s carriage cancels on both sides of that dot product, which is why no transport between frames is required and why nothing has to be trained to supply one\. With the same weights, the same tokens and the same unit, the basis is the whole of the difference between77percent and58\.958\.9\.

What a unit*does*respond to is measured by substitution rather than inferred\. On a disjoint set of contexts𝒞′\\mathcal\{C\}^\{\\prime\}, so that the naming and the ground truth cannot share sampling noise,

a¯u\(t\)=1\|𝒞′\|∑c∈𝒞′au\(z\(l\)\(c,t\)\),𝒜uK=arg​top−Kt∈Va¯u\(t\)\.\\bar\{a\}\_\{u\}\(t\)=\\frac\{1\}\{\|\\mathcal\{C\}^\{\\prime\}\|\}\\sum\_\{c\\in\\mathcal\{C\}^\{\\prime\}\}a\_\{u\}\\big\(z^\{\(l\)\}\(c,t\)\\big\),\\qquad\\mathcal\{A\}\_\{u\}^\{K\}=\\operatorname\*\{arg\\,top\-\}K\_\{\\,t\\in V\}\\;\\bar\{a\}\_\{u\}\(t\)\.\(14\)and the two sets are compared, corrected for the rate at which a random list of the same length would agree:

ov​@​K​\(u\)=\|𝒫uK∩𝒜uK\|K,ov^​@​K=𝔼u​\[𝒫uK∩𝒜uK\]/K−K/\|V\|1−K/\|V\|\.\\mathrm\{ov\}@K\(u\)=\\frac\{\|\\mathcal\{P\}\_\{u\}^\{K\}\\cap\\mathcal\{A\}\_\{u\}^\{K\}\|\}\{K\},\\qquad\\widehat\{\\mathrm\{ov\}\}@K=\\frac\{\\mathbb\{E\}\_\{u\}\[\\mathcal\{P\}\_\{u\}^\{K\}\\cap\\mathcal\{A\}\_\{u\}^\{K\}\]/K\-K/\|V\|\}\{1\-K/\|V\|\}\.\(15\)The candidate setVVis the most frequent tokens in the corpus, and its size sets the chance rate: the sweep of Figure[7](https://arxiv.org/html/2609.10210#S3.F7)uses\|V\|=512\|V\|=512, so chance atK=8K=8is1\.61\.6percent and atK=256K=256is5050, while the unit\-against\-channel comparison of Table[15](https://arxiv.org/html/2609.10210#S3.T15)uses256256\. Since a chance\-corrected overlap must reach zero onceKKapproaches\|V\|\|V\|, the two are read against their own candidate sets rather than against each other\. A name that survives this is the set that moves the unit\.222Equation[14](https://arxiv.org/html/2609.10210#S3.E14)averages activations and takes the topKK\. Ranking within each context and averaging the ranks instead is not interchangeable with it: on the baseline the choice is worth1818points of agreement, on a bounded unit22\. The sweeps overKKand the ceiling comparison of Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)use the mean; the per\-seed tables use the rank, and each says which\. Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4)says why both are reported\.

Four quantities recur from here on, each reported under the conditions in Table[14](https://arxiv.org/html/2609.10210#S3.T14)\. The sweep overKKis the one worth naming here: the baseline’s own agreement changes sign betweenK=8K=8andK=256K=256, so a paper quoting only the first number would report a conventional transformer as substantially legible\.

Table 14:The measurements used throughout the paper, and the condition each is reported under\. Each condition is a specific failure from Appendix[C](https://arxiv.org/html/2609.10210#A3)that moved a number in the flattering direction\.##### How far down the list the name holds\.

The58\.958\.9percent is one point on a curve that falls \(Figure[7](https://arxiv.org/html/2609.10210#S3.F7)\)\. By the time two hundred and fifty\-six tokens are asked for, the parameters select worse than a random draw\. So a unit has a few tokens named at the head of its list, and where that head ends is set by the activation function rather than by the read row running out of what it knows\. The read row itself orders the vocabulary well throughout, which Section[5\.1](https://arxiv.org/html/2609.10210#S5.SS1)demonstrates on models differing only in that function\.

A second instrument asks a different question and reaches further down\. Rather than asking what a unit reads, collect the positions in ordinary text where it fires hardest and describe it by the tokens sitting there —*when it turns on*\. That description needs a floor of its own, because a unit’s firing correlates with general context and any random token would name it a little: the floor is what the same unit yields at two dozen to five dozen tokens back\. Against it, two units in three clear at forty positions and three in four at eighty\. Neither instrument substitutes for the other — the parameters say what moves a unit and stop after a handful of tokens; the firing context describes far more units and says nothing about cause\.

Figure 7:Whether a baseline unit’s parameters identify the set that drives it, against how much of that set is requested\. Chance\-adjusted per Equation[15](https://arxiv.org/html/2609.10210#S3.E15); the raw chance rate rises from 1\.6 percent atK=8K=8to 50 percent atK=256K=256\. Agreement decays withKKand passes below chance, where the parameters select tokens overlapping the true set less often than a random draw\. Table from 16 contexts, causal behavior from 16 disjoint contexts, context length 2048\. Agreement here is taken under the mean summary of Equation[14](https://arxiv.org/html/2609.10210#S3.E14), chance\-corrected; Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4)says what the alternative would give\.
##### How much of a model works on concepts\.

That the parameters pick out the right tokens says nothing about whether those tokens have anything to do with each other\. Somewhat over a quarter of a unit’s response falls on tokens forming a concept the token embedding can see and we would identify as related\. The question resists a yes or no, for a reason that belongs to the embedding rather than to the units\. Similarity in embedding space,

coh⁡\(S\)=1\|S\|​\(\|S\|−1\)​∑t≠t′∈SE^​\[t\]⋅E^​\[t′\]‖E^​\[t\]‖​‖E^​\[t′\]‖,E^​\[t\]=E⁡\[t\]−E¯\.\\mathrm\{coh\}\(S\)=\\frac\{1\}\{\|S\|\(\|S\|\-1\)\}\\sum\_\{t\\neq t^\{\\prime\}\\in S\}\\frac\{\\hat\{E\}\[t\]\\cdot\\hat\{E\}\[t^\{\\prime\}\]\}\{\\\|\\hat\{E\}\[t\]\\\|\\,\\\|\\hat\{E\}\[t^\{\\prime\}\]\\\|\},\\qquad\\hat\{E\}\[t\]=E\[t\]\-\\bar\{E\}\.\(16\)geometrizes*paradigmatic*classes, where members are mutually substitutable and cosine finds them\. A*relational*class is different: a corner can be on a street, in a room, or of a bed, and the members belong to a common concept by co\-occurring in a structure, which nothing in the training objective pushes together\. A unit responding to such a class is invisible to Equation[16](https://arxiv.org/html/2609.10210#S3.E16)by construction\.

We score each token in a unit’s set by how well it belongs to the others, and the cut is calibrated against a random set of the same size rather than chosen in raw cosine units\. What is reported is the share of the unit’s response on either side of that cut\. On the baseline eleven of a unit’s top forty tokens fall on the coherent side and carry2828percent of its response, leaving twenty\-nine tokens and the other7272percent to the remainder\.

That is a much smaller fraction than Table[13](https://arxiv.org/html/2609.10210#S3.T13)reports for the same models, because the two are asking different questions\. Equation[16](https://arxiv.org/html/2609.10210#S3.E16)asks whether a set is paradigmatic, and answers in the embedding, where relational and grammatical classes are invisible by construction\. The naming asks whether a set holds together in the corpus, where a class of tokens that occur in the same places does hold together whether or not any word covers it\. Nine sets in ten pass the second test and one in three passes the first, and the distance between those numbers is the size of what a token embedding does not group together: negated auxiliaries, sentence\-openers, units of measurement, the tokens that close a parenthesis\. Those are real categories with no name in the vocabulary\.

One thing that fraction cannot separate\. The decode centers both sides across the candidate vocabulary, which removes what the architecture’s carriage does to every token alike but not what it does to one token in particular, so a unit whose work is moving a specific token’s information from one depth to another reads here as a unit that responds to that token\. Since most of what a component writes is carriage in the first place \(Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)\), both shares are upper bounds on what is semantic: a component doing a little prediction and a great deal of transport is indistinguishable here from one doing only the first\.

Taken together a unit is nameable on both sides by different means, with the embedding the weak link in naming it further\.

### 3\.3Naming an attention channel

A channel is named the same way a unit is \(Table[13](https://arxiv.org/html/2609.10210#S3.T13)\)\. A prediction’s strongest contributors include channels, so what a channel writes is named on the same terms and to the same standard, and its names are of the same kind — numbers, the marks that close a clause, the openings of sentences\. Counted apart at a matched budget the two kinds come out close: about a third of each is characterized, and the sets that result are coherent within a few points of one another\. The naming reaches channels as readily as units\. Where the two kinds differ is in depth\. A channel is characterized in almost no cases through the first third of the stack — under a twentieth of them, against an eighth of feed\-forward units at layer zero\. This rises to a third by layer six, and from layer eight onward runs at three quarters to five sixths, which is where units already are\.

##### What a channel collects\.

Attention admits the same two questions a feed\-forward unit does\. What a channel produces at a position is gathered from other positions, and what it*collects*from a position is local, which puts it within reach of the same techniques used to name a unit’s inputs\. In fact, the instrument that names a unit’s read side works better on a channel, without modification\. A channel’s value\-projection row is its read row; decode it against the same layer\-native table and check it against the tokens the channel measurably picks up when one is substituted at the position it reads\. On the baseline the agreement is0\.8440\.844at eight tokens against the unit’s0\.5980\.598, and where the unit’s falls away as more of the list is asked for — to0\.2790\.279by sixty\-four — the channel’s*rises*, to0\.8920\.892\(Figure[8](https://arxiv.org/html/2609.10210#S3.F8)\)\.

That difference is the activation function\. A unit’s ordering has to pass through its activation function on the way to its behavior, and a non\-monotone one does not preserve it, where a channel’s value path is linear and nothing stands between the row and what it collects\. Where the read side is concerned attention is the easier component to name\. Described by when it turns on instead, a channel does slightly less well than a unit: three in five at forty firing positions against two units in three\.

Table 15:Whether a component’s parameters name the tokens that drive it, against how much of the list is asked for\. Both kinds are scored by one instrument: a unit’s read row and a channel’s value\-projection row are each decoded against the layer’s own token table \(Equation[13](https://arxiv.org/html/2609.10210#S3.E13)\) and checked against the set that causally moves the component, measured by substitution on disjoint contexts and corrected for chance as in Equation[15](https://arxiv.org/html/2609.10210#S3.E15)\. A channel is flat or rising in every model because its value path is linear\. A unit’s ordering must survive its activation function first, and what happens then depends on the function: the baseline falls below chance by128128tokens, ReLU decays without crossing, and the sigmoid model holds almost flat\. The candidate set here is256256tokens, half the size used in Figure[7](https://arxiv.org/html/2609.10210#S3.F7), so the baseline crosses at about two fifths of the candidate set in both —K≈104K\\approx 104of256256here andK≈208K\\approx 208of512512there\. Where a unit’s parameters stop naming its inputs is a share of the pool they are scored against rather than a fixedKK\. Agreement here is taken under the mean summary of Equation[14](https://arxiv.org/html/2609.10210#S3.E14), chance\-corrected; Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4)says what the alternative would give\. This table is the one exception to the disjoint sets of Table[14](https://arxiv.org/html/2609.10210#S3.T14): its layer table and its causal profile are accumulated in a single pass over the same contexts, which Appendix[C](https://arxiv.org/html/2609.10210#A3)measures at about a third of a point\.What a channel collects at the position it is reading is a separate question, and read that way an attention channel reads tokens, and reads them at the bottom of the stack\. Substitute a token at one position, read the value projection there, and take the share of a channel’s variance owed to that token rather than to the context around it\. At layer zero the quantity is1\.0001\.000on all six models\. Above it the quantity falls, by an amount that differs sharply between models, and the feed\-forward units of the same layers sit lower at almost every layer of every model, with ReLU \(Table[16](https://arxiv.org/html/2609.10210#S3.T16)\) the exception\.

A channel readsln1​\(x\)\\mathrm\{ln\}\_\{1\}\(x\)and the feed\-forward unit above it readsln2​\(x\+attn​\(x\)\)\\mathrm\{ln\}\_\{2\}\(x\+\\mathrm\{attn\}\(x\)\), so the drop in token share between them at one layer is the context that layer’s attention delivered\.

Two things about that quantity hold on every model measured\. The first attention block delivers little of it in absolute terms, between0\.000\.00and0\.150\.15, so the token is present at the bottom of the stack and moves little between positions — though on the softplus and unshaped sigmoid models that little is a large share of the small total those models deliver anywhere\. And the layer that delivers most of it always sits in the first half of the stack\. What does not hold everywhere is the balance between the two ends: four of the six models deliver more context in their first third than their last, and the softplus model and the unshaped sigmoid run the other way\. Why those two run the other way is left for later work\.

Figure 8:Left: the read side of an attention channel and of a feed\-forward unit, scored with the same instrument in one pass per model\. Right: both against depth\. A channel’s read side is measured from its value projection, which is evaluated at the position being attended to and therefore moves when the probe moves; a channel’s*output*at a position is assembled from other positions and does not\.Table 16:What an attention channel collects at the position it is reading, and what its layer delivers from elsewhere\. Token share is the fraction of a channel’s variance owed to the token at that position rather than to its context; layer zero must return1\.0001\.000, since the residual there is the token, which is the check that the measurement works\. “Layers\>\>FFN” counts the layers at which the channel’s token share exceeds that of the feed\-forward unit above it, which reads the residual after attention has written to it\. The context a layer delivers is that difference, peaking in the first half of every stack\.All of this reaches the value side at a single position\. What a channel finally contributes is that value aggregated over the sequence by a pattern we do not explore further in this work\. Decoding attention across the entire context is left to future work\.

Finally, naming this way reaches only the components that write a prediction\. A quarter of the model contributes to predictions without ever being among the strongest contributors to one, and those components are not idle: ranking contributions to the*read rows*of the strongest components rather than to the readout, a third of the leaders one level back are drawn from that band\. They drive the components that write, which no ranking against the readout can see\. Reaching them is what the graph built in Section[2](https://arxiv.org/html/2609.10210#S2)is for\.

### 3\.4What the graph shows

The three quarters of a unit’s response that the embedding groups into nothing \(Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2.SSS0.Px2)\) — call them its*anonymous*tokens — have an origin even where they have no name\. Section[2](https://arxiv.org/html/2609.10210#S2)supplies a graph over the same components, and read through that graph those tokens are written by components upstream of the unit\. Saying which ones takes care \(Table[17](https://arxiv.org/html/2609.10210#S3.T17)\)\. WriteCCfor the unit being explained andAAfor a component credited with driving it\. Of the three shapes that produce that credit, the one to locate is the*direct*source\. That means ruling out*siblings*, which read whatCCreads and write nothing into it, and*mediated*sources, which reachCConly through something else\.

Table 17:The three shapes that would putAAhigh inCC’s ranking\. Ranking cannot tell them apart, because all three makeAA’s firing trackCC’s\. Deletion kills the sibling case: with no edge between them, removingAAleavesCCuntouched\. Deletion cannot separate the mediated case, a real path, from a direct one, so the sufficiency search reaches it instead, because a set that needsBBhas to keepBBto work\.What follows measures where a unit’s*feed\-forward*drive comes from; a channel’s value side is taken up at the end of the section\.

##### Ruling out siblings\.

The anonymous tokens are not noise\. Split sixteen contexts into two groups of eight, average a unit’s response over each group separately, and the two profiles agree at0\.6480\.648, so the unit responds to those tokens the same way on text it has not seen before\. Fitting one does not name it either — every fitted estimator tried returns negative held\-out variance \(Appendix[C\.10](https://arxiv.org/html/2609.10210#A3.SS10)\)\.

No fitting is necessary, however\. The architecture that supplies the atoms also supplies an ordering among them\. Take a unituuat layerℓ\\ellwith read rowwuw\_\{u\}\. Every upstream componentvvat layerℓv\\ell\_\{v\}has a write columncvc\_\{v\}, and an activation on tokenttwhich we writeav​\(t\)a\_\{v\}\(t\), in the sense of Equation[4](https://arxiv.org/html/2609.10210#S2.E4)\. The drivevvcontributes touuis then

su\(t\)=∑v:ℓv<ℓωvav\(t\),ωv=cos\(carry\(cv,ℓv,ℓ\),wu\),s\_\{u\}\(t\)\\;=\\;\\sum\_\{v\\,:\\,\\ell\_\{v\}<\\ell\}\\omega\_\{v\}\\,a\_\{v\}\(t\),\\qquad\\omega\_\{v\}\\;=\\;\\cos\\\!\\big\(\\mathrm\{carry\}\(c\_\{v\},\\ell\_\{v\},\\ell\),\\,w\_\{u\}\\big\),\(17\)withcarry\\mathrm\{carry\}the composition of fitted layer maps of Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)and no coefficient fitted anywhere\. The weightωv\\omega\_\{v\}is how much of whatvvwrites lands along the rowuureads with\.

Ranked bysus\_\{u\}, the tokens that driveuusit above the rest of the candidate pool, for the anonymous part as well as the nameable one, in both families and at every seed that trained \(Figure[9](https://arxiv.org/html/2609.10210#S3.F9), left\)\. Three nulls say the ordering is not an artifact of the machinery that produced it: permuting which component holds which alignment, setting every weight to one, and substituting components at the target’s own layer that cannot have driven it\. All three return chance or close to it\.

The sibling shape is not disposed of that way\. Two components that read the same thing fire at the same times, so a sibling ofuuhas a firing profile shaped likeuu’s — and a sum of enough siblings rebuildsuu’s own profile, which then “predicts” what drivesuuwhile none of them wrote anything into it\.

Turning that shape into a score takes a substitution\. Weight each writer not by whether it wrote into the rowuureads with, but by whether it*resembles*uu, meaning it reads whatuureads:

ωvsib=cos⁡\(wv,wu\),\\omega^\{\\text\{sib\}\}\_\{v\}\\;=\\;\\cos\\\!\\big\(w\_\{v\},\\,w\_\{u\}\\big\),\(18\)read row against read row, with only the weight changed\. Equation[17](https://arxiv.org/html/2609.10210#S3.E17)is the direct shape scored and Equation[18](https://arxiv.org/html/2609.10210#S3.E18)is the sibling shape scored\. On the baseline the two rank a unit’s drivers about equally well, so no ranking can choose between them\.

Deleting components can, because the two shapes predict different damage\. Ablate a setAAand re\-read the unit; what a score expects to lose, token by token, is that same sum restricted to what was removed,

d^u​\(t\)=∑v∈Aωv​av​\(t\)\.\\hat\{d\}\_\{u\}\(t\)\\;=\\;\\sum\_\{v\\in A\}\\omega\_\{v\}\\,a\_\{v\}\(t\)\.\(19\)withωv\\omega\_\{v\}either weight\. With\|A\|=1\|A\|=1the two differ by a positive scalar, so one deletion can never separate them\. With\|A\|\|A\|larger the weights become a mixture over profiles\. Ablating twenty\-four components at random, so the choice favors neither shape, the direct weighting tracks the damage and the sibling weighting stays flat, in both families\.

Figure 9:Left: how far Equation[17](https://arxiv.org/html/2609.10210#S3.E17)ranks a unit’s drivers above each of its two nulls, one point per seed\. Plotted as the excess, so the null is the dotted line and any point above it is the ordering beating that null; filled circles are the excess over the shuffle, open triangles over the same\-layer control\. Every group clears both, on the anonymous part as well as the nameable one\. Right: the same score split by how far each term was carried, as the gap between the two halves\. A bar above zero is a band that favors the nameable tokens and one below it favors the anonymous tokens, so the sign change across the bands is the result: what arrives at the embedding is what a static token table can read, and what has been carried furthest is what it cannot\.That settles who writes the anonymous tokens\. Where those writers sit explains why the vocabulary misses them\. Split Equation[17](https://arxiv.org/html/2609.10210#S3.E17)by how far each term was carried and the two halves of a unit’s set separate: the embedding term favors the*nameable*tokens; terms carried six layers or more favor the anonymous ones \(Figure[9](https://arxiv.org/html/2609.10210#S3.F9), right\)\.

This is the shape[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14)predicts\. A term that enters at the embedding and is read a few layers later has been turned only a few times and still lies near the coordinates the token table is written in\. A term written at layer one and read at layer nine has been turned nine times and lies somewhere else\. Decoding both through one static table shows the first alone, and what a reader then sees is a nameable quarter beside an anonymous remainder — one set of tokens, read in two frames, with the table in only one of them\.

Table 18:One unit’s causal top forty, separated by where each token’s drive arrives from\. Same units, same tokens, and same measurement as Table[46](https://arxiv.org/html/2609.10210#A3.T46), which lists them as a single list and where no description fits; each token is assigned here to the source band supplying most of its drive, standardized per band so the assignment is not simply the largest band\. The parts are categories: capitalized sentence\-openers, modals, do\-support, prepositions and wh\-words, third\-person pronouns\. Baseline model,K=40K=40, context length 2048\.Sorted by where it came from, the remainder stops looking like a grab\-bag\. Assigning each of a unit’s tokens to the band supplying most of its drive splits the set into a median of four groups, and those groups are categories: a nearest\-centroid classifier fitted to half a unit’s tokens recovers the source of the other half well above a shuffled control, and part\-of\-speech purity within a group exceeds a matched random partition\. Mean pairwise cosine, the measure Equation[16](https://arxiv.org/html/2609.10210#S3.E16)uses to call a token nameable, barely moves on the same partition\. The groups are classes like auxiliaries and copulas, degree adverbs, clause openers, numbers and time words, whose members are mutually substitutable in a grammatical position rather than close in a distributional neighborhood\.

So the remainder is written by components upstream of the unit, and it separates into categories once the terms are sorted by where they came from\. The reason a reader misses it is that the basis it is read in is the wrong one\. Table[46](https://arxiv.org/html/2609.10210#A3.T46)is what one unit’s set looks like with every term decoded through the same table; Table[18](https://arxiv.org/html/2609.10210#S3.T18)is the same set with the terms separated by where they came from\.

The same ordering reaches a channel\. Equation[17](https://arxiv.org/html/2609.10210#S3.E17)with the unit’s read row replaced by the channel’s value\-projection row, and the gain of the norm feeding attention rather than the one feeding the feed\-forward block, traces what drives a channel to the components upstream of it\. Attention at layerℓ\\ellreads the pre\-attention residual, so summing strictly belowℓ\\ellexcludes its own layer’s attention from its sources\. Against the same two nulls the ordering holds on both families, and the controlled figure — the excess over same\-layer channels — separates the baseline from everything else across ten models and seeds: the baseline’s three seeds sit well below the five shaped models, which cluster tightly together\. A channel is about twice as traceable as a unit on the baseline and about as traceable on the rest\. The baseline is the hard case here as it is for naming\.

Those two figures rest on different footings\. A channel’s drive is a linear read of the residual and Equation[17](https://arxiv.org/html/2609.10210#S3.E17)is a linear sum, so the predictor matches the target’s functional form exactly; a feed\-forward unit’s response is measured after its activation function, and the same linear score is fed through a nonlinearity\. Attention scores higher for that reason alone, before anything about legibility enters\. The same effect raises its same\-layer null on the baseline, where channels reading the identical state in parallel correlate with the target for free, which is why the excess over that null rather than the raw figure is the number quoted\.

An attention channel now has incoming edges, and few of them clear the same two percent of incoming drive that a feed\-forward node’s sources clear: across the baseline’s gallery pages, most channels carry a read side and only a handful carry an edge into it\. A channel draws on the residual broadly rather than on a handful of writers, which is a different shape of dependency from the one the feed\-forward graph has\.

A component, then, carries a name on each side\. Close to half the model’s components are named by what they write, read off the predictions they drive, and the share each supplies belongs to the component rather than to the context\. The head of a unit’s inputs is named from its weights, decoded in the frame of its own layer, and where that head ends is set by the activation function\. The remainder — three quarters of what a unit reads, which the embedding groups into nothing — is named by which components supply it, and sorted by where it came from, it is grammar\. A channel is named on the same terms on both sides, and more easily on the read side, because nothing stands between its row and what it collects\.

Table 19:The provenance ordering and the five objections it was tested against\. Rows above the deletion block are areas under the ROC curve, where0\.50\.5is chance; the sibling and deletion blocks are differences, where00means the two weightings are indistinguishable\. The baseline is the hard case throughout, ruling out siblings only under deletion and passing the depth split only in its embedding half, where the sigmoid model passes every one by more\. Context length 2048, 16 contexts, 512 candidate tokens\.

## 4Editing a Model: Say what?

Naming a component says what it is responsible for\. This section writes to the components so named, and the edits come in three kinds of increasing reach\. The first turns a component up and asks whether the tokens its name predicts are the ones that move\. The second installs an association the model does not hold into a spare component, key and value both read from the parameters\. The third drives a unit the model trained for itself, from two layers below it, with nothing installed at all\. Between the second and the third sit two questions every edit has to answer — how deep to write, and in which frame — and a circuit built across two layers that depends on both answers\. The section closes by scoring the install against the standard rank\-one editor\.

### 4\.1Turning a component up

The simplest edit follows from a component being identified by what it writes and who reads it: multiply a component’s activation at the position being predicted by a gainα\\alphaand run the model again, with nothing else touched and nothing trained\. That is the smallest intervention that still has a predicted consequence — it changes how loud a component is without changing what it says, so the scope its parameters name is the same before and after and the prediction is entirely about where the logits should move\.

Three quantities are read together\. The*own scope*is the set the edited component’s write column promotes\. The*other scopes*are the sets named by the other components edited on the same predictions, which controls for an edit that simply raises whatever it touches\. And a size\-matched*random*draw from the vocabulary catches an edit that moves the distribution wholesale\. Any edit is respected if pushed hard enough and harmless if pushed gently, so strength is a variable\. We sweepα∈\{2,4,8\}\\alpha\\in\\\{2,4,8\\\}, over360360edits spanning eight depths of the baseline\.

Table 20:An edit installed in one component of the baseline, scored against the scope its own parameters name and against the scopes of the other components edited in the same predictions, at matched strength\. “Respects the name” is the share of edits moving their own scope more than another component’s\.Table[20](https://arxiv.org/html/2609.10210#S4.T20)reports the results\. The edit moves the tokens the parameters named and leaves the other components’ targets alone in ninety\-seven percent of cases, and the effect on the named scope exceeds the effect elsewhere by more than an order of magnitude\.

Figure 10:Left: what a gain ofα\\alphadoes to the three sets, median over360360edits with the interquartile range shaded\. The named scope grows roughly in proportion toα\\alphawhile the collateral stays flat at zero, so the ratio between them*widens*with strength — ten times atα=2\\alpha=2and sixty\-eight atα=8\\alpha=8\. This is the answer to the objection the definition itself raises: the edit is not merely respected because it was pushed hard, since pushing harder buys no collateral damage\. Right: the same edit by the depth of the component it was installed in, atα=4\\alpha=4\. It lands at every depth measured, including layer zero\.Figure[10](https://arxiv.org/html/2609.10210#S4.F10)sweeps the strength\. Doublingα\\alpharoughly doubles the movement on the named scope —0\.300\.30,0\.890\.89,1\.771\.77— while what happens to every other component’s scope stays flat and near zero at all three settings\.

### 4\.2Installing an association that was not there

A gain is not what is usually meant by writing to a model\. The model\-editing literature installs associations a model does not hold\[[36](https://arxiv.org/html/2609.10210#bib.bib16)\], treating a feed\-forward layer as a linear associative memory and applying a rank\-one update: a key for the subject, a value for the target\. We adopt that framing here\.

Writing a target token’s unembedding direction into a single unit’s output column, in closed form and with no gradient, is already established — by[Dai et al\. \[13\]](https://arxiv.org/html/2609.10210#bib.bib44)through token embeddings and by[Hakimi et al\. \[23\]](https://arxiv.org/html/2609.10210#bib.bib46)through unembedding columns, the latter explicitly a per\-unit version of the rank\-one mechanism\. What no method we can find does is choose the*site*from the weights\. ROME localizes by causal tracing; the knowledge\-neuron line scores neurons by integrated gradients; the closest parameter\-side method still localizes with a logit\-lens score against cached activations\. Every one of them reads the site off what the model*does*on some input\. Section[3](https://arxiv.org/html/2609.10210#S3)finds what a unit reads, which makes the site available directly\.

That is worth having for a reason the same literature supplies\.[Hase et al\. \[24\]](https://arxiv.org/html/2609.10210#bib.bib45)report that where causal tracing localizes a fact predicts almost nothing about where editing it succeeds — the tracing effect accounts for a fraction of a percent of the variance in edit success, against the9494percent explained by the choice of layer alone, with the raw correlation slightly negative\. The causal tracing that picks the site, the expensive step of the standard pipeline, is not doing the work it appears to do\.

Legibility supplies both halves of the update:

- •thekeyis the layer\-native row for tokenAA,E~​\[l,A\]\\tilde\{E\}\[l,A\]of Equation[7](https://arxiv.org/html/2609.10210#S2.E7)centered over the candidate vocabulary — the state the layer sees whenAAis the current token, measured once and fitted to nothing;
- •thevalueis the readout direction for tokenBB,UB−U¯U\_\{B\}\-\\bar\{U\}, so that a component firing on that key lands onBB’s logit\.

They are written into a*spare*component, the one whose activation over the corpus is smallest, so nothing the model was using is destroyed and the cost is bounded by construction\. The pair is drawn at random from frequent tokens, and the control is built in:BB’s rank whereAAappears is several hundred before the edit, so the model cannot be said to have been leaning toward it\.

Table 21:Installing “afterAA, sayBB” into one spare component, over1,1761\{,\}176trials across seven token pairs, twelve depths and seven strengths, with the read gain swept separately in Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)\. Each site is shown at the weakest strength that works\. The edit movesBBto the top of the distribution whereAAoccurs and leaves its rank elsewhere alone; what separates the two sites is the price\.Figure 11:Left: how often the edit takes, against how hard it is pushed\. A feed\-forward unit takes it on nearly every trial fromβ=2\\beta=2upward, and at every depth through layer eight; an attention channel needs roughly eight times the strength and succeeds at scattered depths\. Right: held\-out loss against the same strength, with the unedited model dashed\. The unit’s working range costs almost nothing and the channel’s does not\.It works, on one component \(Figure[11](https://arxiv.org/html/2609.10210#S4.F11)\)\. A single spare feed\-forward unit atβ=1\\beta=1movesBBfrom rank578578to rank11at the positions whereAAoccurs, leaves its rank elsewhere where it was \(778778to782782\), and costs a quarter of one percent of held\-out loss \(Table[21](https://arxiv.org/html/2609.10210#S4.T21)\)\. Raisingβ\\betabuys nothing after that: the association is already first, and the loss climbs, reaching half again the unedited value byβ=16\\beta=16\. Neither key nor value was fitted, traced, or optimized\.

### 4\.3Editing attention

A simple edit to attention — amplifying a channel and reading the result at the positions where that channel is actually active — moves the tokens it promotes on8888to9191percent of trials, against9393to9494percent for a unit, with the collateral on other components’ scopes near zero in both cases; the channel’s movement is about a quarter of the unit’s at matched strength\. The sign matters here\. What a channel gathers flips sign with context, so its output column demotes exactly the tokens it promotes whenever the gathered value is negative, and a scope read without that sign scores the wrong end of the list about half the time — taken unsigned the same measurement returns chance,4747to4949percent, with the scope moving the wrong way\.

Installing a new behavior into attention is harder\. An edit counts only ifBBreaches the top ten whereAAoccurs, is*not*promoted where it does not, and itself costs under ten percent of held\-out loss — which is what separates an installed association from a component shouting\. On that criterion1010percent of channel edits succeed against3232percent of unit edits\.

The reason is the same fact that makes a channel’s read side legible\. A channel’s value is collected from whatever position it attends to, so what arrives atAAis an attention\-weighted average rather than the value itself\. The rank ofBBaway fromAA*falls*, from826826to11421142, so the channel demotes the target where it was not installed\. What happens instead is*attenuation*at the place the edit was aimed\. Measured over twelve token pairs and five depths, an installed channel gathers−0\.06\-0\.06whereAAis absent and\+0\.38\+0\.38whereAAis the current token, a contrast a third the size of what a unit delivers\. That is why the channel needsβ=8\\beta=8where the unit needsβ=1\\beta=1, and why it costs a tenth of held\-out loss where the unit costs a quarter of a percent\. The extra strength makes up for the averaging, but at a cost\.

Which head takes the edit matters\. Heads differ in how much of what they deliver is about the position they are at rather than the positions they looked at, and the difference is large: put the same read row into one channel of every head at once and the correlation between what a head gathers and the value at the current position runs from−0\.02\-0\.02to0\.750\.75\. In a separate sweep over twelve pairs, five depths and five strengths, installing in the quietest channel of the layer installs and stays local on11percent of trials; installing instead in the head that gathers the key most sharply raises what arrives at the key from0\.300\.30to1\.531\.53, brings the target from rank574574to rank1414, and takes that share to88percent\.

What does succeed is editing the query and key as well\. Addingκ⁡\(kC⋅zq\)​\(w⋅zp\)\\kappa\\,\(k\_\{C\}\\\!\\cdot\\\!z\_\{q\}\)\(w\\\!\\cdot\\\!z\_\{p\}\)to a head’s score makes it attend to positions carryingCC\. Butκ\\kappahas no absolute meaning: the term competes with the head’s existing score, whose scale differs by orders of magnitude between heads\. Aκ\\kappachosen once is far too small to move some heads and saturates others\. Withκ\\kappaswept in multiples of the head’s own score standard deviation there is a window, and inside it the installed term and the existing one compose rather than compete: the existing score carries distance, so their sum expresses*recent andCC*rather than either alone\. What that buys is a fact conditioned on a token the position does not carry; how much it buys depends on where the reading component takes its read row, and Section[4\.6](https://arxiv.org/html/2609.10210#S4.SS6)reports the result once the reader has been built\.

What attention writes at a position is a mixture over*other*positions, so a value edit alone arrives diluted at the reader\. A feed\-forward unit’s output at a position depends on that position alone, which is what lets an edit there be both strong and local\. The model\-editing literature writes to feed\-forward layers predominantly, and the work here suggests that is a reasonable default rather than a necessary one\. Attention takes an edit when the pattern is installed alongside the value and the strength is set in the units the head itself has been trained to\.

An attention channel is the more legible of the two on its read side and the worse of the two to install into; Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)finds the same on the activation arms\.

### 4\.4Where to place an edit

What to write is a question about meaning, and Section[3](https://arxiv.org/html/2609.10210#S3)answers it: a component’s parameters say which tokens turn it on and which it promotes\. Three questions about the architecture remain — how deep a write has to be made before it arrives, which of several available frames it should be expressed in, and what has to be true for anything to read it — and the answer to all three starts from the recurrence\. A transformer carries its state in a stream that every layer adds to:

hℓ\+1=hℓ\+Aℓ​\(LN⁡\(hℓ\)\)\+Fℓ​\(LN⁡\(hℓ\)\),h\_\{\\ell\+1\}\\;=\\;h\_\{\\ell\}\\;\+\\;A\_\{\\ell\}\\\!\\left\(\\mathrm\{LN\}\(h\_\{\\ell\}\)\\right\)\\;\+\\;F\_\{\\ell\}\\\!\\left\(\\mathrm\{LN\}\(h\_\{\\ell\}\)\\right\),\(20\)withAℓA\_\{\\ell\}the attention block at depthℓ\\ellandFℓF\_\{\\ell\}the feed\-forward block\. The addition in Equation[20](https://arxiv.org/html/2609.10210#S4.E20)is the whole of the architecture’s contribution to an edit\. Anything written intohhat depthℓ\\ellis still inhhat every later depth, in the same coordinates, because nothing in the recurrence moves it\. What changes with depth is how much company it has\. Every layer above adds its own contribution on top, and a write made early is a smaller and smaller share of what the readout finally sees\.

Writing a direction into the residual stream and observing the consequence is the mechanism behind activation steering\[[55](https://arxiv.org/html/2609.10210#bib.bib12),[43](https://arxiv.org/html/2609.10210#bib.bib13),[28](https://arxiv.org/html/2609.10210#bib.bib22)\]\. The measurement here asks the prior question: of what is written, how much is still there at the end\.

A unit directionwwis added to the stream at1010percent of positions immediately before layerXX, scaled to a tenth of the median residual norm at that depth\. WritinghLh\_\{L\}for the state arriving at the final norm andΔ=hLwritten−hLclean\\Delta=h\_\{L\}^\{\\,\\text\{written\}\}\-h\_\{L\}^\{\\,\\text\{clean\}\}for the change it causes, two quantities are read at the written positions:

survival⁡\(X\)=cos⁡\(Δ,w\),spill⁡\(X\)=medp∉W⁡∥Δp∥medp∈W⁡∥Δp∥,\\mathrm\{survival\}\(X\)\\;=\\;\\cos\\\!\\left\(\\Delta,w\\right\),\\qquad\\mathrm\{spill\}\(X\)\\;=\\;\\frac\{\\operatorname\{med\}\_\{p\\notin W\}\\lVert\\Delta\_\{p\}\\rVert\}\{\\operatorname\{med\}\_\{p\\in W\}\\lVert\\Delta\_\{p\}\\rVert\},\(21\)whereWWis the set of written positions\. Survival asks whether the write still points its own way\. Spill asks how much of it has been carried to positions it was never made at\.333[Timkey and van Schijndel \[54\]](https://arxiv.org/html/2609.10210#bib.bib50)show that a few dominant coordinates can control a cosine measure\. Both quantities here are differences, in which a near\-constant coordinate cancels before the cosine is taken, and masking the ten coordinates that carry most of the measured change moves the comparison of Figure[13](https://arxiv.org/html/2609.10210#S4.F13)by at most0\.0120\.012\.

Four kinds of direction are written\. A*private*direction is drawn at random, so nothing in the model was built to read it\. A*layer\-native*direction is a row of the token tableE~​\[X,⋅\]\\tilde\{E\}\[X,\\cdot\]of Equation[7](https://arxiv.org/html/2609.10210#S2.E7), returned to residual coordinates by dividing out the layer’s norm gain\. A*working\-state*direction is a leading principal direction of the residual atXXwith the readout subspace projected out\. A*readout*direction isUB−U¯U\_\{B\}\-\\bar\{U\}for a tokenBB\.

Figure 12:Left: how much of a written direction still points its own way at the readout, against the depth it was written at, for four kinds of direction\. Right: spill, the change at positions the write was never made at, relative to the change where it was\. Written at1010percent of positions, at a tenth of the median residual norm\.Figure[12](https://arxiv.org/html/2609.10210#S4.F12)reports the result\. Depth dominates\. A write made before layer two survives at0\.110\.11and a write made before layer eleven at0\.330\.33to0\.810\.81, a spread far larger than any difference between the four kinds of direction at fixed depth\. Spill falls by a factor of ten over the same range, from0\.400\.40at layer zero to0\.040\.04at layer eleven\. An early write is buried under what follows it and carried to positions it was never made at\. A late write stays where it was put in both senses, which is why every edit in this paper is made in the last third of the stack\.

The private direction is the best preserved of the four at every depth, which is the key to reading the rest of the panel\. Preservation is the default behavior of Equation[20](https://arxiv.org/html/2609.10210#S4.E20), so the gap between a family and the private baseline measures how much the model*acted*on it\. On that reading the layer\-native direction is barely touched, sitting level with the private baseline\. The working\-state direction is the most transformed at seven of the eight depths measured, which is the rotating stream of[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14)seen from the inside\. The readout direction sits between them, and its consequence is the only one of the four aimed anywhere in particular\.

WritingPPfor the projection onto the leading6464principal directions of the centered unembedding, the share of the response that is visible to the readout is∥P⊤​Δ∥/∥Δ∥\\lVert P^\{\\\!\\top\}\\Delta\\rVert/\\lVert\\Delta\\rVert, which for an isotropic direction in768768dimensions is0\.2890\.289\. The three non\-readout families sit at0\.230\.23to0\.300\.30at every depth, which is to say at chance\. The readout family rises from0\.270\.27to0\.430\.43, with the excess growing toward the layers where the prediction is committed\. A write in the readout frame is the only one the model ultimately answers\.

That is also why the constructions here stop one layer short of the top\. Preservation rises with depth for every family, with one exception: over the last step the readout frame is the only one that gets*worse*, by0\.1220\.122on the baseline and0\.0860\.086on the sigmoid model, where the layer\-native, working\-state and private families all improve substantially over the same step\. The last layer reads readout\-frame content and answers it, which consumes it, and does nothing of the kind to the other three\. Both models agree\. A write placed there to be read by something above it has nothing above it left, and a write placed there to survive is the one kind the commit layer spends\. Every construction below installs its reader at layer ten of twelve for that reason\.

A division of labor follows from the two token frames behaving oppositely — the layer\-native frame carried by the stream untouched, the readout frame the one every later component responds to — and it is the reason a circuit needs more than one component\. A write in the readout frame acts on the output directly, at the price of being the frame every later component is listening to\. A write in any other frame is inert at the output and reaches it only through something that reads it\. A circuit therefore carries its intermediate state in one frame and commits its answer in the other\.

### 4\.5What the turn does to a write

The frame turns from depth to depth, and there are two ways to measure that turn\. Only one of them describes what happens to content already in the stream\. Orthogonal Procrustes between consecutive layer\-native token tables gives the map that describes how each depth’s own reading frame is oriented,

Rℓ=arg⁡minR∈O⁡\(d\)⁡∥Xℓ​R−Xℓ\+1∥F=U​V⊤,Xℓ⊤​Xℓ\+1=U​Σ​V⊤,R\_\{\\ell\}\\;=\\;\\arg\\min\_\{R\\in O\(d\)\}\\lVert X\_\{\\ell\}R\-X\_\{\\ell\+1\}\\rVert\_\{F\}\\;=\\;UV^\{\\\!\\top\},\\qquad X\_\{\\ell\}^\{\\\!\\top\}X\_\{\\ell\+1\}=U\\Sigma V^\{\\\!\\top\},\(22\)and the same fit run between the residual*states*rather than the tables gives a second mapQℓQ\_\{\\ell\}\. The two differ by a factor of four: the table turns by a median plane angle of5656degrees per step where the state turns by1010to1818\. Content sitting in the stream is acted on by the second of these, and a comparison against the first says nothing about it\.

One step is unlike the rest\. The state turns by8181degrees between layers zero and one, against1010to1818everywhere above, so the bottom of the stack is a different operation from the body of it\. The model is stashing away the input token and building the initial workspace\. What follows does not apply to those early one to two layers\.

The difference of two conditional means is a*difference in differences*\. Capture the reader’s input with the marker written and again without it, and take the change at the marked positions only,

Δh→r=Ep∈W⁡\[xrwritten\]−Ep∈W⁡\[xrclean\]\.\\Delta\_\{h\\to r\}\\;=\\;\\operatorname\{E\}\_\{p\\in W\}\\\!\\left\[x\_\{r\}^\{\\,\\text\{written\}\}\\right\]\-\\operatorname\{E\}\_\{p\\in W\}\\\!\\left\[x\_\{r\}^\{\\,\\text\{clean\}\}\\right\]\.\(23\)Equation[23](https://arxiv.org/html/2609.10210#S4.E23)is compared against the direction as written and against the same direction carried through the composed state map∏ℓ=hr−1Qℓ\\prod\_\{\\ell=h\}^\{r\-1\}Q\_\{\\ell\}\. Because that turn is small the two candidates are themselves close, so a third quantity is needed to isolate what the turn contributes on its own: remove the written component from both and ask how much of what is left lies along the turned increment\.

Figure 13:A direction written into the residual before layer three, followed to every later depth\. Solid is its alignment with itself, dashed with the same direction carried through the composed state\-level turn, and the lower curve is what the turn contributes once the direct part is removed from both\. Medians over probe directions at ten percent of positions\.Figure[13](https://arxiv.org/html/2609.10210#S4.F13)reports the result\. A write made before layer three is aligned with itself at0\.700\.70three layers later and with its turned image at0\.600\.60, and the turn’s own contribution is0\.0650\.065\. The ordering holds at every gap: the write is more itself than it is a turned copy of itself, and the turn adds a flat0\.0650\.065to0\.110\.11on top\. Both components are present, as the recurrence requires, and the direct one is the larger\.

What grows with distance is the model’s response to the write rather than a turned copy of it\. Since the alignment with the write falls from0\.890\.89at one layer to0\.250\.25at nine, the share of the change orthogonal to the write rises from0\.450\.45to0\.970\.97over the same range\. By the far end almost all of what a reader sees is the response, and no map fitted here predicts its direction\.

Two rules follow, and both are used in everything below\. A read row is measured at its destination, as the conditional mean difference of the reader’s own input, rather than assumed from the writer’s intent or carried there analytically: the write arrives diluted and wrapped in a response that nothing predicts, so the only reliable statement about the reader’s input is a measurement of it\. And the turn is a correction worth a tenth of the signal rather than the thing to correct for, so a read row carried to its destination through the turn is at best a tenth better than one that ignores the turn\. On a thirty\-six\-layer stack the same ordering holds from layer seven through thirty\-one, and the turned image leads only above a single anomalous twenty\-degree step near the top, on one model of unusual shape\.

### 4\.6A circuit across two layers

An edit that spans layers has to carry its intermediate state in some frame, and the stream between writer and reader is a communication channel\. There are two ways to use it\. The first is to write in the model’s own geometry so that components already present read the result\. The price is that the write has to be expressed in a frame that must be estimated, and the estimate is what the accuracy of the edit then rests on\. The second way is available whenever both ends of the circuit are supplied, and it is the one the construction below uses: carry the intermediate state on a direction drawn at random, which is a coordinate system chosen rather than found and therefore known exactly\.

A channel of that kind is available because a trained transformer operates in a fraction of its width, and in the same fraction more or less at every depth\. The residual’s effective dimensionality, measured as a participation ratio of the covariance spectrum, is roughly170170to195195through the middle of the stack once the small number of very high variance coordinates of[Sun et al\. \[50\]](https://arxiv.org/html/2609.10210#bib.bib49)is set aside\. Pooling the layers a wire has to cross barely raises it: over layers eight to eleven the union measures183183dimensions where the largest single layer in that span measures about175175, and four disjoint subspaces of that size would need some six hundred\. The layers share one working subspace rather than each occupying its own, so a direction drawn once is untouched at every depth it passes through, and about five hundred and eighty of the model’s seven hundred and sixty\-eight dimensions are free for the purpose\.

Being private buys knowledge of the carrier rather than separation from the model\. A random direction still places about175/768≈0\.48\\sqrt\{175/768\}\\approx 0\.48of its length inside the space the model uses, and that overlap costs little: holding it and the norm fixed and varying only how many active directions carry it, from all of them down to one, leaves held\-out loss flat, with the cost instead growing roughly quadratically in the size of the write\. Nor does the carrier’s frame change what the circuit achieves\. Repeating the construction with the marker drawn from the readout frame, from the layer’s own token frame, and from the working state leaves the target at median rank1212and the separation at71\.671\.6in every case, because the reader’s row is measured where it will be used and a measured conditional mean isolates whatever was written there\. What a private channel saves is the estimate, which is what the construction turns out to be limited by \(Figure[15](https://arxiv.org/html/2609.10210#S4.F15)\)\.

Circuits of this shape occur in trained models without being put there\.[Neo et al\. \[37\]](https://arxiv.org/html/2609.10210#bib.bib48)report attention heads that recognize a context and activate a downstream token\-predicting unit accordingly, which is the found version of what follows\. Both halves of the division of labor appear in a single construction\. An attention head at depthhhis installed to fire where a chosen token occurs and to write a private direction\. A feed\-forward unit at depthr\>hr\>his installed to read that direction and to write a target token’s readout direction\. Neither component existed before, and the model supplies only the stream between them\.

The unit’s read row is the conditional mean of Equation[23](https://arxiv.org/html/2609.10210#S4.E23), measured at its own layer, and a component installed this way needs a silent state at the positions the edit is not aimed at\. These models carry no bias anywhere, so one is built from the near\-constant direction of the read space\. That direction is closely related to the massive activations of[Sun et al\. \[50\]](https://arxiv.org/html/2609.10210#bib.bib49): a small number of coordinates that are large and nearly input\-independent, and that serve as an implicit bias the architecture never declares\. Withx¯\\bar\{x\}the mean input to the reading layer,u=x¯/\(x¯⋅x¯\)u=\\bar\{x\}/\(\\bar\{x\}\\cdot\\bar\{x\}\)satisfiesu⋅x≈1u\\cdot x\\approx 1across positions, and subtractingt​ut\\,ufrom a read row subtractsttfrom its pre\-activation:

r^=r−t​u,t=Qq​\(\{r⋅xp:p∉W\}\),\\hat\{r\}\\;=\\;r\\;\-\\;t\\,u,\\qquad t\\;=\\;Q\_\{q\}\\\!\\left\(\\left\\\{\\,r\\cdot x\_\{p\}\\;:\\;p\\notin W\\right\\\}\\right\),\(24\)with the threshold taken as a quantile of the row’s own projection over positions the edit is not aimed at, so it is measured rather than tuned against the target\.

Figure 14:One installed pair, end to end\. The head marks the positions, the stream carries the marker two layers without turning it, and the unit answers there\. Ranks are for the target token at the positions the head fires on and at all other positions\.Figure[14](https://arxiv.org/html/2609.10210#S4.F14)shows an instance\. A head at layer eight fires where’ did’occurs; a unit at layer ten answers’ good’, which moves from rank920920to rank33where the head fires while its rank elsewhere goes from17511751to18161816\. The cost is0\.1110\.111nats of held\-out loss\.

The same construction answers a condition carried from elsewhere in the sequence, which is the installation deferred from Section[4\.3](https://arxiv.org/html/2609.10210#S4.SS3)\. Over2424triples and2,4002\{,\}400trials on the baseline, with the correction of Section[5\.10](https://arxiv.org/html/2609.10210#S5.SS10)applied, the target reaches median rank2929whereCChas occurred against4,9704\{,\}970elsewhere, a separation of171171\. On9696percent of triples some setting brings the target into the top ten there, and on5454percent it does so inside the ten percent loss budget\. Every triple that clears the budget also leaves the target’s rank elsewhere alone, and the cheapest setting that does so costs0\.0290\.029nats\.

Reporting a construction like this at a single operating point hides the shape of what it buys, so the frontier is the object to report: for each loss budget, the best rank any setting achieves within it\. Over the full cross of write strength, read gain, saturation and read\-row calibration, the baseline brings the target into the top ten for0\.1210\.121nats, and for0\.1790\.179once the specificity criterion is imposed \(Figure[27](https://arxiv.org/html/2609.10210#S5.F27)\)\. Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)reports the same frontier for the activation arms\.

If the circuit is limited by the model, the amount of data used to measure the read row should barely matter; if by the estimate, a great deal\.

Figure 15:Separation achieved by the cross\-layer circuit, against the number of positions used to form the measured read row, with every other setting held fixed\.Figure[15](https://arxiv.org/html/2609.10210#S4.F15)answers it\. Separation rises from5\.15\.1at two positions to71\.671\.6at the full sample, monotonically and without turning over at the largest sample available\. The binding constraint on a circuit installed this way is the accuracy of one’s estimate of the destination frame, which is a measurement that can be improved rather than a property of the model that cannot\.

### 4\.7A reader is what makes off\-axis content visible

The claim that a write outside the readout frame is inert until something reads it can be put to a single test, with both halves and their controls in one measurement\. A direction is written at layer eight at1010percent of positions\. An unrelated target token is then scored three ways: with the write alone, with a reading unit at layer ten alone, and with both\.

Figure 16:Promotion of an unrelated target token at the written positions, relative to each arm’s own unedited baseline, for four kinds of written direction\. The reading unit at layer ten has a measured, thresholded read row and writes the target’s readout direction\.Figure[16](https://arxiv.org/html/2609.10210#S4.F16)depicts the result\. The reading unit alone leaves the target where it was, at1\.01\.0for every kind of write, which is the threshold of Equation[24](https://arxiv.org/html/2609.10210#S4.E24)doing its work\. The write alone moves the target by1\.11\.1times\. The two together move it by3\.53\.5, for0\.130\.13nats\. The largest effect belongs to the private direction, and the two model\-derived directions give the least, which is consistent with the reader having to find a frame the model is also acting on\.

### 4\.8Driving a component the model already has

Everything to this point installs both ends of a circuit\. A unit the model trained can be driven the same way, with nothing installed, through the row it already reads with\. A unit’s pre\-activation at depthℓ\\ellisau=Wa​\[u\]⋅\(g⊙LN⁡\(h\)\)a\_\{u\}=W\_\{a\}\[u\]\\cdot\\left\(g\\odot\\mathrm\{LN\}\(h\)\\right\), withggthe layer’s norm gain\. To first order in a perturbation of the stream,

∂au∂h∝Π⟂𝟏​\(Wa​\[u\]⊙g\),\\frac\{\\partial a\_\{u\}\}\{\\partial h\}\\;\\propto\\;\\Pi\_\{\\perp\\mathbf\{1\}\}\\\!\\left\(W\_\{a\}\[u\]\\odot g\\right\),\(25\)the read row multiplied elementwise by the gain and projected off the constant direction, since layer normalization removes the mean and a component along𝟏\\mathbf\{1\}adds to∥h∥\\lVert h\\rVertwithout reaching the unit\.444Both details matter in practice\. Dividing by the gain rather than multiplying, or retaining the constant component, produces a direction that suppresses the unit it was meant to excite\.Units are selected by the margin between the top two tokens their write column decodes to, which picks out components that answer with a single token\.

Whether the write reaches the target through the unit is a mediation question\. Writingr0r\_\{0\}andrrfor the target’s rank before and after the write in the intact model, andr0ablr\_\{0\}^\{\\text\{abl\}\}andrablr^\{\\text\{abl\}\}for the same pair with the unit’s write column zeroed,

m=1−r0abl−rablr0−rm\\;=\\;1\\;\-\\;\\frac\{r\_\{0\}^\{\\text\{abl\}\}\-r^\{\\text\{abl\}\}\}\{r\_\{0\}\-r\}\(26\)is the share of the promotion that the tapped unit carries\. Comparing the ablated arm against the intact model’s baseline rather than its own inflatesmmwithout bound, because ablating a unit that normally promotes the target moves that baseline\.

Figure 17:An instance of the tap\. The write at layer eight uses the layer\-ten unit’s own read row as its direction, and no component is installed anywhere\. Ablating that single unit removes ninety percent of the target’s promotion\.Figure 18:Share of the target’s promotion carried by the tapped unit, by how the driving direction was obtained, over twelve units per model at four write strengths\. The random control moves neither the unit nor the target, so it has no effect to mediate\.Figures[17](https://arxiv.org/html/2609.10210#S4.F17)and[18](https://arxiv.org/html/2609.10210#S4.F18)report the result\. A write at layer eight raises a layer\-ten unit’s activation by24\.124\.1standard deviations and moves the token that unit writes from rank11161116to rank3232, for0\.0490\.049nats\. Ablating that one unit takes the same write from981981to869869, so ninety percent of the promotion travels through it\. Over twelve units the figure is0\.860\.86\.

What makes this work is driving the unit in the frame it already reads in\. A direction taken from the weights through Equation[25](https://arxiv.org/html/2609.10210#S4.E25)gives0\.860\.86, where one estimated from the unit’s activations gives0\.550\.55\. A native unit’s read row is available exactly, inWaW\_\{a\}, and estimating it from activations only adds noise\.

### 4\.9Scoring against a rank\-one editor

We implemented the rank\-one update of[Meng et al\. \[36\]](https://arxiv.org/html/2609.10210#bib.bib16)and scored it under the criterion used above\. Writingk∗k\_\{\*\}for the key,v∗v\_\{\*\}for the value andC=E⁡\[k​k⊤\]C=\\operatorname\{E\}\\\!\\left\[kk^\{\\\!\\top\}\\right\]for the second moment of the layer’s keys over a corpus sample, that update is

W^=W\+\(v∗−W​k∗\)​\(C−1​k∗\)⊤\(C−1​k∗\)⊤​k∗,\\hat\{W\}\\;=\\;W\\;\+\\;\\frac\{\\left\(v\_\{\*\}\-Wk\_\{\*\}\\right\)\\left\(C^\{\-1\}k\_\{\*\}\\right\)^\{\\\!\\top\}\}\{\\left\(C^\{\-1\}k\_\{\*\}\\right\)^\{\\\!\\top\}k\_\{\*\}\}\\,,\(27\)which spreads the write across the layer alongC−1​k∗C^\{\-1\}k\_\{\*\}rather than loading it into one unit\. The valuev∗v\_\{\*\}is obtained by gradient descent on the target’s likelihood\.

Equation[27](https://arxiv.org/html/2609.10210#S4.E27)has two ingredients, and separating them is informative\. The covariance term is what preserves specificity: replacingC−1C^\{\-1\}by the identity brings the target to rank five on the baseline while promoting it seventy times as far everywhere else\. The gradient is the expensive one\. Substituting a closed\-form value, the target’s own readout direction, in place of the optimizedv∗v\_\{\*\}brings the target into the top ten for0\.0680\.068nats, where the gradient version reaches rank one for0\.6110\.611\. The part of the method that carries its specificity is the fast part to compute\.

Figure 19:Reach against held\-out loss for the install of Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)and for the rank\-one update of Equation[27](https://arxiv.org/html/2609.10210#S4.E27), both at layer ten and both swept over their own strength\. The dashed curve substitutes a closed\-form value for the gradient\-optimized one; the marker is the method with its gradient value\. Specificity is unconstrained in these panels and reported in the text\.Figure[19](https://arxiv.org/html/2609.10210#S4.F19)places the two on the same axes, with each swept over its own strength so the comparison is curve against curve\. On the baseline, reaching the top ten costs the single\-unit install0\.00160\.0016nats against0\.06810\.0681for the closed\-form rank\-one update, a factor of forty, and the target’s rank elsewhere is1\.021\.02times where it started against0\.810\.81\. The published method, with its gradient value, reaches rank one for0\.6110\.611nats and holds specificity at0\.800\.80\. Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)runs the same comparison across the activation arms\.

Two differences of kind survive the comparison, and they run in opposite directions\. A rank\-one update needs a corpus sample to estimateCCand a gradient pass to obtainv∗v\_\{\*\}, where both halves of the install here are read from the parameters\. Against that, the rank\-one update writes through the whole layer and therefore needs no spare component, where an install into a single unit consumes one\. The measurement supports treating them as complementary\. The covariance term is worth adopting on its own terms wherever a single unit cannot be silenced cheaply enough for a one\-unit install to stay local, which is the situation the shaped activations of Section[5](https://arxiv.org/html/2609.10210#S5)present\.

A name, then, is an address that can be written to\. A gain on one component moves the tokens its parameters name and nothing else, and pushing harder buys no collateral\. An association the model does not hold installs into one spare unit, key and value both read from the parameters, for a quarter of a percent of held\-out loss\. A unit the model trained for itself is driven from two layers below through the row it already reads with, and carries eighty\-six percent of the effect\. A circuit across two layers runs on a direction drawn at random, which the stream carries untouched because this model works in a quarter of its width, and what limits the circuit is the estimate of the reader’s frame rather than anything in the model\. Against the rank\-one editor the install reaches the top ten for a fortieth of the cost, and the covariance term is the part of that method worth keeping\. A rank\-one update replaces this with that\. A circuit is a mechanism, and the components here are read, built and driven in the model’s own terms\.

## 5What the activation function changes

Sections[2](https://arxiv.org/html/2609.10210#S2)to[4](https://arxiv.org/html/2609.10210#S4)took a conventional transformer as it comes\. This section asks what changes if the activation function is chosen for legibility rather than inherited\. Four arms are trained beside the baseline, matched to it on everything but the unit: sigmoid, softplus, ReLU, and a fourth built from bounded set operators with two operands per unit\. The diagnosis comes first, on the baseline’s own pre\-activations, since it says which functions to try\. Then the arms: their quality, the size of their circuits at one hop and at closure, how well their parameters read, which property of the function was responsible, and what each will and will not let an edit do\.

### 5\.1Why a conventional unit stops being readable

Section[3](https://arxiv.org/html/2609.10210#S3)left a diagnosis, and the intervention follows from it\. The reading procedure orders tokens bywu⋅E~​\[l,t\]w\_\{u\}\\cdot\\tilde\{E\}\[l,t\], which estimates the ordering by*pre\-activation*\. The causal ground truth of Equation[14](https://arxiv.org/html/2609.10210#S3.E14)orders by activation*averaged over contexts*\. Between the two sit the activation function and an average, and either can reorder\. Take a trained baseline, capture its pre\-activations on the same contexts and candidate tokens, and hold the model, the weights, the tokens and the predictor fixed while changing only the function applied before the average\.

Every order\-preserving choice reads at9696to9999percent and is flat inKK\(Table[22](https://arxiv.org/html/2609.10210#S5.T22)\)\. GELU reads at6767and decays\. The ordering the weights supply is very nearly recoverable from the averaged behavior, unless the function in between is GELU\.

The cause is visible in the function itself\. GELU decreases belowx∗≈−0\.752x^\{\*\}\\approx\-0\.752: on that interval a*larger*wu⋅zw\_\{u\}\\cdot zproduces a*smaller*activation, so the ordering the weights supply is not blurred there but reversed\. Across three converged seeds,59\.859\.8to62\.062\.0percent of all pre\-activation mass sits belowx∗x^\{\*\}\(Figure[20](https://arxiv.org/html/2609.10210#S5.F20)\)\. Nearly two thirds of the time, a unit is operating on the part of its activation function that runs backward\.

That explains the shape of Figure[7](https://arxiv.org/html/2609.10210#S3.F7)as well as its level\. SmallKKselects the strongest inputs, which are the positive excursions, where GELU is increasing and reads at6767percent\. LargeKKreaches the body of the set, where most of the mass lives, which is the reversed interval, and that is exactly where agreement crosses below chance\. A model that looked like it had a few nameable inputs and an incoherent remainder is reproduced, on the same model, by the readout function alone\.

This is an ablation of the reading rather than a prediction about training: a network trained with a differentϕ\\phiarrives at a different distribution of pre\-activations, and nothing here guarantees the property survives\. And the predictor is the pre\-activation mean rather thanwu⋅E~w\_\{u\}\\cdot\\tilde\{E\}, which isolatesϕ\\phicleanly at the cost of putting these numbers on a different scale from the rest\. Section[5\.3](https://arxiv.org/html/2609.10210#S5.SS3)trains the monotone model this predicts and reports what happens\.

Table 22:One trained baseline model, one set of pre\-activations, one predictor\. Only the function applied before averaging over contexts differs\. Chance\-corrected agreement, three converged seeds, context length 2048\.Figure 20:Where the baseline actually operates on its activation function\. The curve is GELU; the shaded density under it is the*measured*distribution of pre\-activationswu⋅zw\_\{u\}\\cdot zover a trained baseline,1\.2×1091\.2\\times 10^\{9\}samples at context length 2048\. GELU decreases belowx∗≈−0\.752x^\{\*\}\\approx\-0\.752, and63\.363\.3percent of the mass lies there on this denser estimate, against the59\.859\.8to62\.062\.0percent the per\-seed estimate in the text gives, so for most inputs a larger pre\-activation produces a smaller activation and the ordering the weights supply is reversed\. Section[5\.3](https://arxiv.org/html/2609.10210#S5.SS3)puts a monotone function beside this one\.Neither observation proves the activation is responsible\. Changing it and holding everything else fixed is how to find out, and Section[6](https://arxiv.org/html/2609.10210#S6)later supplies an independent check: among twelve models trained by other people, the one built on ReLU has by far the least cancellation between opposing contributions\.

### 5\.2Which activations make sense

For the ordering the parameters supply to survive into the ordering the causal measurement sees,ϕ\\phimust at least preserve order: ifϕ\\phiis strictly increasing, then within any single context the two rankings are identical, and the only remaining discrepancy comes from averaging\. GELU is not strictly increasing, and most of a trained model’s pre\-activation mass sits where it decreases\.

Two standard functions satisfy the requirement and sit on opposite sides of a second property that might plausibly matter\. The logistic function

ϕσ​\(z\)=11\+e−z∈\(0,1\)\\phi\_\{\\sigma\}\(z\)=\\frac\{1\}\{1\+e^\{\-z\}\}\\in\(0,1\)\(28\)is strictly increasing and*bounded*, so a unit’s mean activation on a token is the fraction of contexts in which it fired, and no single context can dominate\. Softplus

ϕ\+​\(z\)=log⁡\(1\+ez\)∈\(0,∞\)\\phi\_\{\+\}\(z\)=\\log\\\!\\big\(1\+e^\{z\}\\big\)\\in\(0,\\infty\)\(29\)is strictly increasing and*unbounded*, and behaves like the identity for largezz\. Both are injective\. If order\-preservation is what matters they should behave alike; if boundedness is what matters they should not\.

A third, ReLU, is monotone but not injective: it maps the entire negative half\-line to zero, so all the tokens a unit is silent on become indistinguishable in its output\. It is the control that separates*order\-preserving*from*information\-preserving*\.

A fourth alternative arrives from a different direction and meets the requirement twice over\. Three earlier papers built feed\-forward layers out of bounded fuzzy set operators —A∩BA\\cap BandA∖BA\\setminus Bon operands squashed into\[0,1\]\[0,1\]— on the argument that a unit computing a named logical operation on two named sets is legible by construction\[[42](https://arxiv.org/html/2609.10210#bib.bib9),[40](https://arxiv.org/html/2609.10210#bib.bib10),[39](https://arxiv.org/html/2609.10210#bib.bib8)\]\. Every unit computes a pair of operands from the residual stream,

Au​\(z\)=σ⁡\(wua⋅z\),Bu​\(z\)=σ⁡\(wub⋅z\),A\_\{u\}\(z\)=\\sigma\\big\(w^\{a\}\_\{u\}\\cdot z\\big\),\\qquad B\_\{u\}\(z\)=\\sigma\\big\(w^\{b\}\_\{u\}\\cdot z\\big\),\(30\)and writes the two operations into separate columns,

FFN⁡\(z\)=∑u\[Au​Bu⏟A∩B​cu∩\+Au​\(1−Bu\)⏟A∖B​cu∖\]=∑uAu​\[cu∖\+Bu​\(cu∩−cu∖\)\]\.\\mathrm\{FFN\}\(z\)=\\sum\_\{u\}\\Big\[\\underbrace\{A\_\{u\}B\_\{u\}\}\_\{A\\cap B\}\\,c^\{\\cap\}\_\{u\}\\;\+\\;\\underbrace\{A\_\{u\}\(1\-B\_\{u\}\)\}\_\{A\\setminus B\}\\,c^\{\\setminus\}\_\{u\}\\Big\]\\;=\\;\\sum\_\{u\}A\_\{u\}\\Big\[\\,c^\{\\setminus\}\_\{u\}\+B\_\{u\}\\big\(c^\{\\cap\}\_\{u\}\-c^\{\\setminus\}\_\{u\}\\big\)\\Big\]\.\(31\)Both operands are bounded and strictly increasing in their pre\-activations\. The rearrangement on the right says what the two are for:AAis a gate deciding*how much*the unit writes andBBis a selector deciding*which*of two fixed columns is written\. The parameter budget is held to the baseline’s by narrowing the layer, since each unit now carries two read rows and two write columns\.

Those papers could not run the construction everywhere\. Below roughly half conversion, training was indistinguishable from a conventional model; above it training diverged\. The arm reported here has no conventional units left in it, which is what makes it a clean fourth point rather than a mixture of two designs\. What removed the ceiling is an auxiliary objective borrowed from separate work on where a transformer holds its intermediate content\[[41](https://arxiv.org/html/2609.10210#bib.bib14)\]: it prescribes, layer by layer, the angle between a layer’s residual decode and the final prediction — orthogonal through the first half of the stack, aligned through the second\. Why it works is not explored here; that it does is eight seeds of eight converging \(Section[5\.4](https://arxiv.org/html/2609.10210#S5.SS4)\)\. Appendix[B](https://arxiv.org/html/2609.10210#A2)gives the construction, the objective and the recipe in full\.

### 5\.3The construction

Everything in the baseline of Section[2\.1](https://arxiv.org/html/2609.10210#S2.SS1)is retained — depth, width, head count, parameter budget, data, schedule — and the unit is

au​\(z\)=ϕ⁡\(wu⋅z\),FFN⁡\(z\)=∑uau​\(z\)​cu,a\_\{u\}\(z\)=\\phi\\big\(w\_\{u\}\\cdot z\\big\),\\qquad\\mathrm\{FFN\}\(z\)=\\sum\_\{u\}a\_\{u\}\(z\)\\,c\_\{u\},\(32\)withϕ\\phione of Equations[28](https://arxiv.org/html/2609.10210#S5.E28)or[29](https://arxiv.org/html/2609.10210#S5.E29), or ReLU; the set\-operator arm replaces Equation[32](https://arxiv.org/html/2609.10210#S5.E32)by Equation[31](https://arxiv.org/html/2609.10210#S5.E31)and is otherwise identical\. Read row, write column and the reading procedures of Sections[3\.1](https://arxiv.org/html/2609.10210#S3.SS1)and[3\.2](https://arxiv.org/html/2609.10210#S3.SS2)are unchanged, which is the point: the same instrument is pointed at a model differing only in what a unit does with its pre\-activation\.

Two further departures from the baseline are part of the construction\. The attention heads pass their value vectors through a sigmoid, so what a head gathers from a position is a bounded quantity in\[0,1\]\[0,1\]rather than an unbounded projection\. The weighting over positions is the ordinary softmax, unchanged from the baseline\. That squashing is the same in the sigmoid, softplus and ReLU arms whateverϕ\\phiis, so it is held fixed across the comparison rather than varying with the activation function; the set\-operator arm bounds what its heads gather too, by applying Equation[31](https://arxiv.org/html/2609.10210#S5.E31)’s operations there as well\. And a small auxiliary term rewards a bounded activation for sitting near either end of its range rather than in the middle, at weight0\.0030\.003against the language\-modeling loss \(Appendix[B](https://arxiv.org/html/2609.10210#A2)defines it\)\. That term is defined only for a bounded activation, so the softplus and ReLU arms carry it at zero and the sigmoid arm is trained both ways; the two sigmoid variants are equivalent on every measurement here, so the cross\-activation comparison is not confounded by it\. Section[5\.8](https://arxiv.org/html/2609.10210#S5.SS8)separates the contribution of these two, neither of which is an activation function, fromϕ\\phi’s\.

### 5\.4Quality

Eight seeds of the sigmoid construction were trained under Section[2\.1\.1](https://arxiv.org/html/2609.10210#S2.SS1.SSS1)’s protocol and screened by the same criterion, and they behave the same way the baseline did: five converged and three did not\. Whatever makes a run of this size fail at this budget is not something the activation function introduced or removed\.

Table 23:The sigmoid construction, eight seeds at one epoch, screened by the criterion of Section[2\.1\.1](https://arxiv.org/html/2609.10210#S2.SS1.SSS1)and reported the same way as Table[1](https://arxiv.org/html/2609.10210#S2.T1)\. The non\-converged seeds are printed for the same reason they were there\.ov​@​8\\mathrm\{ov\}@8here is taken under the ranked summary of Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4), raw rather than chance\-corrected\.Table 24:Quality across activation functions at one epoch\. The three arms whose operands are strictly increasing sit inside the baseline’s converged range on both measures\. ReLU sits outside it on both\. Every row below the rule comes from an arm matched to the baseline on width, data and schedule\.Table[24](https://arxiv.org/html/2609.10210#S5.T24)settles quality\. Both injective activations sit inside the baseline’s converged range on perplexity and on LAMBADA, and softplus was trained at eight seeds of which all eight pass the screen\.

The set\-operator arm sits there too, at eight seeds of which all eight converged:19\.2319\.23to19\.4319\.43in perplexity, inside the baseline’s range and tighter than it, and0\.2340\.234to0\.2740\.274on LAMBADA\. Eight of eight landing within0\.20\.2nats of each other is not the signature of a model training at the edge of stability, which is the state prior work left off at\[[42](https://arxiv.org/html/2609.10210#bib.bib9)\]\.

While the ranges overlap, the medians do not coincide\. The baseline leads sigmoid and softplus by about a point of accuracy, in a consistent direction, on samples of five, five and eight seeds whose ranges span two to four points each\. A real cost of that size cannot be ruled out at this sample\. What the experiment does establish is that replacing the activation function costs nothing like what ReLU costs\.

ReLU is not like other models\. Its seeds sit about ten percent behind every other arm on perplexity and below every converged seed of every other construction on accuracy\. It trains stably to a ceiling and settles lower\. The one function in this comparison that discards information rather than merely reshaping it is the one that cannot reach the quality the others reach, which is the first indication that injectivity is doing work beyond bookkeeping\.

Every model trained for this paper sits on one pair of axes in Figure[21](https://arxiv.org/html/2609.10210#S5.F21), quality against legibility\. Read Table[23](https://arxiv.org/html/2609.10210#S5.T23)straight down and the legibility column rises monotonically as quality falls, every excluded seed scoring above every converged one\. That is why the sigmoid figures quoted below come from the converged seeds alone\.

Figure 21:Every trained model in this paper\. Horizontal axis is quality; vertical axis is how well a unit’s parameters name the tokens that causally drive it*in the vocabulary basis*, both under one protocol\. Read in a different basis, the same units order their drivers well above chance \(Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)\); what this axis measures is whether the token table is the right place to look, not whether anything is there\. The shaded band is the baseline’s converged quality range\. The baseline sits low and to the right — good quality, poor legibility — and every order\-preserving arm sits about twice as high at the same quality; ReLU sits higher still and to the left, buying legibility with accuracy\. In*both*families the pale points, the seeds that failed the convergence screen, sit*above*their own converged siblings without exception, which is why quality is on this axis at all\.
### 5\.5Circuit size across the arms

The instrument of Section[2](https://arxiv.org/html/2609.10210#S2)applies to these constructions unchanged, and this is the cleanest comparison available: the arms share a width, a dataset, a schedule and a converged quality, and differ only inϕ\\phi\. Whatever separates them separates because of the activation function\.

Table 25:The accounting of Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)across the arms\. The inflation column is what the absolute\-value convention would report divided by what the signed one does\. Every arm cancels heavily, the ReLU arm least, by a factor of five\.Figure 22:Left: the three denominators across the arms, on a log axis\. The gap between the signed count and the other two is the same two\-to\-three orders of magnitude found in the baseline, and it is present in every arm — choosing the activation changes the size of the circuit, not whether the convention matters\. Right: circuit size against how confident the model is in its own prediction\. The arms keep their order at every confidence, so the comparison does not depend on where the confidence floor of Table[14](https://arxiv.org/html/2609.10210#S3.T14)is set\.Figure 23:Removing thekkhighest\-contributing components, across the arms\. The random control is pooled over all of them\. Replacing the conventional activation moves the curve left: the median prediction turns over at a quarter of thekk\.Table 26:Circuit size and the point at which the median prediction changes, with the range across seeds\. Four to eight seeds per arm where several were trained\.The accounting behaves the same way in every arm and differs only in what it counts \(Table[25](https://arxiv.org/html/2609.10210#S5.T25)\)\. Two things separate \(Table[26](https://arxiv.org/html/2609.10210#S5.T26)\)\. The circuit is smaller — twenty\-six to thirty\-six components against the baseline’s fifty\-three — and it turns over sooner, the median prediction changing at a quarter of thekkthe baseline needs \(Figure[23](https://arxiv.org/html/2609.10210#S5.F23)\)\. Both hold across every seed trained, and Figure[22](https://arxiv.org/html/2609.10210#S5.F22)shows they hold across the confidence range rather than only at the floor these counts are measured at\. The set\-operator arm, measured under the same procedure and not in these two floats, sits with the rest:3636components at ninety percent of the net and an inflation of277277times against the baseline’s5353and452452\.

### 5\.6The whole dependency graph

Everything above counts components contributing directly to the readout — one hop\. The recursion of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)follows the graph the rest of the way, and reports two things: the closure, which is what a prediction draws on, and the required graph, which is what ablation can reduce it to\. Every arm closes, at four to six levels, on ninety to a hundred predictions each\.

Table 27:The backward dependency graph of a prediction, closed\. “Attn heads” is how many distinct \(layer, head\) pairs the closure touches, out of the model’s total: attention components are named locations and are ablated with the rest, but are not followed\. “Ablated” removes the closed graph, against a size\-matched random set\.Table 28:The required graph: the smallest prefix of the one\-hop set whose removal changes the prediction, found by scanning prefix lengths upward on each prediction separately\. Bisection is not available here: the flip is not monotone in the prefix length, and on1111to2828percent of predictions a larger prefix restores a prediction a smaller one broke\. “Defined on” is the share of predictions for which such a prefix exists at all; the columns to its left are medians over those\.At one hop the arms sit within a factor of two of the baseline\. At closure they separate from it by four to seven times \(Table[27](https://arxiv.org/html/2609.10210#S5.T27)\)\. The recursion exposes an effect of the activation function on how much machinery a prediction draws on that a one\-hop measurement mostly misses\. The set\-operator arm closes on120120components in six levels, between the sigmoid arm’s9999and the ReLU arm’s182182\.

The required graph separates them too, at a small absolute size\. Removing1313of the baseline’s components changes its prediction; for the arms it is33to55\. A prediction draws on hundreds of components and cannot survive losing a handful of them, and the handful is smaller when the activation is order\-preserving\.

The branching factor compounds far less than multiplication would suggest, because the sources of different nodes overlap: multiplyingn90n\_\{90\}by it overstates the measured two\-level count by a factor of four, so the counts here are the union taken at every level rather than a product\.

Two caveats from Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)apply to the arms as they do to the baseline, one of them harder\. Cancellation arriving at a component is far more severe than cancellation leaving the model on the arms too, though by several hundred times rather than the baseline’s thousand\. And a signed count of what reaches a node is undefined wherever that drive sums to zero or below, which is a third of the baseline’s nodes but more than half of the sigmoid model’s, so the branching medians above are taken over the nodes where the quantity exists and rest on a smaller share of the sigmoid model’s graph than of the baseline’s\.

### 5\.7Legibility

Figure 24:Agreement between the parameter\-derived set and the causal set, against how much of the set is requested\. The baseline curve is the one from Figure[7](https://arxiv.org/html/2609.10210#S3.F7)\. The three arms built on order\-preserving activations are flat inKKand sit near the dashed reproducibility ceiling — the rate at which the causal measurement agrees with*itself*on independent contexts, which no reading procedure can exceed — and they land on each other\. Set\-operator agreement is taken on operandAA\. Context length 2048, candidate pool\|V\|=512\|V\|=512, table and causal profile from 16 disjoint contexts each\.The measurement of Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2.SSS0.Px1)wasov^​@​K\\widehat\{\\mathrm\{ov\}\}@Kswept overKK, and it is repeated here unchanged\. The baseline decays from58\.958\.9percent atK=8K=8to−9\.5\-9\.5atK=256K=256\(Figure[24](https://arxiv.org/html/2609.10210#S5.F24)\)\. The other three do the opposite: they*rise*slightly and then flatten, sigmoid from78\.078\.0to82\.482\.4, softplus from80\.080\.0to82\.482\.4and the set\-operator arm from78\.878\.8to82\.882\.8across the same sweep\. AtK=256K=256, where the baseline selects tokens overlapping the true set less often than a random draw would, they recover it at better than four times chance\.

The flatness and how little separates the three matter more than the level\. A procedure that reads the strongest few inputs and then fails produces a decaying curve, which is what the baseline gives\. A procedure that recovers the whole set produces a flat one\. Section[5\.1](https://arxiv.org/html/2609.10210#S5.SS1)predicted exactly this from the untrained readout ablation — every monotone choice was flat inKKon the baseline’s own pre\-activations — and training with those activations reproduces it\. Figure[25](https://arxiv.org/html/2609.10210#S5.F25)rules out the operating point as the explanation: the sigmoid model’s pre\-activations sit further into the tail than the baseline’s, and its reading is unaffected\.

Figure 25:The sigmoid arm’s operating point, drawn on the same axes and to the same recipe as the baseline’s in Figure[20](https://arxiv.org/html/2609.10210#S5.F20): the curve is the function and the shaded density under it is the measured distribution of pre\-activations over a trained model of that arm\. Its mass sits further left than the baseline’s, and it does not matter, because a monotone function preserves the ordering wherever its argument happens to land\. It is not the operating point that causes the baseline’s reversal — it is that GELU has an interval where the ordering can be lost at all\.The qualitative change behind those numbers is that the sigmoid model’s token sets support a description where the baseline’s do not — places, or the copula cluster around*as*and*is*\. Table[47](https://arxiv.org/html/2609.10210#A3.T47)sets the two side by side in Appendix[C](https://arxiv.org/html/2609.10210#A3)\.

##### The numbers are at the instrument’s ceiling\.

A reading procedure cannot agree with the causal set better than the causal set agrees with itself, and that self\-agreement is finite because it is estimated from a sample of contexts\. Measured on independent halves and chance\-corrected, the baseline’s causal top\-8 reproduces itself at72\.472\.4percent and its top\-256 at69\.669\.6; the sigmoid model reproduces at79\.279\.2and81\.681\.6\. Reading each model against its own ceiling:

The sigmoid model is at its ceiling, and so are the other two: softplus and the set\-operator arm end the sweep at82\.482\.4and82\.882\.8against sigmoid’s82\.482\.4, differences smaller than the ceiling can resolve, so nothing here separates the three from each other\. The residual disagreement is the measurement’s own noise, and collecting more contexts would raise both numbers together\. The claim is that the parameters recover the unit’s set as well as the set can be resolved, which is stronger than a ratio between two models\.

##### What the gain costs\.

The obvious objection to these numbers is the one Appendix[C](https://arxiv.org/html/2609.10210#A3)raised first: a model reads as more legible when it has less to say\. The dimensionality column of Table[23](https://arxiv.org/html/2609.10210#S5.T23)answers it\. The converged sigmoid seeds carry*more*independent directions over the first five layers than the baseline does, not fewer, while scoring roughly twice as high on agreement\. Whatever the change did, it did not do it by making the representation smaller\.

The same column also disciplines the excluded seeds\. Two of the three collapsed the way the baseline’s did, and their inflated scores are explained\. The third, seed 6, is healthy on that column and still scores above every converged seed\. Collapse does not account for its elevated legibility, which is unexplained\. It is excluded on quality\.

##### Smaller sets, read through the graph\.

Agreement says the parameters name the right tokens without saying the tokens belong together\. By Equation[48](https://arxiv.org/html/2609.10210#A3.E48)the median effective set size falls by about a third under sigmoid, a real but modest concentration\. Coherence over the set as a whole stays out of reach in the token basis \(Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2.SSS0.Px2)\), so the graph supplies it instead\. With the tokens sorted by which upstream source supplies each, a classifier assigns a token to its source well above a shuffled control, and part\-of\-speech purity within a source group exceeds a matched random partition\. Both effects are larger than on the baseline \(Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)\)\.

The same graph names the components the vocabulary leaves off its frame, separating the arms less cleanly than agreement does\. Describing a component from what its incoming edges carry recovers something it measurably fires on far more often than a control does, in every arm and in the baseline alike \(Table[29](https://arxiv.org/html/2609.10210#S5.T29)\)\. Ranked by that controlled margin, only sigmoid and the set\-operator arm clear the baseline\. The set\-operator arm’s10\.010\.0times on components that name themselves is the largest margin in the table and the least well sampled number in it:4242such components against the baseline’s119119, and a control of55percent against every other model’s1313to2020\. Softplus reads highest of any arm before the control is applied, and its control is twice every other model’s because its edge labels repeat — the ten commonest hold a quarter of its label pool where the others hold an eighth — so a random description hits more often and the corrected margin collapses\. Read through the graph rather than the vocabulary, replacing the activation buys a large improvement in two arms and nothing reliable in the other two\.

Table 29:Describing a component by what its inputs carry\. A six\-token description is built from the incoming edges that pass the frame\-agreement test of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1), weighted by each edge’s share of the node’s incoming drive, and scored on whether it contains a token the component measurably fires on\. The control builds the same description from edge labels drawn elsewhere in the same model\. “Names itself” splits the components by whether their own write places the predicted token in its first hundred\.The two measurements ask different questions\. Agreement asks whether a unit’s own parameters pick out the tokens it fires on, and the construction was designed to fix exactly that; the graph asks whether a component’s*inputs*describe it, which the activation function was never aimed at\. The arms separate on the first alone, which places the construction’s effect on the read side of a component rather than on its position in the graph\.

A component whose own write is off the token frame stays readable in every arm\. Its incoming edges carry tokens, and those tokens describe it: the description contains something the component fires on5656percent of the time against a control at2020, and the rate is the same for components that can name themselves and components that cannot \(6060against5656percent, Table[29](https://arxiv.org/html/2609.10210#S5.T29)\)\. What a component’s own parameters say about it does not predict whether its inputs describe it\. The description is weak in absolute terms — it covers1717percent of what the component responds to — and it is a label rather than a name, but it is available directly\.

##### Two operands are two things to read\.

Equation[15](https://arxiv.org/html/2609.10210#S3.E15)scores one read row against one causal set, so a unit with two readable operands and a unit with one readable row are indistinguishable to it however differently they are organized\. Both halves of every unit are readable, and both at roughly twice the baseline:68\.568\.5to72\.072\.0for operandAAand64\.764\.7to68\.968\.9for operandBB, against the baseline’s single row at33\.633\.6to38\.438\.4, all under the ranked summary of Appendix[C\.4](https://arxiv.org/html/2609.10210#A3.SS4)\. They also carry different jobs rather than duplicating one\. Take each unit’s top\-40 tokens byAAand split them by whetherBBis high or low: coherence over the centered token embeddings rises by0\.01660\.0166, a3131percent increase over the unsplit set, where a random split of the same tokens into halves of the same sizes moves it by0\.00020\.0002\. The comparison is between two ways of splitting the same tokens against a control that splits them at random, so a similarity measure blind to some concept is blind to it in both arms equally\. What it establishes is thatBBseparatesAA’s set along an axis the embedding can see rather than one a person would name\.

### 5\.8Which property was responsible

Sigmoid and softplus differ in boundedness and agree on everything measured here: quality within each other’s range, agreement flat inKKat8282, both at their ceiling\. The operative property is that the function does not destroy the ordering the weights supply, which also disposes of the more intuitive explanation, that a bounded unit is legible because its activation reads as a probability\.

The set\-operator arm makes the same point from the far end of the design space\. It carries two read rows, two write columns and a named logical operation per unit, where Equation[32](https://arxiv.org/html/2609.10210#S5.E32)carries one row, one column and a swapped function and is what a feed\-forward unit already was\. On theKKsweep the two are within a point of each other everywhere; on quality they overlap; at closure they sit twenty\-one components apart\. If the question is how much better the parameter\-derived set matches the causal set, the elaborate construction is not better\. Three constructions with nothing in common but an order\-preserving activation land on one curve, and a fourth that is monotone but not injective lands somewhere systematically different, which is the strongest evidence here for which property is doing the work\.

ReLU makes the point from the other side\. It is order\-preserving and scores*highest*of all on agreement,75\.675\.6to79\.579\.5percent atK=8K=8against sigmoid’s66\.666\.6to71\.171\.1, softplus’s71\.271\.2to74\.874\.8and the set\-operator arm’s68\.568\.5to72\.072\.0, all under the ranked summary of Table[23](https://arxiv.org/html/2609.10210#S5.T23)\. It is also the only arm that loses quality, and its units respond to far fewer tokens: arity2\.42\.4to2\.72\.7against sigmoid’s5\.35\.3to6\.76\.7\. Its seeds also vary much more in how many units answer to nothing —0\.90\.9,4\.74\.7and10\.210\.2percent, against0\.20\.2to4\.14\.1for sigmoid — though those ranges overlap, so the dead\-unit count is not a finding on its own\. Discarding the negative half\-line makes what remains easier to read and leaves the model less to say\. That is a legibility–quality exchange rate, visible in a single controlled comparison, and it marks the edge of what this change buys for free\.

##### What the construction separates\.

A second reading of the same result does not concern the token ordering at all\. Run against the carriage background of Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6), the search returns1818components on the baseline and44on the sigmoid model\. The two are under one procedure, so the comparison between them is the measurement, and neither is to be read against the zeroed\-background sizes of Table[7](https://arxiv.org/html/2609.10210#S2.T7)\. The same construction that makes a unit readable also makes the prediction separable from the traffic it travels in\. Whether the two effects are one mechanism or two is not settled here\. The ordering result is measured on what drives a unit and this one on what the model delivers, and a construction could plausibly improve either alone\.

Sigmoid, softplus and set\-operator units are more legible than GELU units, then, and by the same property\. Whether that translates into editability is the next question\.

### 5\.9Editability

The standard promote edit comes first: add a multiple of a target token’s embedding to one unit’s write column,

cu←cu\+β​E​\[T\],c\_\{u\}\\;\\leftarrow\\;c\_\{u\}\+\\beta\\,E\[T\],\(33\)which credits the same mechanism as established weight\-editing methods\[[36](https://arxiv.org/html/2609.10210#bib.bib16)\]and changes nothing about the model but one column\. Two things are then measured on held\-out text\. The*efficacy*is how much the target token’s logit rises at the positions where the unit fires:

eff⁡\(u\)=𝔼fires​\[logitTedited−logitTorig\]\.\\mathrm\{eff\}\(u\)=\\mathbb\{E\}\_\{\\text\{fires\}\}\\big\[\\,\\mathrm\{logit\}\_\{T\}^\{\\text\{edited\}\}\-\\mathrm\{logit\}\_\{T\}^\{\\text\{orig\}\}\\,\\big\]\.\(34\)The*collateral*is how much everything else moved, as the mean divergence between the original and edited distributions at the positions where the unit does*not*fire:

coll\(u\)=𝔼not fires\[KL\(Porig∥Pedited\)\]\.\\mathrm\{coll\}\(u\)=\\mathbb\{E\}\_\{\\text\{not fires\}\}\\big\[\\,\\mathrm\{KL\}\\big\(P^\{\\text\{orig\}\}\\,\\\|\\,P^\{\\text\{edited\}\}\\big\)\\,\\big\]\.\(35\)Restricting collateral to non\-firing positions matters\. Averaging over all positions rewards a unit simply for being rare, since a unit that almost never fires disturbs almost nothing; asking what happened where the unit was*not*supposed to act removes that advantage\.

Neither number is meaningful alone, as Section[1](https://arxiv.org/html/2609.10210#S1)noted: any edit is respected if pushed hard enough and any edit is harmless if pushed gently\.β\\betais swept over four strengths per unit, scaled by each unit’s own activation range so that the promote signal at a typical firing position is comparable across architectures with different activation scales, and collateral is read at*matched efficacy*by interpolating each model’s curve\.

Table 30:Collateral damage at matched efficacy, median over 34–40 edited units per seed, target token held fixed\. Lower is better, and entries are per\-seed values where there are two and ranges where there are three\. The last column is the range of efficacy each model’s units reach across the four strengths tested, which is where the families separate without overlap\. On a set\-operator unit the edit adds to one of the two write columns, so it lands on one branch of the unit rather than on the unit as a whole\.The cleanest difference is reach rather than collateral \(Table[30](https://arxiv.org/html/2609.10210#S5.T30)\)\. Across the same four strengths, baseline units top out at an efficacy of3\.33\.3and3\.93\.9, while the monotone units reach6\.36\.3to9\.39\.3and the set\-operator units10\.110\.1to14\.314\.3\. The ranges do not overlap\. A baseline unit resists being told what to do past a certain strength\. Pushing harder stops raising the target logit and only adds damage — visible in the per\-unit curves, where efficacy peaks at the second of four strengths and then*falls*while collateral keeps climbing\. The monotone units do not have that turning point in the range tested\.

Collateral points the same way\. Four of the five monotone runs sit below both baseline runs at every matched efficacy, by factors of four to fourteen\. But the ranges overlap: one sigmoid seed \(0\.0410\.041at efficacy3\.03\.0\) is worse than one baseline seed \(0\.0230\.023\), so the statement is that the monotone models are typically more targeted rather than uniformly so\. The set\-operator seeds are the one place the ranges do separate:0\.0030\.003to0\.0050\.005at efficacy3\.03\.0against the baseline’s0\.0230\.023and0\.0850\.085, five to twenty times less disturbance for the same effect, and below sigmoid’s0\.0060\.006to0\.0410\.041as well\. An edit to one of its write columns acts only whereAAopens*and*BBselects that branch, which is the narrowest circumstance any construction here puts a write in\.

The two results are the same fact seen twice\. An edit tocuc\_\{u\}acts whereveruufires, so its precision is bounded by how well\-defined “whereveruufires” is\. If a unit’s set is a grab\-bag, an edit aimed at one part of it lands on all of it, and the model resists the edit exactly to the extent that the unit was doing several unrelated jobs\. That is why the baseline’s efficacy saturates and the monotone models’ does not: the edit is competing with the unit’s other responsibilities\. Legibility is what makes editing possible, which is why the edit checks the reading\.

##### Changing facts\.

Equation[33](https://arxiv.org/html/2609.10210#S5.E33)adds to a write column a component already has\. Two further operations ask more of it\.*Installing*an association puts a key for one token and a value for another into a component the model was not using, as Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)does on the baseline\.*Rewiring*keeps the write column exactly as trained and replaces the read row, so the component keys on a different token and delivers what it always delivered\. Measured across the arms, the three order the models differently \(Table[31](https://arxiv.org/html/2609.10210#S5.T31)\)\.

Table 31:Three edit operations across the arms\.*Gain*is the share of edits moving a component’s own named scope more than another component’s, at matched strength\.*Install*is the share placing a new token at the top of the distribution where its key occurs*and*not elsewhere, at the strongest setting whose held\-out loss stays within ten percent of the unedited model\.*Rewire*is how far the component’s own write moves toward the token its read row was changed to\. The operations are measured on one protocol each, across both kinds of component\.Table 32:The scope test of Section[4](https://arxiv.org/html/2609.10210#S4)on the sigmoid model: amplify a component and ask where the logits move — on the tokens its own write column names, on another component’s, or on random tokens — at matched strength\. “Respects the name” is the share of edits moving the component’s own scope more than another’s\. Same protocol as Table[20](https://arxiv.org/html/2609.10210#S4.T20), so the rows are comparable across the two models\.Amplification is indifferent to the activation\. Every arm takes it, on both kinds of component, on6868to9999percent of trials — scaling a column commutes with whatever the activation does, so it asks nothing of the function\. The scope test of Section[4](https://arxiv.org/html/2609.10210#S4), however, asks whether the movement lands on the tokens the component’s own column names rather than on another component’s\. The sigmoid model respects the name on every edit tried against9797percent for the baseline, and moves its own scope by1\.621\.62against the baseline’s0\.890\.89\(Table[32](https://arxiv.org/html/2609.10210#S5.T32)\)\. The other two operations are not indifferent at all\. No monotone arm takes an installed association at any strength of either knob*when the edited unit is the only thing changed*, where the baseline manages3232percent; and rewiring, which the baseline’s units accept at\+2\.94\+2\.94, moves a sigmoid unit by\+0\.002\+0\.002\.

Table 33:Installing an association across the four activations, at the strongest setting whose held\-out loss stays within five percent of the unedited model, so the sites are compared at matched damage rather than at matched strength\. The last column is the same models’ read side from Table[15](https://arxiv.org/html/2609.10210#S3.T15)\. The unit beats the channel on the baseline; on the other three neither site takes the edit at any strength, so the two are tied at zero\. The ordering of the two columns is close to opposite\.The failure has one shape \(Table[33](https://arxiv.org/html/2609.10210#S5.T33)\): raise the write gain untilBBarrives whereAAis, andBBhas arrived everywhere else too, its rank away fromAAfalling from the high hundreds to11with held\-out loss following\.

Neither knob explains that away\. The write gain sets how much ofBBarrives and the read gain scales the installed key; both were swept, over four orders of magnitude in the second\. ReLU takes no specific edit at any setting of either\. Sigmoid and softplus reach a third of trials only at a read gain of256256and a held\-out loss of1111to2323against baselines near3\.23\.2— the association can be forced in by saturating the unit, and what remains is not a model\. The baseline needs neither knob raised: at a read gain of11it installs on4848percent of trials for five percent of held\-out loss, and on2424percent for one percent of it\.

At least part of the reason is that an installed association stays local only where the component can be silent on the tokens it was not installed for, andσ⁡\(0\)=12\\sigma\(0\)=\\tfrac\{1\}\{2\}leaves a sigmoid unit no such state\. Measured on the candidate vocabulary, an installed key gives the baseline a sixty\-to\-one contrast between the token it was installed for and the rest, and sigmoid two\-to\-one, with no token silent at all\. ReLU fails for a different reason: its contrast is fine, at forty to one, but the unit emits about a ninth of what the baseline’s does, so it must be driven nine times harder to be heard — into a residual that is smaller in the same proportion, which is why the same strength costs it four times the loss\. The baseline’s advantage is sharper still: off its installed token it emits a slightly*negative*amount, so the edit pushesBBdown everywhere it was not meant to act, which is where the improving collateral in Table[21](https://arxiv.org/html/2609.10210#S4.T21)comes from\.

The same ceiling caps rewiring, since a unit whose output lies in\[0,1\]\[0,1\]can never deliver more than one column’s worth however its read row is changed\. Softplus, unbounded above, sits between the two as that account predicts, at\+0\.36\+0\.36\. The set\-operator arm behaves like the bounded arms on all three, which is what its form implies: its outputs are products of sigmoids and so lie in\[0,1\]\[0,1\]too\. Section[5\.12](https://arxiv.org/html/2609.10210#S5.SS12)works through the second write column it has and they do not\.

### 5\.10The edit these models will take

The previous section reads like all is lost for editing these models, but it also points to the repair\. What an installed unit emits away from its key is a*constant*: the resting value of the activation times the value column, at every position alike\. A constant added to the residual can be subtracted from it, and the subtraction need not come from the edited unit\. It has to come from somewhere, though, and that is where the architecture binds\. These models carry no bias — neither the read projection nor the write projection has one — so there is no term in the layer whose whole job is to hold a constant\. The only place the correction can go is the write columns of the other units\. But it can be put there\.

Install the key and the value as before, run the model on held\-out text, and take the positions where the key is*absent*\. Over those positions, ask the layer to reproduce the constant the installed unit emits: solve for coefficientsθ\\thetaover the units whose activation is most nearly constant off\-key, and subtractθv​β​v\\theta\_\{v\}\\beta vfrom each of their write columns\. The fit is a ridge solve against activations the model supplies, with nothing trained and no gradient taken\. The offset is removed and the excursion where the key occurs survives, because those positions were held out of the fit\.

Table 34:Installing an association, over2323token pairs, five depths and five strengths per model\. The compensating fit and the threshold’s quantile are both taken on a block of contexts disjoint from the one the edit is scored on\. An edit counts only if the target reaches the top ten where the key occurs, is*not*promoted anywhere else, and itself costs under ten percent of held\-out loss — the criterion of Table[31](https://arxiv.org/html/2609.10210#S5.T31)\.*Corrected*subtracts the leak from other units’ write columns;*thresholded*instead gives the unit an off state by subtracting a threshold from its read row, taken as a quantile of the key’s own projection over positions where the key is absent\. The two repairs are independent and the table crosses them\. “Target elsewhere” is the median ratio of the target’s rank away from the key after the edit to before it, where11is untouched\.The leak is exactly what the diagnosis said it was, and it is confined to the constructions the diagnosis is about \(Table[34](https://arxiv.org/html/2609.10210#S5.T34)\)\. Uncompensated, a sigmoid unit promotes the target twelve times over the rest of the corpus and softplus eight times, where the baseline moves it by three percent:GELU⁡\(0\)=0\\mathrm\{GELU\}\(0\)=0, so a conventional unit is already silent off its key and has nothing to subtract\. Compensated, every arm’s leak is gone — the target’s rank away from the key returns to where it started or slightly worse\.

On the baseline the correction is not the best repair\. What the unit emits away from its key is a constant*because its read row has no off point*: the row fires partway wherever the key is even weakly present, and the activation turns that into a floor\. A threshold in the read row removes the cause rather than the symptom\. These models have no bias, so it is built the same way the correction is — from the near\-constant direction of the layer’s input, with the offset taken as a quantile of the key’s own projection over positions where the key is*absent*, so nothing is fitted on the positions the edit is aimed at\.

The threshold works on every model: it takes the unit’s mean activation off its key from0\.120\.12to−0\.031\-0\.031on the baseline and from0\.420\.42to0\.0040\.004on the sigmoid arm, a hundredfold\. But on the shaped arms the install still fails, on9292to9999percent of attempts, and it fails on one clause — the target is promoted*elsewhere*\. What is left below the threshold is small on every model and its sign differs:GELU\\mathrm\{GELU\}is negative just below zero, so the residue demotes the target away from the key, which the criterion permits; a bounded or rectified unit cannot go below zero, so however small its residue it promotes the target everywhere, which the criterion forbids\. That is what the correction removes, and the numbers say so directly — with the threshold alone the share of edits that stay local is11to88percent on the shaped arms and7575on the baseline; adding the correction takes the shaped arms to2727to3636and the baseline*down*to6767, because there it subtracts something that was helping\. So the baseline needs the threshold and not the correction, reaching7575percent; the shaped arms need both, reaching2727to3636where the correction alone reaches55to1616\.

With the leak removed the association installs\. Every order\-preserving arm goes from taking no edit at all to taking one, softplus on99percent of attempts and ReLU on1616, and the edits are cheap: the least expensive passing edit costs a thousandth of a nat or less, which is under a twentieth of a percent of held\-out loss on any of them\. The same correction helps the baseline, from2525percent to3232: a GELU unit is nearly silent off its key rather than exactly silent, and the residue is worth removing\.

One control decides how the correction has to be built\. Given every unit in the layer, the fit reconstructs the installed unit’s activation atR2R^\{2\}between0\.850\.85and1\.001\.00— including at the key positions it never saw, which are not linearly independent of the rest — so what it subtracts is the edit rather than the leak\. On the sigmoid and set\-operator models the target then falls past rank eight thousand and rank thirty\-eight thousand, where the capacity\-limited fit had brought it inside the top ten; on the others it is pushed back by an order of magnitude\. That arm is worse than the capacity\-limited fit on every model\. The correction has to be limited to units that cannot track the input: a unit with a large mean and little variance can supply a bias and nothing else, which is precisely the term the architecture is missing\.

So the construction changes what an edit has to consist of\. A conventional unit can be told something new on its own: a threshold in its own read row is enough, and it is the best repair there is for that model\. An order\-preserving unit cannot, and the threshold is what shows it — the same repair, confined to the unit itself, leaves the shaped arms at00to99percent, not because the unit goes on firing but because what it emits when it is off has the wrong sign\. The residue has to be taken back somewhere else in the same layer, which the layer can express, at a cost too small to matter\. These arms buy precision on the operations they support at the cost of installing a fact as a two\-part edit rather than a one\-part one\. Section[5\.12](https://arxiv.org/html/2609.10210#S5.SS12)shows the set\-operator arm taking the correction out of its own second write column instead of out of the layer\.

##### Their attention fails the other way\.

The same question asked of these models’*attention*comes out differently, and it is a cost of the second departure of Section[5\.3](https://arxiv.org/html/2609.10210#S5.SS3)rather than of the activation function\. Their heads bound the*value*rather than the weighting over positions, which is the ordinary softmax\. What a head gathers is therefore an average of bounded quantities, and a sigmoid rests atσ⁡\(0\)=12\\sigma\(0\)=\\tfrac\{1\}\{2\}, so every position contributes that resting level whatever the head attends to\. Install a key into a channel and measure what it gathers where the key is present and where it is absent\. What an edit can use is the*difference*between the two — what arrives at the key over and above what arrives everywhere\. On the baseline the channel collects−0\.06\-0\.06where the key is absent and\+0\.38\+0\.38where it is present, a usable difference of0\.440\.44\. On the sigmoid model the same measurement returns0\.4850\.485against0\.5240\.524: a difference of0\.040\.04, and on softplus0\.070\.07and on ReLU0\.020\.02\. Both of those numbers sit on either side of12\\tfrac\{1\}\{2\}, which is where a sigmoid with nothing driving it rests\. So the usable signal is a sixth to a twentieth of the baseline’s, and it arrives on top of a resting level seven to twenty\-six times its own size\. That level is the same at every position, so an installed association is delivered everywhere the head looks rather than only where the key is: the target’s rank away from the key improves fiftyfold on the sigmoid model and a hundredfold on the other two, where on the baseline it does not move at all\.

Subtracting that level removes the delivery everywhere and leaves too little behind\. Compensated, the target stops arriving away from the key on all three, and no edit of this form passes at any strength\. Nor is the trouble that these models lack local heads: measured as Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)measures it, their most local heads reach0\.870\.87and0\.890\.89against the baseline’s0\.750\.75, and installing in them sharpens what arrives — the target reaches rank22to55at the key — while the share of edits that install*and*stay local remains zero\.

A head that puts the target at rank22at the key is a head that found the key — what fails is not the choice of where to look\. What is flattened is what the head carries there\. A value squashed into\[0,1\]\[0,1\]and read by a key row that was not scaled for it sits in the sigmoid’s near\-linear middle, so what arrives departs from the resting level by very little however sharply the head attends\. The two sublayers therefore fail for opposite reasons in these models\. The feed\-forward unit discriminates and leaks, which a subtraction repairs\. The attention channel discriminates too, and delivers a value flattened toward its resting level; the subtraction removes that level and leaves too little to work with\. Section[5\.12](https://arxiv.org/html/2609.10210#S5.SS12)scales a key row to the value’s range and recovers a usable difference for a marker read by a feed\-forward operand, though not for an association read at the output\.

### 5\.11Placement, circuits and the rank\-one editor across the arms

With the repair in hand, the rest of Section[4](https://arxiv.org/html/2609.10210#S4)runs across the arms\. Reporting an install at one operating point hides the shape of what it buys, because there are two independent prices rather than one\. Figure[26](https://arxiv.org/html/2609.10210#S5.F26)reports the frontier: for each budget of held\-out loss, the best rank the target reaches over the compensation arm, the write strength and the threshold quantile\. Giving the read row a threshold moves every curve to the left, by amounts that separate the activations\. The baseline reaches the top ten for0\.00320\.0032nats without one and0\.00160\.0016with one, a factor of two\. Sigmoid needs0\.350\.35without one and0\.00200\.0020with one, a factor of a hundred and seventy, and its cheapest route to rank one falls from1\.311\.31nats to0\.00920\.0092\.

Figure 26:The best rank the target reaches for a given held\-out loss budget, at layer ten\. Each frontier is taken over four compensation arms and five write strengths\. Left: the read row as installed\. Right: the same with a threshold subtracted from the read row, swept over three quantiles\.The placement results of Section[4\.4](https://arxiv.org/html/2609.10210#S4.SS4)carry over to the arms without qualification\. On the sigmoid arm a write keeps0\.390\.39of its direction at the readout when made before the last layer against0\.100\.10before layer two, spill falls from0\.400\.40to0\.040\.04over the same range, and the readout frame is again the only family whose response is readout\-visible above chance, rising from0\.290\.29to0\.520\.52\. Injected content is found in the coordinates it was written in there too, at0\.980\.98with no gap\. A unit the sigmoid model trained for itself is driven from two layers upstream with0\.880\.88of the effect passing through it, against0\.860\.86on the baseline, and the read row’s estimate governs the result the same way, separation rising from1\.91\.9at two positions to24\.824\.8at the full sample\.

The same frontier drawn for the circuit of Section[4\.6](https://arxiv.org/html/2609.10210#S4.SS6)separates the arms less than the install does\. The baseline reaches the top ten for0\.1210\.121nats and the set\-operator arm for0\.2400\.240, and imposing the specificity criterion takes those to0\.1790\.179and0\.5510\.551, where ReLU pays nothing for it at0\.1390\.139either way\. The comparison of Section[4\.9](https://arxiv.org/html/2609.10210#S4.SS9)also holds across the arms: the single\-unit install reaches the top ten for less held\-out loss than the closed\-form rank\-one update on every one of the five models, by factors running from nine on sigmoid to seventy\-two on the set\-operator arm\.

Figure 27:The best rank the target reaches for a given held\-out loss budget, over the full cross of write strength, read gain, saturation, read\-row calibration and marker tilt, with heads at layers four, six, eight, nine and ten reading at layer ten\. Left: any setting\. Right: the same with the specificity criterion imposed, so the target may not be promoted where it was not asked for\. The gap between the panels is the price of specificity\.
### 5\.12Editing through the second operand

Everything above edits what a unit writes\. The set\-operator unit has a second place to act, on the input side, and three edits use it\. What they establish is what this construction expresses directly; a conventional model can be made to express the same things, and the end of the section says how\. LeavingWaW\_\{a\}untouched, so the unit keeps whatever it was already triggered by, and keyingWbW\_\{b\}on a tokenCCmakes the intersection branch fire on*its usual trigger andCC*and the difference branch fire on*its usual trigger and notCC*\. One write installs a conjunction and a suppressor at the same time\.

It works at no cost in held\-out loss\. Over twelve units per layer at every depth, the intersection branch fires five times more often whereCCis present than where it is not, and sharpens as the gate is driven harder; the difference branch fires*exactly*zero whereCCis present, at every strength tried; and held\-out loss moves from3\.1563\.156to3\.1573\.157\. The exactness is the same property that costs this family everywhere else\. Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)found that a bounded activation will not, on its own, be told a new association, becauseσ\\sigmahas no silent state\. Here that is what makes the gate clean:BBsaturating at one makesA∖B=A⁡\(1−B\)A\\setminus B=A\(1\-B\)exactly zero rather than approximately zero, so the suppression is total\. Boundedness is the mechanism the construction uses\.

##### A unit that cancels its own leak\.

Section[5\.10](https://arxiv.org/html/2609.10210#S5.SS10)installs an association into an order\-preserving unit by taking the constant it leaks back out of the other units in the layer\. This construction can do the same without leaving the unit\. Set the intersection column toβ​v\\beta vand the difference column to−α​β​v\-\\alpha\\beta v, and what the unit contributes is

\[A​B−α​A​\(1−B\)\]​β​v=A⁡\[\(1\+α\)​B−α\]​β​v,\\big\[AB\-\\alpha A\(1\-B\)\\big\]\\,\\beta v\\;=\\;A\\big\[\(1\+\\alpha\)B\-\\alpha\\big\]\\,\\beta v,\(36\)which is identically zero wherever the bracket vanishes, whateverAAis doing\. Choosingα=B¯/\(1−B¯\)\\alpha=\\bar\{B\}/\(1\-\\bar\{B\}\)for the resting valueB¯\\bar\{B\}of the second operand puts the zero where the unit is not wanted, and the only quantity measured isB¯\\bar\{B\}\.α\\alpharuns from0\.560\.56to1\.621\.62with a median of0\.930\.93, which is what both operands resting nearσ⁡\(0\)=12\\sigma\(0\)=\\tfrac\{1\}\{2\}gives\. Uncompensated, an installed unit here promotes the target six times over the rest of the corpus; with the second column set this way it promotes it0\.820\.82times\. Takingα\\alphafrom a high quantile ofBBinstead over\-cancels — the target is pushed back twelvefold away from the key — and passes: atβ=12\\beta=\\tfrac\{1\}\{2\}the target moves from rank4646to rank66where the key occurs and from346346to611611elsewhere, for0\.0180\.018nats\. It does not beat the fitted correction, which passes on3\.73\.7percent of attempts against this form’s2\.32\.3, within each other’s noise at this sample\. What it establishes is what the structure makes expressible: a single unit can be given a fact and silenced off it using only its own parameters, where the other arms spend part of the layer on the correction\.

##### Installing a fact under a condition\.

The third edit installs a fact*under a condition*: fire where tokenAAis current and tokenCCoccurred in the preceding forty\-eight tokens\. The feed\-forward block cannot read that condition off its own position\. A rowE~​\[l,C\]\\tilde\{E\}\[l,C\]of the layer\-native table is the state the layer sees whenCC*is the current token*, so keying an operand on it asks whetherCCis here, not whetherCCwas there; at anAA\-positionCCis not here and the gate never opens\. CarryingCCforward is attention’s office, so the condition is delivered by an attention channel installed in the same edit: the channel readsCCinto its value row and writes what it gathered along a private direction, and the unit’s second operand reads that direction\. Five weight writes, no gradient, the two sublayers joined by hand\.

Two settings are needed, and both follow from the construction rather than from tuning\. The channel’s value is bounded in\[0,1\]\[0,1\], so a key row not scaled for it sits in the sigmoid’s near\-linear middle and what the channel gathers barely leaves its resting level: the contrast between gathering atCCand away from it is1\.021\.02to1\.451\.45times\. Choosing a threshold and gain from the measured distribution ofkey⋅z\\mathrm\{key\}\\cdot zinstead — one quantile and one scale, nothing fitted against the target — takes that contrast to901901\. And the operand that reads the marker has no off state, becauseσ⁡\(0\)=12\\sigma\(0\)=\\tfrac\{1\}\{2\}and these models carry no bias parameter; supplying one from the layer’s near\-constant direction takes the operand’s firing afterCCagainst elsewhere from1\.21\.2times to4343\. The first of the two is the question Section[5\.10](https://arxiv.org/html/2609.10210#S5.SS10)leaves open, where an association installed into these models’ attention channels arrived everywhere at the resting level and would not separate: scaled here, a channel does deliver a usable difference — for a marker read by a feed\-forward operand, which is a weaker thing to ask of a channel than an association read at the output\.

With both set, the edit lands\. Over six triples, three depths and five strengths, the unit reaches the top ten whereAAis current andCCfell in the preceding forty\-eight tokens, stays out of it in both other groups, and holds held\-out loss inside ten percent on3131of9090attempts\. Its operands readA=0\.924A=0\.924at those positions against0\.0200\.020and0\.0230\.023at the other two, andB=0\.997B=0\.997against0\.0020\.002, so the intersection separates by about130130and90,00090\{,\}000times\. The sharpest circuit puts the target at rank11whereAAfollowsCC, against10,17010\{,\}170whereAAoccurs without it and14,21514\{,\}215elsewhere, for1\.61\.6percent of loss; the cheapest moves it from rank39023902to99where the condition holds, leaves it at48694869whereAAoccurs withoutCC, and costs0\.20\.2percent\. An independent replication over a different range of strengths passes1717of100100\.

The head is what carries the condition\. Repeat every weight write and leave the attention head alone, so the operand reads a direction nothing writes, and the conditional ratio falls from17\.6517\.65times to1\.241\.24, with one attempt of ninety passing rather than thirty\-one\.

##### Two can play at this game\.

Two operands are worth less within this model than the construction suggests\. Pin the second operand open, which is the same circuit with one operand and no threshold, and88of9090pass — at the*same*conditional ratio,18\.4918\.49against17\.6517\.65\. The second operand does not sharpen the condition at all\. What it does is narrow where the write happens: with one operand the target is promoted whereverCCoccurred regardless ofAA, and its rank away from both groups sits at764764against16161616with both operands\. Put both detectors into a single read row of the same model and raise the threshold, so that a thresholded sum performs the conjunction, and1818of9090pass at a conditional ratio of9\.749\.74, against the two\-operand form’s3131of9090at17\.6517\.65\(p=0\.044p=0\.044on the counts\)\. That margin is understated rather than inflated, because the thresholded sum was given the best of five thresholds and the two\-operand form was not tuned that way\.

None of that makes the edit a capability of this construction\. Run the same circuit on the conventional baseline — one read row carrying both detectors over a raised threshold, the head installed the same way — and2828of9090attempts pass, against the set\-operator arm’s3131, which is not a difference at this sample; its own no\-head control passes none, at a ratio of1\.341\.34\. On the sharpness of the condition the conventional unit is an order of magnitude ahead: a conditional ratio of194\.5194\.5against17\.6517\.65, and a median rank of11whereAAfollowsCCagainst110110\. What the second operand buys, on the evidence here, is locality rather than the condition itself\.

One part of the construction does*not*earn its place here\. The self\-cancellation of Equation[36](https://arxiv.org/html/2609.10210#S5.E36), which suggested this edit, contributes nothing to it:3131attempts pass with the difference column set and3131without\. The algebra says why it cannot\.A∖B=A⁡\(1−B\)A\\setminus B=A\(1\-B\)is nonzero only whereBBis off, which is at*non*\-AApositions, while the leak that breaks the conjunction is atAA\-positions, where the difference column is already zero\. The cancellation works for the case Equation[36](https://arxiv.org/html/2609.10210#S5.E36)was derived for, both operands keyed on one token; it does not extend to a condition carried in from somewhere else\.

So what the second operand is worth is narrow\. It buys no better reading — theKKsweep of Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)put every order\-preserving arm on one curve — and on the write side it buys locality rather than capability: the second operand narrows where a write lands without changing how sharply the condition binds\. Two smaller things come with it, a suppressor that is exactly off rather than approximately off and a leak the unit takes back out of its own parameters instead of out of the layer\. The cleverness is not in the set\-operator model so much as in the fact that a conventional model can be compelled to perform the same tricks\.

What the activation function changes, then, is this\. An order\-preserving unit reads at the instrument’s ceiling, flat inKK, at a quality inside the baseline’s converged range; a conventional unit reads its strongest few inputs and then reverses, because nearly two thirds of its pre\-activations sit where GELU runs backward\. The order\-preserving arms’ circuits are half to two thirds the size at one hop and four to seven times smaller at closure, and the handful a prediction cannot lose shrinks from thirteen to a few\. The promote edit reaches twice as far before it saturates\. What the change costs is the install: a bounded unit has no silent state, so a new fact leaks until the layer takes the constant back, and installing becomes a two\-part edit\. Three constructions with nothing in common but an order\-preserving activation land on one curve, and the one that discards the negative half\-line buys its extra legibility with quality — so the property doing the work is that the function does not destroy the ordering the weights supply, and nothing more elaborate is needed to get it\.

## 6Models trained by other people

Everything so far has been measured on models trained for this paper\. The instrument does not depend on how a model was built, so the more useful question is what it says about models built by other people for other reasons\. This section applies it unchanged to twelve of them: seven architecture families, three activation functions, eight organizations, and a range of124124million to77billion parameters\. None was trained, fine\-tuned or modified for this\. The procedure is the one in Section[2](https://arxiv.org/html/2609.10210#S2), at the same confidence floor, with the same matched random control\.555Two of the twelve, OLMo\-2 and Gemma\-3, normalize a sublayer’s output before adding it to the residual rather than normalizing its input; an RMS norm divides by a scalar, so with that scalar held at its measured value the per\-layer factor folds into the readout direction, which is the same approximation already made for the final norm\.The accounting, the removal curves, the closure, the required graph and the sufficiency search run on all twelve; the composition, edit, margin and horizon measurements were run on subsets, and each says which\.

Table 35:Twelve off\-the\-shelf models under the accounting of Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)\.n90n\_\{90\}is the number of components reaching ninety percent of the net; “absolute” is the same quantity under the absolute\-value convention; “inflation” is their ratio\. The last column is where the median prediction changes when the top\-ranked components are removed\. Components are feed\-forward units and attention \(head, channel\) pairs pooled together, so the totals range from46,08046\{,\}080for the smallest model to630,784630\{,\}784for the largest\.Figure 28:Circuit size against parameter count, one point per model\. Across a56×56\\timesspan in parametersn90n\_\{90\}spans10\.6×10\.6\\times, and the spread within a size band is wide: the three11B models range from7676to208208\. Circuit size grows far more slowly than the model, so the*fraction*of a model carrying one prediction falls with scale\. The removal curves these points summarize are Figure[29](https://arxiv.org/html/2609.10210#S6.F29)\.The accounting makes small counts visible everywhere\. Every model reaches ninety percent of its prediction on a few dozen to a few hundred components, and every model reports two to three orders of magnitude more under the convention in common use \(Figure[30](https://arxiv.org/html/2609.10210#S6.F30)\)\. That the same gap appears in a124124million parameter model from 2019 and a77billion parameter model from 2024, across seven independent implementations of the transformer, is the strongest evidence available here that it is a property of transformers more generally\.

Llama\-3\.2 3B reaches ninety percent on fewer components than Gemma\-3 1B, a model three times smaller \(Figure[28](https://arxiv.org/html/2609.10210#S6.F28)\)\. Across the set the share of a model carrying a single prediction falls with scale, from about two tenths of a percent in GPT\-2 to a few hundredths in the largest models here\. On this evidence larger models are more concentrated, though the sample does not support a curve\.

The activation function shows up here too, on models nobody chose for the purpose\. OPT is the only stock model in the set built on ReLU, and it has by far the least cancellation of the twelve — close to the figure measured on the ReLU arm trained for Section[5](https://arxiv.org/html/2609.10210#S5), which was built and trained independently\. Whatever the bounded and half\-wave activations do to how much a model’s contributions cancel, they do it in models trained by other people as readily as in ours\.

Figure 29:Removing the top\-ranked components changes the prediction in all twelve models; removing the same number at random does not\. The random control is pooled across the set and stays below five percent untilkkexceeds a hundred\.Figure 30:Left: the three denominators across the twelve models, on a log axis\. The gap between the signed convention and the others is two to three orders of magnitude in every one\. Right: circuit size against the confidence of the prediction, which is why all figures here are read at one floor\.### 6\.1Composition

Table 36:What the*one\-hop*sets are made of, for the models where that measurement was run\. “Last quarter” is the share of members in the final quarter of layers, against2525percent if membership were spread evenly; these are shares of direct contribution, which is where a prediction is credited rather than necessarily where it is decided\. “Seen once” is the share of distinct components appearing in exactly one prediction\. Figure[31](https://arxiv.org/html/2609.10210#S6.F31)asks the same two questions of the sufficient sets, which are the smaller object and cover fewer models\.Figure 31:Where the*sufficient*circuits sit, across every model the search was run on\. Left: share of the set’s members by relative depth, one line per model; the dotted line is what an even spread would give\. Right: the share falling in the last quarter of the stack, against the2525percent an even spread would put there\. Members are those of the pruned sufficient sets of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5), not of the one\-hop accounting\.The circuits sit late, in every model\. The last quarter of the stack holds2828to7676percent of a sufficient set where an even spread would put2525, and no model here falls below that line \(Figure[31](https://arxiv.org/html/2609.10210#S6.F31)\)\. They are also mostly specific to the prediction they were found for: between7979and100100percent of the distinct components in a model’s sets appear in exactly one of them, so a small shared remainder does the recurring work and the rest does not recur at all\.

Attention supplies between2222and4040percent of the components in the one\-hop sets \(Table[36](https://arxiv.org/html/2609.10210#S6.T36)\), so a trace restricted to the feed\-forward stack would be missing between a fifth and two fifths of the machinery\. Neither this share nor the two above sort by architecture family, by size, or by activation: the two Llama\-3\.2 models sit at3434and3737percent and the two Qwen models at2424and3434, ranges that overlap\. Whatever sets any of them is not visible at this sample size\.

### 6\.2The whole dependency graph

The recursion of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)extends to these models unchanged\. Every one of them closes, at seven to ten levels, on fifty to a hundred predictions each\.

Table 37:The backward dependency graph, closed, for the twelve models trained by other people\. Columns as in Table[27](https://arxiv.org/html/2609.10210#S5.T27)\.Table 38:The required graph for the twelve models trained by other people, defined as in Table[28](https://arxiv.org/html/2609.10210#S5.T28)\.Tables[37](https://arxiv.org/html/2609.10210#S6.T37)and[38](https://arxiv.org/html/2609.10210#S6.T38)say three things\. The first is that the graph closes at all, in every model\. Following “what does this need” from a prediction all the way back terminates on a few percent of the model’s components\. The share does not grow with size: Mistral\-7B, sixty times the size of GPT\-2, closes on a slightly smaller fraction of itself than GPT\-2 does\.

The second is that the required graph is two orders of magnitude smaller than the closure, and does not grow with the model either\. A prediction from a seven\-billion\-parameter model can be changed by removing thirty\-four components\. Removing the closure changes the prediction almost always, where removing a size\-matched random set almost never does\.

The third is that the branching factor in the wild does not sort by activation function\. It runs from1616to119119, almost an order of magnitude among conventional activations alone, and the intervals interleave \(Figure[32](https://arxiv.org/html/2609.10210#S6.F32)\): OPT’s ReLU at1616\[14,18\]\[14,18\]overlaps GPT\-2’s GELU at2020\[16,25\]\[16,25\], and Llama\-3\.2 1B’s SiLU at2424\[22,30\]\[22,30\]overlaps Gemma\-3’s GELU at2222\[18,30\]\[18,30\]\. The controlled comparison of Section[5\.6](https://arxiv.org/html/2609.10210#S5.SS6)holds everything but the feed\-forward unit fixed and finds a clean separation; these models hold nothing fixed and show none\. Of the two statements about the same quantity, only the first isolates the activation function\. Nothing here supports reading a branching factor off a model’s activation function without the rest of the training held fixed\.

Figure 32:Branching factor with bootstrap intervals, every model in this paper\. Outlined bars are the models trained here, which differ from one another only in the feed\-forward unit; the rest hold nothing fixed\.
### 6\.3The smallest sufficient set across models

Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)measures, on the baseline, the smallest set of components that reproduces a prediction when everything else at that position is zeroed\. The measurement depends on neither the construction nor who trained the model\.

Table 39:Sufficiency on GPT\-2, one of the twelve\. “Keep\-more” is the rise in preservation from keeping nothing to keeping2,0482\{,\}048components, the check that keeping more helps\.Sufficiency reads the same way here as on the models trained for this paper: the closure does not reproduce a prediction reliably and a same\-sized set taken further down the one\-hop ranking does better \(Table[39](https://arxiv.org/html/2609.10210#S6.T39), in full in Appendix[D](https://arxiv.org/html/2609.10210#A4)\)\. What does reproduce a prediction is the small set of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5), which behaves on these models as it does on the ones trained here\.

Table 40:The smallest set that reproduces a prediction, across models\. Columns as in Table[7](https://arxiv.org/html/2609.10210#S2.T7)\. “Floor” is the share of predictions the token embedding alone already returns; those are excluded from the columns to its right, because no set kept on top of the floor can be credited with the answer\. “Converged” is how many of the sampled predictions Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3)verified, and the size and the controls are over those\.Every model the search was run on has such sets, of two to twenty\-one components out of tens of thousands \(Table[40](https://arxiv.org/html/2609.10210#S6.T40)\)\. The range holds from124124million parameters to seven billion, and the largest models sit at its small end: Mistral 7B and Qwen2\.5 7B rest on two components each, three parts in a million of the model\.

The search does not finish on every prediction\. The repair step is given a fixed number of rounds, and where it runs out the kept set never becomes sufficient and there is nothing to report\. Those predictions are counted in the “converged” column and excluded from everything to its right\. The measurement covers all six models trained here and all twelve taken off the shelf\. The controls behave the same way everywhere too\. A set of that size drawn at random reproduces the prediction on*no*prediction of any model — not one — so whatever these sets are doing is not a property of their size, and not an artifact of a test that deletes the competition along with everything else\. The plain contribution ranking at the same size manages00to2222percent, which is the closest any control comes\.

### 6\.4Naming a component and acting on the name

Saying what a component is*for*is a stronger claim than identifying which components carry a prediction, and the way to test it is to act on the name and see where the consequences land\. A component’s parameters name a scope with nothing fitted: the tokens its write column promotes, read through the unembedding\. Amplifying the component and asking which logits move then has a right answer fixed in advance\.

Dropping the sign that scope carries loses the result on the gated models\. A component’s effect on the logits isau​cua\_\{u\}c\_\{u\}rather thancuc\_\{u\}\. In a gated feed\-forward the write\-site value isϕ⁡\(Wg​x\)⊙\(Wv​x\)\\phi\(W\_\{g\}x\)\\odot\(W\_\{v\}x\)and the value branch is linear and unbounded, soaua\_\{u\}is freely signed: on Qwen2\.5\-1\.5B and Llama\-3\.2\-1B,4747and4848percent of the components carrying a prediction fire negative\. For those, reading the scope offcuc\_\{u\}alone inverts the targets, so amplifying the component moves the named tokens the wrong way\. Table[41](https://arxiv.org/html/2609.10210#S6.T41)shows the consequence, which is not subtle: read without the sign, edits on negative\-firing components land correctly44to1111percent of the time, and the models as a whole score barely above chance\. Ungated models are untouched, because almost none of their components fire negative — zero percent in GPT\-2\. Reading the scope with the sign the component actually has fixes it everywhere\.

Table 41:What the sign of the activation is worth\. “Fire negative” is the share of prediction\-carrying components whose write\-site value is negative\. The middle columns score edits against a scope read from the write column alone; the last reads it with the activation’s sign\. The failure is confined to gated feed\-forwards and to exactly the components that fire negative\.Table 42:An edit installed in one component, scored against the scope its own parameters name and against the scopes of the other components edited in the same predictions, at matched strength\. The last column is the share of edits moving their own scope more than another component’s, across all strengths and all edits\.With the scope read correctly, Table[42](https://arxiv.org/html/2609.10210#S6.T42)reports the test on the three models taken off the shelf it was run on\. In every one, an edit moves what the parameters said it would and leaves the other components’ targets alone, in9494to9797percent of cases —9797on the baseline \(Table[20](https://arxiv.org/html/2609.10210#S4.T20)\) and100100on the sigmoid model \(Table[32](https://arxiv.org/html/2609.10210#S5.T32)\)\. This asks the parameters to predict the consequence of an intervention rather than merely to rank components\.

### 6\.5Where the instruments are stressed

Two of the choices made in Section[2](https://arxiv.org/html/2609.10210#S2)looked like bookkeeping on the model they were demonstrated on\. Across twelve models they are not, and this is where that shows\.

Table 43:The raw logit drop against the margin, on models where they diverge\. Sign disagreement is how often the two disagree about the direction of an effect; non\-monotone steps count the times the median drop*falls*as more components are removed\. Target logits span two orders of magnitude across these models, so raw drops are not comparable between rows either\.The raw drop disagrees with the margin about the direction of the effect in up to forty\-five percent of cases, and in two models it is not monotone inkk, reporting the impossibility that removing more components did less damage\. Both failures are invisible on the baseline\. Numbers elsewhere in this paper would be impossible without the correction: measured in raw logits, the ratio of realized to predicted effect is negative for GPT\-2\.

### 6\.6What the constructions are worth

The twelve models above show the lens works on models other groups have trained; what the constructions of Section[5](https://arxiv.org/html/2609.10210#S5)add is a population that differs sharply by construction, in the direction this paper argues\. Figure[33](https://arxiv.org/html/2609.10210#S6.F33)and Table[44](https://arxiv.org/html/2609.10210#S6.T44)show it across every model in the paper\. The five constructions trained here for legibility all sit well above their own GELU baseline, which matches them in width, data and schedule, and the models trained by other people sit lower still — most of them below that baseline\. At the median, roughly two thirds of units are token\-shaped in the models built to be read, against about a third in the rest\.

Figure 33:Left: the share of feed\-forward units whose response to candidate tokens is more concentrated than9595percent of random directions in the same layer\. The dashed line is the55percent such a threshold admits by construction, so it is the level a model with no token\-selective units would show\. Right: the same quantity against relative depth\. The constructions trained here hold their level through the body of the network, and the conventional baseline declines with depth\.Table 44:Token selectivity by model\. “Top\-8 share” is the fraction of a unit’s positive response carried by its eight strongest tokens, and “random” is the same statistic for random directions in the same layer — the comparison that makes the first column mean anything\. “Rank recovery” is how well a unit’s top eight reproduce its full ranking over tokens, with the random value in parentheses\. Rows are medians over layers\.Table 45:Whether a circuit is worth more than an equally sized random set for the tokens that*follow*the prediction, at six horizons nested on one set of positions\. Columns as in Table[10](https://arxiv.org/html/2609.10210#S2.T10);1\.001\.00means it is worth no more\. Measured at each model’s training context, except GPT\-2 and OPT, which cap at10241024positions\.The reading of Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)does not depend on the construction either\. Across nine models the circuit is worth no more than an equally sized random set for the tokens that follow: the per\-prediction ratio stays between0\.730\.73and1\.071\.07at every horizon from sixteen to five hundred and twelve \(Table[45](https://arxiv.org/html/2609.10210#S6.T45)\)\.

The lens transfers unchanged\. The gap between the signed count and the conventions in common use is two to three orders of magnitude in every implementation from 2019 to 2024; the set that builds a prediction runs from two components to sixteen, with the largest models at the small end and the fraction of a model behind one prediction falling with scale; on gated feed\-forwards the sign of the activation has to travel with the name, and with it an edit lands where the parameters said it would\. The constructions’ extra legibility is real, and none of it is needed to read a stock model\. Conventional transformers can be read at the level of which components carry a prediction\. They are not opaque blobs of computation\. They are large collections of interwoven circuits whose predictions mostly cancel, carrying the architecture’s own management work alongside the few components that decide the token\.

## 7Related work

Every ingredient this paper uses has been built before\. What differs from prior work is one accounting choice and the object it is applied to\. This section says which parts are inherited and from where\.

##### Reading a component from its parameters\.

Projecting a feed\-forward component’s write into vocabulary space is established practice\.[Geva et al\. \[22\]](https://arxiv.org/html/2609.10210#bib.bib3)characterize feed\-forward layers as key–value memories, and[Geva et al\. \[21\]](https://arxiv.org/html/2609.10210#bib.bib4)read their value vectors as promoting concepts in the vocabulary;[Dar et al\. \[14\]](https://arxiv.org/html/2609.10210#bib.bib17)treat the parameters themselves as objects in embedding space\. The logit lens\[[38](https://arxiv.org/html/2609.10210#bib.bib1)\]and the tuned lens\[[6](https://arxiv.org/html/2609.10210#bib.bib2)\]apply the same projection to intermediate residual states, and[Bommasani et al\. \[8\]](https://arxiv.org/html/2609.10210#bib.bib18)distill static token representations by averaging contextual ones\. Of the layer\-native table of Section[2](https://arxiv.org/html/2609.10210#S2), a member of this family, only the substitution protocol is specific to this work\.

##### Decomposing a prediction\.

The residual stream framework of[Elhage et al\. \[18\]](https://arxiv.org/html/2609.10210#bib.bib7)makes a logit an exact sum of per\-component terms, which is what allows a prediction to be attributed at all\. Two lines qualify what that sum means\.[Janiak et al\. \[26\]](https://arxiv.org/html/2609.10210#bib.bib25)show that components write in opposing directions, measuring in a four\-layer trained transformer one head whose output is about ninety percent removed downstream and a correlation of−0\.70\-0\.70between writing and erasing components\. They propose no corrected attribution metric, recommending instead that direct attribution be paired with activation patching, which is the procedure this paper adopts\. Separately,[McGrath et al\. \[35\]](https://arxiv.org/html/2609.10210#bib.bib27)and[Rushing and Nanda \[46\]](https://arxiv.org/html/2609.10210#bib.bib28)show that ablating a component causes others to compensate\. Together these predict that a linear attribution will overstate what removing a component actually does, which Table[4](https://arxiv.org/html/2609.10210#S2.T4)measures directly\.[McDougall et al\. \[34\]](https://arxiv.org/html/2609.10210#bib.bib58)describe attention heads whose function is to suppress a token the model is about to copy\. Appendix[C\.8](https://arxiv.org/html/2609.10210#A3.SS8)asks whether suppression of that kind is visible above its floor in the models here, and finds the negative side of a token’s accounting indistinguishable from a control token’s\.

##### Prediction\-specific graphs built backward\.

The recursion of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)— start at the prediction, keep what feeds it above a threshold, repeat — is a shape that already exists\.[Ferrando and Voita \[19\]](https://arxiv.org/html/2609.10210#bib.bib39)build exactly such a graph top\-down for a single prediction, with nodes as token representations, edges as whole operations \(an attention head, a feed\-forward layer\), and importance given by an ALTI proximity score that is non\-negative by construction\.[Abnar and Zuidema \[1\]](https://arxiv.org/html/2609.10210#bib.bib38)aggregate attention across layers as a maximum flow, choosing that formulation because paths through a transformer share edges and summing path weights would double\-count\. Neither the backward construction nor the observation that paths overlap originates here\.

Three things differ in what this section measures rather than in what it constructs\. The unit is a single component — one feed\-forward unit, one attention channel — rather than an operation or a token representation, so an edge relates two components rather than two layers\. The edge weight keeps its sign, which is the distinction Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)is about; a score defined as non\-negative, or a capacity in a flow network, is the convention that section argues changes the answer by orders of magnitude\. And the object reported is the*size*of the graph and what survives removing it: that the recursion reaches a fixed point, on how many components, what fraction of the model that is, and which part of it a prediction cannot survive losing\.[Ferrando and Voita \[19\]](https://arxiv.org/html/2609.10210#bib.bib39)report routes and visualizations rather than sizes, and validate against previously published patched circuits rather than by ablating their own graphs\.

##### Circuit discovery\.

Automated methods locate the components responsible for a behavior: ACDC prunes a computational graph\[[11](https://arxiv.org/html/2609.10210#bib.bib32)\], edge attribution patching approximates that search with gradients\[[52](https://arxiv.org/html/2609.10210#bib.bib33)\], and AtP\* makes the localization efficient at scale\[[27](https://arxiv.org/html/2609.10210#bib.bib34)\]\. The reported sizes live in different bases and are not directly comparable to one another\.[Wang et al\. \[57\]](https://arxiv.org/html/2609.10210#bib.bib21)identify twenty\-six attention heads for indirect object identification;[Marks et al\. \[33\]](https://arxiv.org/html/2609.10210#bib.bib31)report on the order of a hundred sparse\-autoencoder features, and separately about fifteen hundred raw neurons, under dataset\-level faithfulness and completeness measured on a mean\-ablation baseline; the attribution graphs of[Ameisen et al\. \[2\]](https://arxiv.org/html/2609.10210#bib.bib30)operate on cross\-layer transcoder features and prune a representative graph from 236 nodes to 55\. Those graphs rank paths by the absolute value of edge weights, and the pipelines built on them accumulate absolute influence against its total\. This paper keeps the sign and divides by the net\. That normalization is not new:[Chen et al\. \[10\]](https://arxiv.org/html/2609.10210#bib.bib35)call the signed\-sum share the standard one, and[Salek and Enguehard \[47\]](https://arxiv.org/html/2609.10210#bib.bib36)take the net change in the output as the reference quantity an attribution should sum to, noting that when attributions cancel their absolute total exceeds it\. What Table[3](https://arxiv.org/html/2609.10210#S2.T3)adds is the consequence for circuit size, which does not appear to have been measured: on the same predictions and the same attribution, the conventions differ by two to three orders of magnitude in how many components reach ninety percent\.

[Chen et al\. \[10\]](https://arxiv.org/html/2609.10210#bib.bib35)also supply the sharpest caution\. Their credit is propagated recursively through a multi\-layer path expansion, where a small signed denominator compounds with depth, and they report that a pure signed denominator degrades their ranking to chance; their remedy floors its magnitude at a fraction of the absolute mass\. The quantity here is a single sum over components at one position, with no recursion\. A signed denominator is demonstrably unsafe once it is iterated\.

##### The neuron basis\.

Whether raw neurons are an adequate basis is unsettled\.[Ameisen et al\. \[2\]](https://arxiv.org/html/2609.10210#bib.bib30)report that thresholded neurons underperform their transcoder dictionaries on interpretability metrics, which is part of the motivation for working in a learned basis at all\.[Arora et al\. \[3\]](https://arxiv.org/html/2609.10210#bib.bib37)reach the opposite conclusion, reporting that approximately one hundred MLP neurons suffice to control behavior on a subject–verb agreement benchmark and that neuron\-basis circuits are as sparse as feature\-basis ones\. They also trace a neuron circuit for a single prompt on an off\-the\-shelf model, narrowing 257 attributed neurons to 23 by manual curation and steering, and identify one neuron that flips the top output\. That is the closest published work to Section[2](https://arxiv.org/html/2609.10210#S2)\. It differs in what is asked of the set: their nodes are selected by a per\-component attribution threshold and validated by steering, where the sets here are required to change the prediction when removed and are read against a control that removes the same number of components at random\. Their attribution also targets a sum over several candidate tokens rather than one logit, and does not place attention on the same footing as feed\-forward units\.

##### What ablation replaces\.

What a removed component is replaced with is a consequential choice\.[Li and Janson \[29\]](https://arxiv.org/html/2609.10210#bib.bib29)show that zero, mean, and resample baselines can differ by a factor of three on the same components, attribute the gap to the replacement value carrying information of its own, and propose an optimal constant instead\. Circuit evaluation toolkits accordingly report several baselines rather than committing to one\. This paper takes the same care and finds an asymmetry: the necessity measurement is nearly unchanged across the three counterfactuals while the sufficiency measurement moves severalfold \(Table[58](https://arxiv.org/html/2609.10210#A4.T58)\), which is why the claims here are necessity claims\. A related choice is what a component’s*constant*write counts as\.[Sun et al\. \[51\]](https://arxiv.org/html/2609.10210#bib.bib60)show that a few activations of very large and nearly fixed magnitude act as bias terms in the residual stream, and[Xiao et al\. \[59\]](https://arxiv.org/html/2609.10210#bib.bib59)that heads park their attention on early tokens whose values then add a constant at every position; either makes a component’s mean write large\. Appendix[C\.8](https://arxiv.org/html/2609.10210#A3.SS8)reruns its tests with that constant part removed, and has a seed where it dominates the signed sum\.

##### What the feed\-forward stack is for\.

A line of work asks whether the feed\-forward layers are load\-bearing at all\.[Elhage et al\. \[18\]](https://arxiv.org/html/2609.10210#bib.bib7)study attention\-only transformers as a tractable simplification;[Sukhbaatar et al\. \[49\]](https://arxiv.org/html/2609.10210#bib.bib41)fold the feed\-forward computation into attention as persistent memory and lose little; and against that,[Dong et al\. \[15\]](https://arxiv.org/html/2609.10210#bib.bib40)show that attention alone degenerates — without the skip connections and feed\-forward layers, a stack of pure attention loses rank doubly exponentially in depth\.[Geva et al\. \[22\]](https://arxiv.org/html/2609.10210#bib.bib3)give the feed\-forward layers a positive role as key–value memories\.

Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)speaks to this from an unusual direction\. The question here is how much of an existing stack one prediction needs\. The answer depends on depth\. The components that reproduce a prediction sit mostly in the last third of the stack,4343to8181percent of members, and the middle is the thinnest region on eight of the ten models measured, holding1212to5555percent\. Attention’s share of what is kept is nevertheless larger through the middle than at the top on most of them,2222to5454percent there against1919to3535percent in the last third\. For one token, most of the middle of the stack is not among the components that rebuild the answer\. That is consistent with the memories reading of[Geva et al\. \[22\]](https://arxiv.org/html/2609.10210#bib.bib3), since a memory is addressed on some inputs and not others, and it does not contradict[Dong et al\. \[15\]](https://arxiv.org/html/2609.10210#bib.bib40), whose result is about what a stack can represent rather than what one prediction consumes\.

##### Editing a component\.

Model editing installs an association a model does not hold, and its methods split the job the same way this paper does: choose a site, then compute what to write\.[Meng et al\. \[36\]](https://arxiv.org/html/2609.10210#bib.bib16)locate the site by causal mediation — corrupting the subject tokens and restoring hidden states one at a time — and obtain the update by optimizing a value vector against a target probability, normalized by an activation covariance estimated over a large text sample\. MEMIT and its successors inherit those traced layers rather than choosing their own, and the knowledge\-neuron line\[[13](https://arxiv.org/html/2609.10210#bib.bib44)\]scores neurons by integrated gradients\. The construction they share descends from[Bau et al\. \[5\]](https://arxiv.org/html/2609.10210#bib.bib56), who treat a layer as a linear associative memory and derive the covariance\-weighted rank\-one update that[Meng et al\. \[36\]](https://arxiv.org/html/2609.10210#bib.bib16)inherit\.[Huang et al\. \[25\]](https://arxiv.org/html/2609.10210#bib.bib55)take the closest structural approach to the one here, adding a single neuron per correction, with that neuron trained by gradient rather than written\. Every one of these reads the site off what the model*does*on some input\.

A separate line does read a component’s function from weights alone\.[Avrahamy et al\. \[4\]](https://arxiv.org/html/2609.10210#bib.bib19)rotate a unit’s weight matrices into vocabulary space with no forward passes, and[Dunefsky et al\. \[16\]](https://arxiv.org/html/2609.10210#bib.bib52)perform circuit analysis through feed\-forward sublayers with an explicitly input\-invariant component\. Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)contributes the conjunction of the two lines: a site read from the weights, an association installed at it, and the result scored on a criterion an edit can fail\.

The write half is where the overlap is closest\.[Dai et al\. \[13\]](https://arxiv.org/html/2609.10210#bib.bib44)already update a unit’s output column in closed form using token embeddings, and[Hakimi et al\. \[23\]](https://arxiv.org/html/2609.10210#bib.bib46)do so through unembedding columns, explicitly a per\-unit version of the rank\-one mechanism\. Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)takes that construction as given and contributes the other half: a site chosen from the read row of Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2), decoded in the layer’s own frame, with nothing traced and nothing fitted\.

Two results from the critique line bear on whether that is worth having\.[Hase et al\. \[24\]](https://arxiv.org/html/2609.10210#bib.bib45)find that where causal tracing localizes a fact predicts almost nothing about where editing it succeeds, which makes the expensive step of the standard pipeline hard to justify on its own terms\. And[Yang et al\. \[60\]](https://arxiv.org/html/2609.10210#bib.bib47)show that headline success rates fall sharply under evaluation conditions closer to deployment, which is a caution this paper’s own edit numbers inherit: they are measured on single edits, at one position, on a model of124124million parameters\. One more thing is worth noting about scope\. This literature edits feed\-forward layers predominantly, with[Liu et al\. \[32\]](https://arxiv.org/html/2609.10210#bib.bib57)an exception that writes to attention parameters as well, motivated by evidence that attention carries a substantial share of factual storage\[[58](https://arxiv.org/html/2609.10210#bib.bib51)\]\. Section[4\.2](https://arxiv.org/html/2609.10210#S4.SS2)finds a reason for that beyond convention: what attention writes at a position is a mixture over other positions, so an association installed in a channel cannot be made local\. The restriction does not extend to a gain, which attention takes about as reliably as a feed\-forward unit does, and Section[4\.6](https://arxiv.org/html/2609.10210#S4.SS6)shows that it does not extend to an installed head used as a marker for a unit downstream\.

Writing a transformer’s weights by hand rather than training them has its own line\.[Lindner et al\. \[30\]](https://arxiv.org/html/2609.10210#bib.bib53)compile programs into transformer weights, producing a model whose mechanism is known by construction, and[Vergara\-Browne and Soto \[56\]](https://arxiv.org/html/2609.10210#bib.bib54)place an algorithm inside a pretrained model’s residual stream by distillation\. Section[4\.6](https://arxiv.org/html/2609.10210#S4.SS6)writes two components into an otherwise untouched model trained by gradient descent\.

##### Legibility by construction\.

A separate line changes the model rather than the analysis\.[Gao et al\. \[20\]](https://arxiv.org/html/2609.10210#bib.bib26)train transformers with most weights constrained to zero and recover circuits small enough to reverse\-engineer, verified as both necessary and sufficient, at a cost in capability;[Tamkin et al\. \[53\]](https://arxiv.org/html/2609.10210#bib.bib11)replace activations with entries from a learned discrete codebook\. The constructions in Section[5](https://arxiv.org/html/2609.10210#S5)pursue the same goal along a different axis, changing what a feed\-forward unit does with its pre\-activation while leaving connectivity dense, and are evaluated here by the same instrument as the unmodified baseline\.

## 8Discussion

On every model measured here the same three questions have answers — which components carry a prediction, which it cannot survive losing, and which set builds it — and the numbers that come back are small\. What follows is what that makes possible and what it asks\.

##### Reading\.

The first consequence is that “what is this model doing here” becomes a measurement with an error bar\. A model that fails on one input can be audited for the components that produced the failure; a capability can be located before anyone argues about whether it exists; and the circuit for one token, traced through two architectures, can be compared\.

##### Writing\.

An edit installed on a named component moves what the name says it should and leaves other components’ targets alone, and that is a different operation from fine\-tuning\. Fine\-tuning changes a model by showing it data and accepting whatever internal change follows; an edit of this kind changes a stated thing and can be checked against the statement\.

Where an edit can be placed is a property of the architecture rather than of the component chosen — a write is lost to depth, not to distortion — and it leaves an editor a choice it has not usually been offered: write in the model’s own frame so that components already present respond, or write in a frame nothing uses and supply both ends of the circuit\. The second is cheap for a reason worth generalizing, since the space a trained model operates in occupies a modest fraction of its width, leaving room for channels that collide with nothing\. How much room, and whether it shrinks with scale, is left for future work\.

The tap of Section[4\.8](https://arxiv.org/html/2609.10210#S4.SS8)is the direction with the longest reach\. Driving a component the model trained for itself, from two layers upstream, using a read row taken from that component’s own weights, turns a legibility claim into a control surface\. The limit measured here is the accuracy with which that read row can be estimated rather than anything the model imposes, which suggests the constructions get better with better instruments rather than requiring a different architecture\.

An attention channel refuses an installed association for a structural reason rather than a tuning one: what a channel writes at a position is gathered from other positions, so the edit cannot be made local\. That explains as a constraint rather than a habit the pattern in the editing literature of writing predominantly to feed\-forward layers\. The restriction is specific to installing content into a channel; a gain on a channel’s existing scope works about as well as a gain on a unit’s, and an installed head serving as a marker for a unit downstream carries a condition that neither component holds alone\.

The forward\-looking version of all of this is a model that can be programmed and not merely trained\. New skills, factual updates, changes to behavior that would ordinarily require retraining can be installed; if the component carrying the behavior can be named and the edit’s reach measured, the collateral becomes a quantity rather than a surprise\. The measurements here are a beginning: they cover single components and single predictions\.

### 8\.1What the carriage result asks of the architecture

To read a component the lens has to separate predictive work from carriage, and what it separates out is most of the model\. At any position the great majority of contribution mass cancels, leaving a few dozen components to carry what survives\. The canceling mass is neither noise nor waste\. It is moving information into the few places that use it, and keeping a superposed residual stream organized so that many things can share it\.

That reframes the efficiency question\. The usual form is “which computations can be skipped”, and the honest answer from these measurements is that the surplus is not idle, so skipping it is not free\. The better form is the one this instrument makes askable: how much of a model’s capacity goes into carriage rather than into deciding, is that ratio a constant, and is it a property of the architecture or of the training? The evidence here says it is not a constant\. The arms trained for this paper share width, data, schedule and converged quality and differ only in the activation function, and their prediction graphs differ by a factor of four to seven\. Whatever determines how much machinery a prediction needs is a design variable\.

That raises the question of whether the carried part is compressible\. A fixed operator reproducing7474to7979percent of the update needs neither attention nor a feed\-forward block to supply it, and a layer whose sublayers only had to produce the remainder could be considerably thinner\. What the measurement says against this is specific: carriage and content are not separable subspaces\. The prediction lives inside the span the carriage map acts on, and a node’s write column lands there too, so factoring the map out of the architecture is not the same operation as factoring it out of the algebra\. The compressible object is the shared*map*rather than a*subspace*\. That distinction is what a serious attempt would have to respect, and it is why the obvious version — project the residual into a carriage subspace and a content subspace, and give each its own machinery — is the version that already fails\.

Redundancy is the other half of that question\. Started from the required graph and started from a random draw, the same search reaches sufficient sets with*no member in common*on a fifth of predictions \(Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)\), so training leaves more than one set of components able to carry the same answer\. What that costs, and whether a model could be built to carry less of it, is what the carriage asks in a different currency\. The caution from Appendix[C\.6](https://arxiv.org/html/2609.10210#A3.SS6)applies to any attempt: keeping only a pruned graph is not monotone in how much is kept, so an instrument that trims redundancy has to be checked against that failure before its numbers mean anything\.

This instrument can pose a stronger version of the same thought\. A component plausibly does double duty: a little prediction and a great deal of carriage\. If that is right, and if carriage admits many equivalent implementations, then where a design puts its parameters should matter less than how many it has, because the carriage absorbs the difference while the deciding part stays small and much the same\. Something like that is what the field observes — models of a given size land in a narrow band on downstream tasks across quite different designs — and it has not had a measurement attached\. Two measurements bear on it\. The closure sizes of Section[6](https://arxiv.org/html/2609.10210#S6)are1\.11\.1to3\.43\.4percent of a model’s components across seven architecture families, with no trend in size\. And about three quarters of the residual update is predictable from the state before it \(Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)\), which bounds how much of a model’s work between depths could be carriage\. If the carrying is that large a share and admits many implementations, the deciding part is what little is left, and it would take a great deal of design difference to move it\.

If that holds beyond the comparison made here, several things follow\. A measured per\-prediction dependency graph could replace the proxies for importance that currently guide conditional computation and early exit\. Architecture search optimizes quality against parameters and latency; the size of a prediction’s circuit is a third axis, it is now measurable, and nothing says the three are aligned\. And an architecture whose carriage is cheaper — one that needs less machinery to get information to the components that use it — would show up on this instrument as a smaller closure at equal quality, which is a target a designer can aim at directly\.

One reading of all this is uncomfortable\. A model that spends most of each layer moving information rather than computing with it may simply be inefficient, and the architecture may be the reason\. The opposing reading is at least as plausible and follows from the same data\. Carriage is everywhere because putting it everywhere is what makes the residual readable from anywhere, so whichever few components a prediction happens to need can find their inputs without the model having known in advance which those would be\. On that account the carriage is not overhead but the price of a general\-purpose bus, and a design that spent less on it would need to know its circuits ahead of time\. Distinguishing the two means building something that carries less and seeing what it loses, which is an architecture question this instrument can only set up\.

### 8\.2What a one\-component circuit says about the vocabulary

Some predictions rest on a single component\. On Gemma\-3 1B,4949of the134134predictions Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3)verifies are carried by one component out of206,336206\{,\}336—4343of them a feed\-forward unit and six an attention channel\. On3131of the4949the contribution ranking’s own top choice does not reproduce the token, so on those the component is found rather than nominated\. The sample examined below is the prefix\-and\-prune search’s draw on the same model,3737predictions of114114, and what those units predict is mostly punctuation and function words: eleven of the thirty\-seven write a mark rather than a word\.

That a comma can be produced by one unit invites a question about what the unit is for, and the sets answer it differently in different cases\. When each unit is transplanted to another position with the same target token, the general ones carry it and the specific ones do not: a period transfers on eight of twelve attempts and*in*,*and*and*a*transfer on nine of ten, against a control in which a component drawn at random carries no prediction on any model\. Commas do not transfer at all\. The two comma units in this sample are distinct components and neither produces the other’s comma, the behavior of a unit meaning “*this*word takes a comma” rather than one meaning “a comma belongs here”\.

If that reading survives a larger sample — and two units are an anecdote, not a measurement — it bears on where a vocabulary should put its boundaries\. Byte\-pair encoding forbids merges across whitespace and punctuation, and the work that lifts that restriction\[[31](https://arxiv.org/html/2609.10210#bib.bib42),[48](https://arxiv.org/html/2609.10210#bib.bib43)\]argues for it from compression and downstream accuracy\. None of the models measured here uses such a vocabulary: across the six tokenizers checked, spanning32,00032\{,\}000to262,145262\{,\}145entries, not one token joins a word to a following mark\. A word\-specific punctuation unit is what a model learning that association in parameters looks like\. That is evidence of a different kind from compression: the model has spent a component on knowing that a particular word takes a particular mark, which is exactly what a merged token would have supplied for free\. The measurement here is small and one\-sided, but it is the sort of evidence that has been missing, and this instrument produces it directly — for any candidate merge, ask whether the model devotes components to the association, and whether those components are specific to the pair\.

### 8\.3Where the components have no name

Which components carry a prediction can be found in every model measured\. What a component is*for*is harder, and the limit is the vocabulary: a component operating on a direction no token induces has no name there, and no lens can supply one, because the thing to be named is not lexical\. What such a component has is an address — the components upstream that write it \(Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)\) — which is what the gallery prints in place of a word\. Naming those quantities is the open problem; what follows is what is known about them\.

On the baseline, nearly three quarters of a unit’s response falls on tokens no concept explains, and Appendix[C\.13](https://arxiv.org/html/2609.10210#A3.SS13)rules out six accounts of it: sampling noise, several smaller concepts, a relational class, carriage seen from the write side, carriage seen from the read side, and weights the gradient never reached\. What survives is reproducible, single, and no more context\-bound than the part that can be named\.

Superposition, the obvious remaining guess, does not survive contact\[[17](https://arxiv.org/html/2609.10210#bib.bib5)\]\. If a token elicits a response through a direction the unit shares with many others, the response is leakage, and leakage is shared by construction: a small basis fitted across all units at once should account for the remainder while leaving the nameable part alone\. It does the reverse\. At every rank tried, on both models, the shared basis explains the*coherent*response better than the remainder —6868against5555percent at thirty\-two directions on the baseline\. The nameable tokens are the shared structure, which is what makes them recognizable as concepts\. What is left over is idiosyncratic to the unit\.

Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)says what that idiosyncratic part is\. It is written by identifiable components upstream, it arrives having been carried further than the nameable part, and it separates into categories once the terms are sorted by where they came from\. A component whose response is unit\-specific, reproducible, single, context\-independent, and assembled from several upstream sources is behaving like a condition with a list: fire if the input is any of these, then write the one thing this component writes\. The consequent cannot depend on which member fired, because a unit has one write column\. Under that reading the remainder is not a failure of naming\. The disjuncts of a disjunction need not resemble each other, and Equation[16](https://arxiv.org/html/2609.10210#S3.E16)tests precisely for resemblance, so a well\-formed condition with an unlovely list scores as incoherent while being perfectly well defined\. The quarter of a unit’s response that can be named is the part of its list that happens to arrive with few enough rotations to still lie near the vocabulary\.

What the categories are made of is the part this lens still does not settle\. They are partly grammatical — part\-of\-speech purity within a source group exceeds a random partition of the same sizes — and grammatical classes are exactly what Equation[16](https://arxiv.org/html/2609.10210#S3.E16)cannot see, since members of one are substitutable in a position rather than close in distribution\. Whether the rest is morphological, orthographic, or positional needs a method that names*features*rather than tokens, which is what a sparse autoencoder provides\[[12](https://arxiv.org/html/2609.10210#bib.bib15),[9](https://arxiv.org/html/2609.10210#bib.bib6)\]\.

### 8\.4Is this as good as it gets?

Finding a sufficient set efficiently is an open problem\. Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3)was built to find small sufficient sets in reasonable time, not provably smallest ones, and it converges on9696percent of predictions on some models and4444on others\. Every size reported here is an upper bound from that search\. We presume someone more clever will close this gap to100100percent, possibly with minimally sized circuits\.

The dependency graph follows an attention channel back to what it reads at the position being explained and no further \(Section[3\.3](https://arxiv.org/html/2609.10210#S3.SS3.SSS0.Px1)\)\. What a channel contributes is that value aggregated over the sequence, and following the pattern that does the aggregating leaves the position being explained and opens a graph over the whole context\. Every circuit in this paper is for one position; the graph over the context is the next instrument\.

The component pool is attention channels and feed\-forward units reading a sublayer’s input, so architectures shaped differently need extensions not built here: models computing their two sublayers in parallel from the same state, models gating a third class of writer into the residual at every layer, and sparse mixtures of experts, where a layer holds many feed\-forward experts and routes a few of them per token\. The first two want a wider pool\. The third asks something harder, because the router writes nothing into the residual and is credited nothing by an accounting that scores a component by what its own write places on the readout, while deciding which components exist at a position at all; and a circuit quoted as a share of the model no longer has one denominator to be a share of\. This paper had to draw the line somewhere\.

## 9Conclusion

This paper presents a lens: a way to look inside a transformer and say*which*components made a particular prediction\. It is derived from the model’s own parameters and activations, it fits nothing, and it ran unmodified over eighteen models\. What it returns is an accounting of which components carry a prediction, which of them it cannot survive losing, and which set is*sufficient*to produce it: on the baseline fifty\-three, thirteen and eight of46,08046\{,\}080\. The sufficient set is small everywhere — two to sixteen across twelve models trained by other people — and far below what the conventions in common use report for the same predictions\. Every size here is an upper bound returned by a search rather than a minimum, which leaves the smallest circuit an open question rather than a settled one\.

Identifying components is one thing and*naming*them is another\. A name requires knowing the geometry the computation runs in, and much of this paper is spent there\. Most of what a layer writes is not the prediction but the management of that geometry: three quarters of a layer’s update is a fixed map of the state it received, applied whatever the model is about to say, and at the readout the mass pushing away from the prediction is seven times the mass carrying it\. Once that is separated out, close to half of a model’s components can be named by the predictions they drive, and the strongest few inputs to a unit can be read from its own weights in the frame of its own layer\. Most of what drives a unit has no name in the vocabulary at all, and there the dependency graph supplies provenance in place of a word: sorted by which component upstream writes it, the remainder separates into categories a classifier recovers\.

A named component can be acted on\. A gain stays inside the scope the parameters name\. An association the model never held can be installed into one spare component with both halves read off the weights rather than traced or optimized\. A component the model trained for itself can be driven from upstream layers\. And a circuit can be built from components that did not exist before, carrying its intermediate state on a channel nothing in the model is oriented to read and committing its answer on the token axis, which is the one frame the readout answers\. And replacing the activation function with an order\-preserving one — the one controlled comparison here — puts a unit’s inputs at the instrument’s ceiling at parity quality, at the price of turning an install into a two\-part edit\.

Finding, naming, and editing, taken together, are a toolkit — and a roadmap for different ways to build a transformer, to train it, and to use it\.

## Appendix AGallery

Two predictions for each of the eighteen models, drawn mechanically\. Each prediction is shown in full: every member of its set is a row of the table, and every edge among them is in the graph\. Nothing is trimmed to fit\. Where a set is large, the table takes a page of its own and the graph a page turned sideways; smaller ones share a page\. What a table shows is the set that*builds*the prediction: keep these components, zero every other feed\-forward unit and attention channel, let the survivors recompute, and the model emits the same token \(Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)\)\. Nothing is spliced in — a component drawn here fires from the residual that remains, which is the token embedding plus whatever the other survivors have written\. These sets come from a prefix\-and\-prune search rather than from Algorithm[3](https://arxiv.org/html/2609.10210#S2.F3), so a panel is a verified sufficient set but is typically larger than the medians in Table[7](https://arxiv.org/html/2609.10210#S2.T7); a sufficient set is an upper bound, and either search returns a valid one\.

### A\.1How to read a page

##### The header\.

The context, the token the model predicts, and four numbers: the probability it assigns, how many components reach ninety percent of the net contribution, how many of them its prediction cannot survive losing, and what share of the total contribution cancels\.

##### The rows\.

Each row is one member of the set\. The unindented rows run from the components nearest the readout down to the ones furthest from it, and beneath each, indented, are its three strongest suppliers among the members, labeled with their share of*its*own incoming drive; the graph beneath the table carries every edge, not only those three\. A chain therefore reads down the page: a component that promotes the predicted token, then what fed it, then what fed that\. A node reached twice is written once and back\-referenced, because duplicating shared ancestors makes a prediction look like it rests on more machinery than it does\. Printed inboldare the members whose*removal*from the intact model changes the prediction, so one page carries both the components that build the answer and the smaller set it cannot lose\. Where the two do not coincide, a prediction’s necessary components are not the ones that reproduce it — which is worth seeing, and is most likely a consequence of there being more than one sufficient circuit\.

##### The columns\.

*Strength*is how hard the component fires, as a magnitude; which way it pushes is already carried by the write list\.*Fires on*lists the tokens that drive it, scored from its read row against a layer\-native token table\. A driver is printed in ink where it belongs to a group the token embedding can see and in gray where it does not, which is the split of Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2.SSS0.Px2)appearing on the page, token by token; a dash means the read row names nothing\.*Writes toward*lists the tokens its write column promotes, signed by how the component is firing\.666The sign matters\. In a gated feed\-forward the write value isϕ⁡\(gate\)⋅up\\phi\(\\text\{gate\}\)\\cdot\\text\{up\}and the up branch is linear, so a negatively firing unit whose column points away from the target is the one promoting it, and an unsigned decode would report the opposite\.*Answer*is where the predicted token sits in that list, read at the readout — the same quantity on every row, because every write list here is quoted at the readout\. A row reading \#1 promotes the answer; a row reading \#\>\>999 carries the prediction without promoting it, and is still read through its incoming edges, which say what its inputs carry\.

A row marked↢\\leftarrowtailis an attention channel, whose read side is what it picks up from the position it attends to rather than what sits at its own\. Its read side is measured from the value projection and its incoming edges are followed the same way a feed\-forward unit’s are \(Section[3\.3](https://arxiv.org/html/2609.10210#S3.SS3.SSS0.Px1)\)\. What is*not*followed is the attention pattern: the value a channel reads is aggregated over the sequence by weights this lens does not read\.

On an edge row the label beside the share names the tokens the writer and the reader share, meaning tokens both rank in their first eight, up to three of them; where they share none that highly, the edge prints its share alone\. That bar is met on between a quarter and three fifths of edges, depending on the arm\. Where the reader’s fires\-on list is printed, it is led by the first shared token, drawn in the gloss color\. The restraint is deliberate\. The agreement result of Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)is a statement about the population of edges, and it does not license reading one edge’s best token as a fact about that edge\. The writes\-toward list is the component’s own strongest tokens at the readout and is not led by anything\.

In the graph beneath each table, every component in the net at the readout, whether or not it also feeds another component in the set, is joined to an answer node at the right, and that edge is labeled with where the answer sits in what the writer promotes, the table’s last column\.

##### Glosses\.

Where a component’s tokens fall into a nameable category, a short gloss is printed beside them in italics\. The gloss is ours and the tokens are the model’s; it sits next to them rather than in place of them, so a reader can see what it was derived from and disagree\.

##### Concepts\.

An edge is drawn dashed and led by⟨A⟩\\langle A\\rangle,⟨B⟩\\langle B\\ranglewhen writer and reader share no token at all: a quantity with no word but with an address, named by the component that wrote it, which carries the same tag in its writes\-toward column\. Tagging by source rather than with a single placeholder makes reuse visible, since one internal quantity feeding three readers is a fact about the circuit that a uniform label hides\. Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)is what licenses the address — these quantities are written by identifiable components even where no vocabulary describes them\.

##### The graph\.

Beneath each table, the same components in the same relation, with depth running left to right and the answer at the right\. Every member is drawn, since every member has at least its edge to the answer\.

### A\.2How these traces were chosen

Up to twenty predictions were drawn at random from held\-out text under filters fixed in advance, and the draw stops early where the filters exhaust the sampled contexts first — eleven predictions on the baseline, twenty on most of the models taken off the shelf\. The model’s top choice must carry probability at least0\.300\.30, so there is a decision to explain rather than a coin flip; the context must not be a run of one repeated token; and the predicted token must contain a letter, so the sample is not dominated by punctuation\. Of those twenty, the two whose sufficient set is nearest the*median*of the draw were kept\. Selecting for typicality avoids flattering the claim this paper makes about circuit size, as selecting the smallest would have\. Nothing about a trace’s content, or how readable it turned out to be, entered the choice\.

That holds for the first group, two predictions per model\. Most of them use no concepts at all: on the great majority of the predictions drawn this way every edge has some token that both its writer and its reader rank, whether or not the agreement clears the bar for printing it\. A prediction that needs nothing the vocabulary cannot say is the common case, and a gallery that showed only the interesting minority would misrepresent how often it arises\. So the mechanically drawn pages come first and are left as they fell\. The predictions after them are selected*because*they contain edges no token describes, and are labeled as chosen for it\.

The six models trained for this paper come first, in the order of Section[5](https://arxiv.org/html/2609.10210#S5), followed by the twelve we did not train, ordered by size\.

### A\.3What the conventions rest on

An edge is drawn as a concept when its writer and reader share no token at all among a few hundred candidates, so that there is no token\-level content to report and the row does not show a bad list in the same style as a good one\. That happens on a median of22percent of the edges on these pages per model, and on one model far more: OPT\-125M runs at6565percent, and the pooled figure across all eighteen,1111percent, is almost entirely that one model\.

The edge tokens carry a caveat the pages cannot show\. Scored against twenty random controls rather than one,1313percent of the baseline’s edges and1111percent of the sigmoid model’s have a writer that beats its controls atp≤0\.05p\\leq 0\.05; the median edge sits atp≈0\.35p\\approx 0\.35\. The aggregate claim is solid — across twenty\-three thousand edges the real writer wins on6363percent against3232, which is not a close call — but it rests on many weakly informative edges rather than on a subset of strong ones, and the token on any*single*edge is correspondingly weak evidence\. These pages illustrate what a trace looks like rather than claiming anything about particular edges\.

Two things a reader should expect to see\. On the conventional baseline the required components are often attention channels, whose read side is what they collect from elsewhere in the sequence rather than what sits at the position — the boundary of Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)appearing in a concrete trace rather than as a caveat\. And a source’s share of a node’s drive can exceed one hundred percent, because the denominator is the*net*drive and other sources cancel against it; that is the same accounting Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)is about, visible one level down\.

Figure 34:baseline \(GELU\), prediction 1: the set, 28 members\. Of 11 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 24\.Figure 35:baseline \(GELU\), prediction 1: the graph, 28 members and 20 edges among them\.Figure 36:baseline \(GELU\), prediction 2: the set, 24 members\. Of 11 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 24\.Figure 37:baseline \(GELU\), prediction 2: the graph, 24 members and 68 edges among them\.Figure 38:sigmoid, prediction 1: the set, 21 members\. Of 18 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 19\.Figure 39:sigmoid, prediction 1: the graph, 21 members and 20 edges among them\.Figure 40:sigmoid, prediction 2: the set, 22 members\. Of 18 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 19\.Figure 41:sigmoid, prediction 2: the graph, 22 members and 21 edges among them\.Figure 42:softplus, prediction 1: the set, 18 members\. Of 15 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 18\.Figure 43:softplus, prediction 1: the graph, 18 members and 11 edges among them\.Figure 44:softplus, prediction 2: the set, 14 members\. Of 15 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 18\.Figure 45:softplus, prediction 2: the graph, 14 members and 25 edges among them\.Figure 46:ReLU, prediction 1: the set, 23 members\. Of 14 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 47:ReLU, prediction 1: the graph, 23 members and 65 edges among them\.Figure 48:ReLU, prediction 2: the set, 24 members\. Of 14 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 49:ReLU, prediction 2: the graph, 24 members and 73 edges among them\.Figure 50:sigmoid, no shaping, prediction 1: the set, 24 members\. Of 16 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 25\.Figure 51:sigmoid, no shaping, prediction 1: the graph, 24 members and 61 edges among them\.Figure 52:sigmoid, no shaping, prediction 2: the set, 27 members\. Of 16 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 25\.Figure 53:sigmoid, no shaping, prediction 2: the graph, 27 members and 48 edges among them\.Figure 54:set operators, prediction 1: the set, 18 members\. Of 13 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 55:set operators, prediction 1: the graph, 18 members and 9 edges among them\.Figure 56:set operators, prediction 2: the set, 26 members\. Of 13 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 57:set operators, prediction 2: the graph, 26 members and 22 edges among them\.Figure 58:GPT\-2 124M, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 8\.Figure 59:OPT 125M, prediction 1: the set, 11 members\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 60:OPT 125M, prediction 1: the graph, 11 members and 41 edges among them\.Figure 61:OPT 125M, prediction 2\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 62:SmolLM2 360M, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 10\.Figure 63:Qwen2\.5 0\.5B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 9\.Figure 64:Gemma\-3 1B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 6\.Figure 65:OLMo\-2 1B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 6\.Figure 66:Llama\-3\.2 1B, prediction 1\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 67:Llama\-3\.2 1B, prediction 2: the set, 13 members\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 68:Llama\-3\.2 1B, prediction 2: the graph, 13 members and 11 edges among them\.Figure 69:TinyLlama 1\.1B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 5\.Figure 70:Qwen2\.5 1\.5B, predictions 1 and 2\. Of 19 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 6\.Figure 71:Llama\-3\.2 3B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 8\.Figure 72:Qwen2\.5 7B, predictions 1 and 2\. Of 17 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 4\.Figure 73:Mistral 7B, predictions 1 and 2\. Of 20 predictions drawn at random, these are two of the four whose sufficient set is nearest the median, 2\.
### A\.4Traces where the vocabulary runs out

The pages above were drawn without regard to their content, and most of them use no concepts\. These were selected for the opposite reason: each carries edges that no token describes, so that the case the rest of this paper argues about can be seen rather than described\. They are not a sample of anything\. OPT\-125M leads them because it is genuinely unlike the others here — about two thirds of the edges in its traces carry no word, against a few percent everywhere else — and it is the one model in the set built on ReLU, whose units are silent on the whole negative half\-line\.

Figure 74:OPT 125M, prediction 3: the set, 16 members\. Selected for containing edges no token describes\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 75:OPT 125M, prediction 3: the graph, 16 members and 59 edges among them\.Figure 76:OPT 125M, prediction 4: the set, 16 members\. Selected for containing edges no token describes\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 13\.Figure 77:OPT 125M, prediction 4: the graph, 16 members and 63 edges among them\.Figure 78:sigmoid, no shaping, prediction 3: the set, 22 members\. Selected for containing edges no token describes\. Of 16 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 25\.Figure 79:sigmoid, no shaping, prediction 3: the graph, 22 members and 71 edges among them\.Figure 80:sigmoid, no shaping, prediction 4: the set, 26 members\. Selected for containing edges no token describes\. Of 16 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 25\.Figure 81:sigmoid, no shaping, prediction 4: the graph, 26 members and 47 edges among them\.Figure 82:baseline \(GELU\), prediction 3: the set, 25 members\. Selected for containing edges no token describes\. Of 11 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 24\.Figure 83:baseline \(GELU\), prediction 3: the graph, 25 members and 11 edges among them\.Figure 84:baseline \(GELU\), prediction 4: the set, 20 members\. Selected for containing edges no token describes\. Of 11 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 24\.Figure 85:baseline \(GELU\), prediction 4: the graph, 20 members and 42 edges among them\.Figure 86:ReLU, prediction 4: the set, 25 members\. Selected for containing edges no token describes\. Of 14 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 87:ReLU, prediction 4: the graph, 25 members and 70 edges among them\.Figure 88:SmolLM2 360M, prediction 4\. Selected for containing edges no token describes\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 10\.Figure 89:sigmoid, prediction 4: the set, 19 members\. Selected for containing edges no token describes\. Of 18 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 19\.Figure 90:sigmoid, prediction 4: the graph, 19 members and 58 edges among them\.Figure 91:set operators, prediction 3: the set, 22 members\. Selected for containing edges no token describes\. Of 13 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 22\.Figure 92:set operators, prediction 3: the graph, 22 members and 25 edges among them\.Figure 93:Llama\-3\.2 3B, prediction 4\. Selected for containing edges no token describes\. Of 20 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 8\.Figure 94:Qwen2\.5 7B, prediction 4\. Selected for containing edges no token describes\. Of 17 predictions drawn at random, this is one of the four whose sufficient set is nearest the median, 4\.

## Appendix BThe set\-operator model and how it is trained

Section[5](https://arxiv.org/html/2609.10210#S5)treats the set\-operator construction as one arm among four, and says only as much about it as the comparison needs\. It was built across three earlier papers\[[42](https://arxiv.org/html/2609.10210#bib.bib9),[40](https://arxiv.org/html/2609.10210#bib.bib10),[39](https://arxiv.org/html/2609.10210#bib.bib8)\], and the ingredient that lets it run at every unit of every layer comes from a fourth\[[41](https://arxiv.org/html/2609.10210#bib.bib14)\]\. This appendix collects the construction and its training in one place: the unit, the attention head, the pressure applied during training, the two shaping terms, and the recipe\. Nothing here is new to those papers except the assembly\.

### B\.1The feed\-forward unit

A conventional unit reads one row and writes one column through a scalar nonlinearity\. A set\-operator unit reads two rows and writes two columns, and what it computes between them is a pair of set operations on two fuzzy memberships\. Withzzthe normalized residual at a position, unituuforms its operands as in Equation[30](https://arxiv.org/html/2609.10210#S5.E30),

Au=σ\(wua⋅z\),Bu=σ\(wub⋅z\),Au,Bu∈\(0,1\),A\_\{u\}=\\sigma\\big\(w^\{a\}\_\{u\}\\cdot z\\big\),\\qquad B\_\{u\}=\\sigma\\big\(w^\{b\}\_\{u\}\\cdot z\\big\),\\qquad A\_\{u\},B\_\{u\}\\in\(0,1\),\(37\)and writes the intersection and the difference of the two, each through its own column:

FFN\(z\)=gℓr⁡\(z\)∑u\[Au​Bu⏟A∩Bcu∩\+Au​\(1−Bu\)⏟A∖Bcu∖\],r\(z\)=rms\(A∩B∥A∖B\),\\mathrm\{FFN\}\(z\)\\;=\\;\\frac\{g\_\{\\ell\}\}\{r\(z\)\}\\sum\_\{u\}\\Big\[\\underbrace\{A\_\{u\}B\_\{u\}\}\_\{A\\cap B\}\\;c^\{\\cap\}\_\{u\}\\;\+\\;\\underbrace\{A\_\{u\}\\,\(1\-B\_\{u\}\)\}\_\{A\\setminus B\}\\;c^\{\\setminus\}\_\{u\}\\Big\],\\qquad r\(z\)=\\operatorname\{rms\}\\big\(A\\cap B\\,\\\|\\,A\\setminus B\\big\),\(38\)wherer⁡\(z\)r\(z\)is the root\-mean\-square of the30723072operation values at that position andgℓg\_\{\\ell\}is one learned scalar per layer\. That normalization is the block’s only departure from a plain sum: the operation values are bounded in\[0,1\]\[0,1\]where a GELU activation is not, and the normalization lets the block set its own scale rather than inherit the interval’s\. It is a scalar the model computes, which is how Table[52](https://arxiv.org/html/2609.10210#A3.T52)treats it\.

The choices in that line each do a job\. The memberships are squashed into\[0,1\]\[0,1\]because that interval is closed under complement, which is what makes negation a primitive:1−B1\-Bis “notBB” in the same units asBB, where a raw direction has no absence\. The operations are the product t\-norm rather thanmin\\minandmax\\max, because the product trains and the lattice operations saturate\. And the two operations chosen are intersection and difference, which between them span conjunction and negation\-under\-a\-condition, the two things a single\-row unit cannot express at all\.

Rearranged, and setting the scale aside, the sum in Equation[38](https://arxiv.org/html/2609.10210#A2.E38)reads

∑uAu​\[cu∖\+Bu​\(cu∩−cu∖\)\],\\sum\_\{u\}A\_\{u\}\\Big\[\\,c^\{\\setminus\}\_\{u\}\+B\_\{u\}\\,\\big\(c^\{\\cap\}\_\{u\}\-c^\{\\setminus\}\_\{u\}\\big\)\\Big\],\(39\)which is how the two operands divide the work\.AuA\_\{u\}is a gate: it scales the unit’s whole contribution and is zero when the unit is off\.BuB\_\{u\}is a selector: it moves the write continuously between the two fixed columns, delivering exactlycu∖c^\{\\setminus\}\_\{u\}atBu=0B\_\{u\}=0and exactlycu∩c^\{\\cap\}\_\{u\}atBu=1B\_\{u\}=1\. Both endpoints are exact rather than approximate, and Section[5\.12](https://arxiv.org/html/2609.10210#S5.SS12)is built on that: keyingBuB\_\{u\}on a token makes the difference branch vanish identically where the token is present\.

The layer is held to the baseline’s parameter count by narrowing it\. A conventional layer of widthdff=3072d\_\{\\mathrm\{ff\}\}=3072has2​d​dff2\\,d\\,d\_\{\\mathrm\{ff\}\}parameters in its two projections\. A set\-operator layer withppoperand pairs has2​d​p2\\,d\\,pin the read rows and2​p​d2\\,p\\,din the write columns, sop=dff/2=1536p=d\_\{\\mathrm\{ff\}\}/2=1536pairs make the two equal\. The write projection is therefore the same3072×d3072\\times dmatrix as the baseline’s, with its first15361536columns carryingc∩c^\{\\cap\}and its last15361536carryingc∖c^\{\\setminus\}; only the activation feeding it has changed\. Every unit in every layer is of this kind\. The earlier papers ran the construction on a fraction of each layer with conventional units alongside as a gradient path, because the pure form did not train; the arm here has none\.

### B\.2The attention head

The same construction is applied on the value side of every attention head, so that what a head gathers from a position is bounded in the same way a unit’s operands are\. A head’s value vectorv∈ℝ64v\\in\\mathbb\{R\}^\{64\}is split in half and each half becomes a membership,

A=σ\(v1:32\),B=σ\(v33:64\),v~=\[A⊙B∥A⊙\(1−B\)\]∈\[0,1\]64,A=\\sigma\\big\(v\_\{1:32\}\\big\),\\qquad B=\\sigma\\big\(v\_\{33:64\}\\big\),\\qquad\\tilde\{v\}=\\big\[\\,A\\odot B\\;\\\|\\;A\\odot\(1\-B\)\\,\\big\]\\in\[0,1\]^\{64\},\(40\)andv~\\tilde\{v\}takes the place ofvvin the attention average\. The query, the key, the softmax over positions and the output projection are unchanged, and no parameter is added: the head’s6464value channels are reinterpreted as3232intersections and3232differences\. What attention delivers at a position is then an average of bounded quantities, which is the property Section[3\.3](https://arxiv.org/html/2609.10210#S3.SS3.SSS0.Px1)reads and Section[5\.10](https://arxiv.org/html/2609.10210#S5.SS10)pays for — a channel gathers nothing far from its resting level unless a key row is scaled for it\.

Everything else is the baseline of Section[2\.1](https://arxiv.org/html/2609.10210#S2.SS1): twelve layers of widthd=768d=768, twelve heads, learned positional embeddings, pre\-norm blocks, no bias anywhere, and tied input and output embeddings\. The parameter count is the baseline’s but for the one scalargℓg\_\{\\ell\}per layer\.

### B\.3The pressure

Left to the language\-modeling loss alone, the pure construction did not train: above roughly half conversion the earlier papers found training diverging late, sooner the larger the converted fraction\. What lets it run everywhere is an auxiliary objective from[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14), where it was introduced for a different purpose — to prescribe where in the stack a conventional transformer holds its intermediate content relative to the readout\.

Writeht\(ℓ\)h^\{\(\\ell\)\}\_\{t\}for the residual after layerℓ\\ellat positiontt, and decode it through the tied readout as the logit lens does,

zt\(ℓ\)=LN⁡\(ht\(ℓ\)\)​E⊤,z~t\(ℓ\)=zt\(ℓ\)−meanv⁡zt,v\(ℓ\),z^\{\(\\ell\)\}\_\{t\}=\\mathrm\{LN\}\\big\(h^\{\(\\ell\)\}\_\{t\}\\big\)\\,E^\{\\top\},\\qquad\\tilde\{z\}^\{\(\\ell\)\}\_\{t\}=z^\{\(\\ell\)\}\_\{t\}\-\\operatorname\{mean\}\_\{v\}\\,z^\{\(\\ell\)\}\_\{t,v\},\(41\)centering over the vocabulary\. Letz~t\\tilde\{z\}\_\{t\}be the same quantity for the model’s final logits\. Nothing is held fixed: the term’s gradient reaches both the layer’s decode and the final prediction, so the pressure can meet its target by moving either\. The objective asks each layer’s decode to sit at a prescribed angle to the final prediction,

ℒ∠=αL​∑ℓ=0L−1𝔼t​\[\(cos⁡∠⁡\(z~t\(ℓ\),z~t\)−cos⁡θℓ\)2\],cos⁡∠⁡\(a,b\)=⟨a,b⟩‖a‖​‖b‖,\\mathcal\{L\}\_\{\\angle\}\\;=\\;\\frac\{\\alpha\}\{L\}\\sum\_\{\\ell=0\}^\{L\-1\}\\;\\mathbb\{E\}\_\{t\}\\Big\[\\big\(\\cos\\angle\(\\tilde\{z\}^\{\(\\ell\)\}\_\{t\},\\,\\tilde\{z\}\_\{t\}\)\-\\cos\\theta\_\{\\ell\}\\big\)^\{2\}\\Big\],\\qquad\\cos\\angle\(a,b\)=\\frac\{\\langle a,b\\rangle\}\{\\\|a\\\|\\,\\\|b\\\|\},\(42\)with the target angle a step over depth,

θℓ=\{90∘,ℓ=0,…,5,0∘,ℓ=6,…,11\.\\theta\_\{\\ell\}\\;=\\;\\begin\{cases\}90^\{\\circ\},&\\ell=0,\\dots,5,\\\\\[2\.0pt\] 0^\{\\circ\},&\\ell=6,\\dots,11\.\\end\{cases\}\(43\)The first half of the stack is asked to keep everything it computes orthogonal to the answer, and the second half to point at it\. That is the two\-phase structure[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14)measured in the conventional baseline — a concept phase held off the readout axis, a token phase committed to it — written down as a target rather than left to emerge\. The expectation runs over128128positions drawn at random from each batch, the same positions for every layer; the coefficient isα=3\\alpha=3and is held constant through training, the term is added to the cross\-entropy, and it is switched off at evaluation\. It touches no parameter and inserts nothing into the forward pass\.

Two facts about this pressure are established elsewhere and matter here\. On the conventional model it is a lottery: prescribing the step through the loss alone,[Oskin \[41\]](https://arxiv.org/html/2609.10210#bib.bib14)report six of eight seeds diverging outright, and the survivors reaching the geometry under strain\. That paper’s remedy is a fixed orthogonal rotation inserted at the boundary layer, which supplies the frame the loss asks for instead of making the model grow one\. The set\-operator arm uses no such device\. Under the loss alone, all eight seeds converge, inside0\.20\.2nats of one another, at a quality inside the baseline’s converged range \(Table[24](https://arxiv.org/html/2609.10210#S5.T24)\)\. The constraint that a conventional transformer meets only by luck, and pays for in quality when it does, this construction meets every time for nothing\. No run of this arm without the schedule is part of this paper, so the comparison is with the earlier papers’ runs rather than a paired ablation\. What those runs suggest is that a bounded stack left to itself has to find the concept–token boundary on its own and to commit on the readout axis at every depth at once, and saturates trying; the schedule tells it where the boundary is\. Why a bounded two\-operand unit is at ease with a geometry a GELU unit is not, and whether that reading is the whole of it, are not settled in this paper; Section[5\.8](https://arxiv.org/html/2609.10210#S5.SS8)finds the construction’s legibility comes from its order\-preserving operands rather than from the set operations, and the trainability may have the same source\.

### B\.4Two shaping terms

Two small auxiliary terms act on the operands during training, both at coefficient0\.0030\.003\. Writingvvfor an operand’s value and the means over the operands of a layer and the tokens of a batch,

ℒcrisp=0\.003​mean⁡\[v⁡\(1−v\)\],ℒvar=0\.003​meanu​\[max⁡\(0,1−Vart⁡vuτ\)\],τ=0\.003\.\\mathcal\{L\}\_\{\\mathrm\{crisp\}\}=0\.003\\;\\operatorname\{mean\}\\big\[\\,v\\,\(1\-v\)\\,\\big\],\\qquad\\mathcal\{L\}\_\{\\mathrm\{var\}\}=0\.003\\;\\operatorname\{mean\}\_\{u\}\\Big\[\\max\\Big\(0,\\;1\-\\frac\{\\operatorname\{Var\}\_\{t\}\\,v\_\{u\}\}\{\\tau\}\\Big\)\\Big\],\\quad\\tau=0\.003\.\(44\)The first rewards an operand for sitting near00or11rather than in the middle of its range\. The second is a floor on how much an operand varies across the tokens of a batch: an operand that has settled at a constant, and so responds to nothing, pays until its variance clearsτ\\tau\. Both are applied to the feed\-forward operands and to the attention memberships of Equation[40](https://arxiv.org/html/2609.10210#A2.E40)\. The crispness term is also the “shaping” of Table[2](https://arxiv.org/html/2609.10210#S2.T2), and it is the one of the two that is defined only for a bounded activation, sincev⁡\(1−v\)v\(1\-v\)needsv∈\[0,1\]v\\in\[0,1\]to be a penalty at all; that is why the softplus and ReLU arms carry it at zero\. The sigmoid arm of Section[5](https://arxiv.org/html/2609.10210#S5)was trained with and without it, and the two agree on the measurements that comparison rests on, so the term is not what produces those results\. They do not agree everywhere: the two part company on where a model delivers context over depth, which Section[3\.3](https://arxiv.org/html/2609.10210#S3.SS3.SSS0.Px1)reports and leaves open\.

### B\.5The recipe

Training is the baseline’s \(Section[2\.1](https://arxiv.org/html/2609.10210#S2.SS1)\): one epoch of OpenWebText at batch size1616and context length20482048, AdamW withβ=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\)and weight decay0\.10\.1, peak learning rate3×10−43\\times 10^\{\-4\}with20002000warmup steps and cosine decay to a tenth of peak,272,687272\{,\}687steps, gradient clipping, andbf16on a single GPU\. The auxiliary terms are added to the cross\-entropy from the first step with their coefficients fixed\. Nothing was tuned for this arm: it inherits the baseline’s learning rate and warmup unchanged\. Eight seeds were trained under the screen of Section[2\.1\.1](https://arxiv.org/html/2609.10210#S2.SS1.SSS1), and all eight pass it\.

What the rest of the paper reads from the result is the same as for any other arm, with one substitution\. A unit’s read side is operandAA’s rowwuaw^\{a\}\_\{u\}, and the agreement figures of Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)are taken on it; Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)also readswubw^\{b\}\_\{u\}and finds both operands nameable at roughly twice the baseline’s single row\. A unit’s write side is two columns rather than one, and an edit to one of them lands on one branch of the unit \(Section[5\.9](https://arxiv.org/html/2609.10210#S5.SS9)\)\. The second operand is the one thing this construction has that the others do not, and Section[5\.12](https://arxiv.org/html/2609.10210#S5.SS12)is what it turns out to be worth\.

## Appendix CAsh heap

Every method below is the sensible thing to do — the obvious statistic, the standard method, the natural correction — and each either does not work or, worse, returns a confident number pointing the wrong way\. The failure that matters is the one that returns 16\.9 times the null on a model that is measurably rank\-one, or that ranks a diverged checkpoint as the most legible thing in the study\. They are set out at length because anyone repeating this measurement will reach for the same ones\. Maybe they can find a way to make them work\. Maybe the text here will spare them the trouble\. They fall into five families: the models can rig the comparison, the estimators are biased, the cuts and summaries are arbitrary, the ablation searches fail their own checks, and the standard reading methods do not survive contact with a conventional unit\. After those come the failures specific to naming a component — which of Section[3](https://arxiv.org/html/2609.10210#S3)’s negative results are the instrument’s fault, the conditions under which its numbers hold, fitting a unit’s drivers from upstream, reading a write column against the vocabulary, and naming a unit by where it fires — and, last, what the failures leave: the tests that say what a unit’s unnameable remainder is\. The dependency graph has an appendix of its own \(Appendix[D](https://arxiv.org/html/2609.10210#A4)\)\.

### C\.1The models can rig the comparison

##### A worse model reads as more legible\.

The largest failure is the one Table[1](https://arxiv.org/html/2609.10210#S2.T1)records\. Define the effective dimensionality of a layer’s token table as the participation ratio of its eigenvalue spectrum,

PR⁡\(E~​\[l\]\)=\(∑iλi\)2∑iλi2,λi​the eigenvalues of​Covt​\(E~​\[l,t\]\),\\mathrm\{PR\}\\big\(\\tilde\{E\}\[l\]\\big\)=\\frac\{\\big\(\\sum\_\{i\}\\lambda\_\{i\}\\big\)^\{2\}\}\{\\sum\_\{i\}\\lambda\_\{i\}^\{2\}\},\\qquad\\lambda\_\{i\}\\text\{ the eigenvalues of \}\\mathrm\{Cov\}\_\{t\}\\big\(\\tilde\{E\}\[l,t\]\\big\),\(45\)which reads as a count of independent directions\. Averaged over the first five layers a converged seed carries 98\.4 to 106\.7 of them, and never falls below 75\.8 at any depth\. The excluded seeds carry 30\.3 to 45\.7, and two of them collapse almost completely at a single layer: 2\.2 and 1\.1 independent directions at layer 1, out of 768\. A collapsed representation is easier to predict because there is less variety left to predict, and the excluded seeds accordingly score 40\.7 to 49\.2 percent onov​@​8\\mathrm\{ov\}@8against 33\.6 to 38\.4 for the five that converged\.*Every excluded seed reads as more legible than every converged one\.*Including a single one would shrink the gap this paper later reports by about a third; the effect is large enough, and in the wrong direction often enough, that quality is reported beside every legibility figure in this paper\.

##### A fairness correction that rewards silence\.

A unit that no token clearly drives has a noisy causal ranking and therefore scores low, which suggests dropping such units before comparing models\. The correction backfires, because the model with more of them gains more from their removal: dropping units with no clear trigger lifts the baseline by about six points\. The correction rewards whichever model has more such units, so applying it in the interest of fairness silently favors one side of any comparison it is used in\. The uncorrected number is reported\.

### C\.2The estimators are biased

##### The dimensionality metric is biased both ways\.

Equation[45](https://arxiv.org/html/2609.10210#A3.E45)looks safer than it is for being scale\-free and parameter\-free\. Each row ofE~\\tilde\{E\}is a mean over\|𝒞\|\|\\mathcal\{C\}\|contexts and carries estimation noise, and that noise is approximately isotropic indddimensions, so the measured covariance is

Covmeas≈Covtrue\+σwithin2\|𝒞\|​I,\\mathrm\{Cov\}\_\{\\text\{meas\}\}\\;\\approx\\;\\mathrm\{Cov\}\_\{\\text\{true\}\}\\;\+\\;\\frac\{\\sigma^\{2\}\_\{\\text\{within\}\}\}\{\|\\mathcal\{C\}\|\}\\,I,\(46\)withσwithin2\\sigma^\{2\}\_\{\\text\{within\}\}the variance of a token’s state across contexts\. An isotropic ridge pulls the participation ratio toward the ambient dimension, so Equation[45](https://arxiv.org/html/2609.10210#A3.E45)*over*states dimensionality\. Worse, the ridge is not constant across the models being compared:σwithin2\\sigma^\{2\}\_\{\\text\{within\}\}measures how much a token’s representation moves with context, and a better model has more of it, so the correction needed is largest exactly where the measurement is being used to argue\. Across the seeds hereσwithin2\\sigma^\{2\}\_\{\\text\{within\}\}varies by a factor of seven hundred\. A second bias runs the other way, since sample eigenspectra are over\-dispersed relative to the population, which makes Equation[45](https://arxiv.org/html/2609.10210#A3.E45)*under*state dimensionality, most severely for the seeds with the highest true value\.

Neither bias can be argued away, so both are removed by measurement\. Splitting the contexts into two disjoint halves and taking the participation ratio of the symmetrized cross\-covariance

C=12​\(M\+M⊤\),M=1\|V\|​∑t∈V\(E~1​\[l,t\]−E~¯1\)​\(E~2​\[l,t\]−E~¯2\)⊤C=\\tfrac\{1\}\{2\}\\big\(M\+M^\{\\top\}\\big\),\\qquad M=\\frac\{1\}\{\|V\|\}\\sum\_\{t\\in V\}\\big\(\\tilde\{E\}\_\{1\}\[l,t\]\-\\bar\{\\tilde\{E\}\}\_\{1\}\\big\)\\big\(\\tilde\{E\}\_\{2\}\[l,t\]\-\\bar\{\\tilde\{E\}\}\_\{2\}\\big\)^\{\\\!\\top\}\(47\)removes the ridge, because noise independent between the halves contributes nothing in expectation\. Doing so moves every seed by less than one unit ofPR\\mathrm\{PR\}\. In this case the objection does not bite\.

##### A small table understates\.

Agreement depends on\|𝒞\|\|\\mathcal\{C\}\|in Equation[7](https://arxiv.org/html/2609.10210#S2.E7)and is still rising at 32 contexts; going from 4 to 32 raises baselineov​@​8\\mathrm\{ov\}@8from 34\.6 to 40\.2 percent\. The estimand is a property of the model rather than of the sample, so a larger table is a better estimate rather than a thumb on the scale\. Two consequences follow\. Absolute figures must be quoted with the table size that produced them\. And a paper that splits its contexts evenly between building the table and testing against it, as the obvious protocol does, is deliberately halving the quality of its own instrument for a hygiene requirement that costs nothing to satisfy another way\.

##### Scoring on the table’s own contexts\.

If𝒞\\mathcal\{C\}and𝒞′\\mathcal\{C\}^\{\\prime\}overlap, thensu​\(t\)s\_\{u\}\(t\)in Equation[13](https://arxiv.org/html/2609.10210#S3.E13)is, up to centering, the mean pre\-activation over the same contexts whose mean activation definesa¯u​\(t\)\\bar\{a\}\_\{u\}\(t\)\. The two sides then share their sampling noise, and for a monotoneϕ\\phithe comparison reduces to a question about the curvature ofϕ\\phirather than about the parameters\. Overlapping the two sets is cheap\. The flag that shares the contexts also doubles the table the naming side is built from, and it is the table that moves the number: holding the table at sixteen contexts, overlapping against disjoint is39\.239\.2against38\.938\.9under the ranked summary and59\.759\.7against59\.559\.5under the mean\. The leakage is about a third of a point\. Every number in this paper uses disjoint sets regardless, which costs nothing and settles the question rather than arguing it\.

##### The reproducibility floor\.

Recomputed from an independent sample of contexts, a baseline unit’s top\-KKlist agrees with itself, atK=8K=8, 64 percent of the time at eight contexts per half and 86 percent at ninety\-six\. Any statement about how well two*different*procedures agree is meaningless without that floor, since a procedure that cannot reproduce itself cannot be expected to match anything else\. Measured against it, several apparently large disagreements fall inside the range the ground truth varies over on its own\.

### C\.3The cuts and summaries are arbitrary

##### Any singleKKis an arbitrary cut\.

Define a unit’s effective set size from its causal profile, as the participation ratio of the response in excess of the unit’s own median:

eu​\(t\)=max⁡\(0,a¯u​\(t\)−medt⁡a¯u​\(t\)\),nu=\(∑teu​\(t\)\)2∑teu​\(t\)2\.e\_\{u\}\(t\)=\\max\\\!\\big\(0,\\;\\bar\{a\}\_\{u\}\(t\)\-\\operatorname\{med\}\_\{t\}\\bar\{a\}\_\{u\}\(t\)\\big\),\\qquad n\_\{u\}=\\frac\{\\big\(\\sum\_\{t\}e\_\{u\}\(t\)\\big\)^\{2\}\}\{\\sum\_\{t\}e\_\{u\}\(t\)^\{2\}\}\.\(48\)A baseline unit hasnu=115n\_\{u\}=115at the median, quartiles 82 and 148 \(Figure[95](https://arxiv.org/html/2609.10210#A3.F95)\)\. Which token lands at rankKKrather thanK\+1K\+1is close to a coin flip, so a measurement read at oneKKspends most of its resolution on boundary churn\. Reportingov^​@​K\\widehat\{\\mathrm\{ov\}\}@Kover a sweep ofKKrather than at a chosen cut is the only version of this measurement reported here\.

Figure 95:Causal response of a baseline unit across candidate tokens, sorted from strongest, median over units\. The response spreads over roughly 115 of 512 candidates and does not fall below a tenth of its peak until rank 111\. A list of any fixed length samples the leftmost sliver: the top eight capture 18 percent of the total, the top thirty\-two 42 percent\. Context length 2048, 32 contexts\.
##### Thresholds chosen in the wrong coordinates\.

Almost every quantity here needs a cutoff somewhere, and a cutoff is a claim about where the interesting part of a distribution lies\. Four separate measurements in this work failed for that reason alone\. A threshold placed at the midpoint of an activation’s nominal range measured silence rather than selectivity, because the units in question operate in the lowest seventh of that range\. A range set by the first and ninety\-ninth percentiles discarded the top one percent of a unit’s response, which for a selective unit is the entire signal\. Quantile bins put resolution where the probability mass is rather than where the signal is, and were non\-monotone in the number of bins as a result\. And a criterion counting tokens more than three standard deviations*above*a unit’s mean is correct for a right\-skewed quantity and silently discards half the evidence for a symmetric one\. The general form is the same each time: a cutoff was chosen in coordinates that suited one of the things being compared\. The defense that works is to plot the distribution before choosing, and to build in a check the measurement must pass, such as requiring quantization error to fall monotonically as bins are added\.

##### The summary statistic moves the answer\.

Equation[14](https://arxiv.org/html/2609.10210#S3.E14)averages activation across contexts, asking what a unit responds to most*strongly*\. Ranking tokens within each context and averaging those ranks asks what it responds to most*reliably*\. Two defensible summaries of the same measurements disagree, moving baselineov​@​8\\mathrm\{ov\}@8from 34\.6 to 52\.4 percent and changing the shape of the curve overKKas well as its level\. No external ground truth settles the choice: any measure of a unit’s downstream effect is itself monotone inaua\_\{u\}, which returns the question to how activations should be aggregated across contexts\.

##### Why two methods give one answer\.

In the face of that disagreement, the tempting inference is that a unit has two token sets and the two statistics find one each\. It does not follow\. Two methods producing two sets establishes only that the set is not unique; there could be three, or many, or none, with each statistic slicing an unstructured surface differently\. Running six summary statistics rather than two shows the actual structure on the baseline: four magnitude\-based measures agree with one another, and the single ordering\-based measure sits outside them\. That is a consensus and an outlier, invisible if the two statistics one happens to run are the two most different\.

### C\.4Which summary the causal set is taken under

The outlier is the one this paper’s per\-seed tables are computed with, and that has to be said\. Equation[14](https://arxiv.org/html/2609.10210#S3.E14)averages a unit’s activation over contexts and takes the topKK: the tokens that drive it hardest\. The alternative ranks the tokens within each context and averages those positions: the tokens that drive it most consistently\. The first is what the model integrates, since a unit’s write is linear in its activation and a token that spikes once does contribute a lot once\. The second is what a*description*ought to mean, since a name that holds in one context out of eight is not a name\.

The two disagree unequally\. Moving from the mean to the ranked summary costs the baseline17\.817\.8points of agreement atK=8K=8and the set\-operator model1\.71\.7\. Compared directly, with no decoding involved — the overlap between the two conventions’ own top\-88lists — they agree on6363percent of the baseline’s units and on7575to7979percent of the bounded models’, against a chance rate of1\.51\.5\.

That is a better result than either number alone\. For a bounded unit, what it responds to most strongly and what it responds to most reliably are close to the same list, so the question “what does this unit respond to” has an answer that does not depend on how it is asked\. For a conventional unit the two lists come apart\. The naming question is well posed for one construction and under\-determined for the other, which is a statement about the constructions rather than about the instrument, and it survives whichever summary a reader prefers\.

Two consequences follow\. The first is that a legibility*ratio*between two architectures is not a portable quantity\. The ratio here moves from about twice to about a third again depending on a convention neither architecture chose, and the movement is architecture\-specific, so comparing that ratio against a number computed elsewhere under an unstated convention is not a comparison\. That applies to this paper’s numbers as much as to anyone’s, which is why the ratio is not the form any claim here is put in\.

The second is narrower and sharper\. Ranks are invariant under any strictly increasing transform, so under the ranked summary a strictly monotone unit’s causal ordering*is*its pre\-activation ordering, exactly\. An order\-preserving arm’s agreement with its own ceiling is then11by construction rather than by measurement, and the set\-operator and sigmoid models duly return identical ceiling and realized figures\. The empirical content of that comparison lives entirely on the baseline’s side, which is why Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)reports it under the mean summary, where both sides can move: there the baseline reaches0\.810\.81of its ceiling and the sigmoid model0\.980\.98\.

### C\.5How the standard reading methods fail on a conventional unit

##### Max\-activating examples look fine\.

The default method in the field is to rank inputs by how strongly they drive a unit and read the top of that list\. Run on the baseline it gives every one of 36,864 units an apparently individual answer: across two converged seeds the strongest candidate is distributed over all 512 candidates, and the most popular single winner accounts for only 1\.1 and 2\.8 percent of units\. The problem is that nothing about the output looks degenerate\. The top of the list is the part the parameters can name — it is the sameK=8K=8regime where agreement is 59 percent above chance — and the list continues into Table[46](https://arxiv.org/html/2609.10210#A3.T46)\. A method that returns a distinct\-looking name for every unit while the set behind that name is a grab\-bag will not report its own failure, and reading only the top is how the failure goes unnoticed\.

Table 46:What six baseline units respond to: the top\-KKtokens of their causal profile atK=40K=40, for units at the median of the effective\-size distribution\. Unselected — these are the units the median picks out, not ones chosen to make a point\. Each row is one unit\. The definition of Section[1](https://arxiv.org/html/2609.10210#S1)asks that a reader be able to say what drives the component; this is the material that reading has to work with, and Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)is what sorting it by source recovers\. Context length 2048, 32 contexts, 512 candidate tokens\.Table 47:The same presentation as Table[46](https://arxiv.org/html/2609.10210#A3.T46), for the sigmoid model: top\-KKtoken sets atK=40K=40for units at the median of the effective\-size distribution, unselected\. Offered for comparison with the baseline’s rows above\. Section[5\.7](https://arxiv.org/html/2609.10210#S5.SS7)reads these through the dependency graph instead, which is the measurement the paper relies on\.
##### What concentration measures\.

The natural way to put a number on “is this unit nameable” is to ask how concentrated its response is on its best single name, scored against a null\. This ranks a collapsed model as the most nameable thing in the study\. A model whose code is nearly rank\-one scored 16\.9 times its null, and under a metric that also counts how many*distinct*names the units receive, the same model produced nine of them across 1536 units, scoring below the random null\. When most units share one dominant component they share their top inputs, so each scores a high concentration while being individually meaningless\. Any nameability score needs a distinctness measure read beside it\.

##### Rotating the model to fit the lens\.

If mid\-stack writes are illegible because their frame disagrees with the readout, the direct fix is to train the model to keep them in the same frame\. We tried this as an auxiliary objective pulling each layer’s lens distribution toward the final one, over a range of strengths\. Two things go wrong\. The angle saturates: past a certain point more pressure buys almost nothing, and the frames stop near fourteen degrees rather than at zero, because the early layers cannot both decode to the output and still compute\. And quality falls sharply well before that, by up to forty\-eight percent on LAMBADA at the strongest setting\. The only setting that costs nothing leaves the frame essentially where it started\. Imposing the alignment architecturally instead, with a learned orthogonal transform after every block trained to cancel each layer’s rotation, fails differently and more informatively: perplexity degrades by roughly an order of magnitude*and*the measured rotation increases\. The geometry resists being removed, consistent with its being load\-bearing rather than incidental\[[41](https://arxiv.org/html/2609.10210#bib.bib14)\]\.

##### Why the lens is a derivation\.

A sparse autoencoder trained on these activations does recover features a person can name, at good reconstruction and with few dead units\. We reproduce that, the strongest positive result any method in this appendix produces\. It is set aside here by choice rather than by definition\. Nothing about legibility forbids a working fitted explainer of this kind\. But an autoencoder is a second model, trained on activations harvested from a corpus, and it explains the network in the same sense that a second network trained to predict a network’s outputs explains it\[[9](https://arxiv.org/html/2609.10210#bib.bib6),[12](https://arxiv.org/html/2609.10210#bib.bib15),[7](https://arxiv.org/html/2609.10210#bib.bib20)\]: understanding the explainer becomes part of the job\. A sparse autoencoder is a real and useful thing to have, and the gap between it and reading the weights is what this paper is about\. One methodological note: anℓ1\\ell\_\{1\}penalty tuned by hand gave a solution with more than two hundred active units out of two hundred and fifty\-six, which is a dense rotation wearing a dictionary’s clothes\. Fixing the active count by construction, with a top\-kkactivation, avoids the blind coefficient search entirely\.

### C\.6The ablation searches fail their own checks

Two of the measurements in Section[2](https://arxiv.org/html/2609.10210#S2)are searches over ablations, and both of the obvious ways to run one are wrong\. Neither failure shows up in a median\.

##### Bisection lacks its invariant\.

The required graph is the smallest prefix of the one\-hop set whose removal changes the prediction, and the cheap way to find it is to bisect on prefix length:log⁡n90\\log n\_\{90\}ablations instead of a scan\. That is valid only ifflip⁡\(k\)\\mathrm\{flip\}\(k\)is monotone — if, once removing the topkkchanges the prediction, removing the topk\+1k\{\+\}1changes it too\. On1111to2828percent of predictions it is not\. Some larger prefix restores the prediction that a smaller one broke, and the same failure appears in all four architectures checked \(Table[48](https://arxiv.org/html/2609.10210#A3.T48)\)\.

The reason is the structure Appendix[D\.1](https://arxiv.org/html/2609.10210#A4.SS1)measures\. The dependency graph is not a tree\. Its nodes share sources heavily — that overlap is exactly why the closure grows so much more slowly than the branching factor would imply — so a prefix of the ranking is not a set of independent contributions that can be removed one at a time\. Removing the topkkdeletes components that the components ranked belowkkalso draw on, which changes what those components compute\. Thek\+1k\{\+\}1\-th ablation is therefore a different perturbation of a different graph, with the remaining nodes moved\. Nothing about that guarantees a monotone sequence, and the surviving prediction after a larger ablation is the downstream compensation familiar as self\-repair\[[35](https://arxiv.org/html/2609.10210#bib.bib27)\], here inside this instrument rather than in a model under study\.

Bisection, with nothing to stand on under those conditions, fails in a predictable direction\. It agrees with the true minimum on8484to100100percent of predictions and otherwise lands high, by as much as2525components on Llama\-3\.2\-1B\. The individual predictions behind a correct\-looking table of medians built by bisection are wrong\.

Table 48:Sweeping every prefix length instead of bisecting\. “Flip at all” counts predictions that change when some prefix is removed; the required graph is undefined for the rest, and they are excluded from the medians reported elsewhere\. “Worst overshoot” is the largest amount by which bisection exceeded the true minimum on a single prediction\.
##### Keeping more can preserve less\.

Appendix[D\.4](https://arxiv.org/html/2609.10210#A4.SS4)reports the sufficiency direction failing on the baseline\. It fails the same way on models this paper did not train, which is what makes it a property of the measurement\. Keeping only a pruned graph preserves the prediction between00and7777percent of the time, and on three of the four models below \(Table[49](https://arxiv.org/html/2609.10210#A3.T49), Figure[96](https://arxiv.org/html/2609.10210#A3.F96)\) the curve is not monotone in how much is kept: GPT\-2 peaks atθ=0\.3\\theta=0\.3and falls thereafter, and Llama\-3\.2\-1B reads1212,1212,00,3838\. A set that genuinely produced a prediction could not do worse when enlarged\.

The cause is that keeping a subset ablates everything else, so the model is evaluated far off its own distribution, and how far depends on the subset in a way that has nothing to do with whether the subset is the circuit\. The necessity direction in the same table is unaffected: it removes a small set from an otherwise intact model, which is a mild perturbation of a model still on its distribution\. That asymmetry is why a set cannot be certified sufficient by enlarging it until it works\. The instrument that does work runs the other way: Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)searches directly for a small set and tests it by running the model, scoring its controls against the floor the ablation itself creates\.

Table 49:Pruning by the per\-node thresholdθ\\theta, across four models\. Columns as in Table[56](https://arxiv.org/html/2609.10210#A4.T56)\. The same predictions are used at everyθ\\theta, so rows within a model are paired\.Figure 96:Both directions on the same pruned graphs\. Left: removing the graph changes the prediction almost as often at a tenth of the closure as at all of it\. Right: keeping*only*the graph, which is the sufficiency direction — the curves are not monotone, and on three of four models keeping more preserves the prediction*less*often\. That is a failure of the instrument rather than a property of the circuit\.

### C\.7Whether the instrument or the model is at fault

Every negative result in Section[3](https://arxiv.org/html/2609.10210#S3)admits two readings — the model has no structure of the kind sought, or the measurement could not have found it — and they are separated here rather than left to the reader, because three of the four turn out to be the instrument\.

The frame is the clearest case\. Scoring a writer through the unembedding and its reader in that reader’s own basis returns a correlation of−0\.02\-0\.02against a random control, which reads as a component passing nothing to the one that consumes it\. The same edges scored in one frame correlate\+0\.28\+0\.28, with nothing about the model changed between the two numbers\.

The fit is the second\. Reconstructing a unit’s drivers from upstream components returns negative held\-out variance under every pool tried, which invites the conclusion that no such relation exists\. Selecting three atoms from a dictionary of1,4401\{,\}440costs about twenty\-nine bits against thirty observations, so the search exhausts the sample before a coefficient is estimated; and the profile being reconstructed reproduces across disjoint contexts at0\.6480\.648, so the signal was there throughout\. An ordering, which selects nothing, recovers it \(Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)\)\.

Attention is the third\. Its components rank a unit’s drivers close to chance, and attention supplies2323percent of the components in a trace and is frequently among those a prediction cannot lose\. Driver sets are built by substituting a token at one position and reading a unit at the same position; attention assembles its output at a position from*other*positions\. A term that does not move when the probe moves cannot order the probe’s results, whatever it contributes to the prediction\.

The fourth is the model\. A component operating on a direction no token induces has no vocabulary name at any resolution\. The write column comes no closer to a token than a random direction through the early and middle stack\. A fitted rotation recovers nothing over not rotating\. And a shared basis fitted across units explains the*nameable*response better than the rest\. Where the reading stops there, it stops because the object is not lexical\.

### C\.8The opposing side of the accounting

Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)sets the opposing mass aside as traffic\. Whether that is right can be tested at the readout directly, by asking whether the mass pushing a token*down*is any different from what an arbitrary token receives\. The test uses the contrastive pairs of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5),A→XA\\rightarrow XandB→YB\\rightarrow Y, where what separates the two positions is what attention carried in from earlier context\. AtBB, whereXXlost, every feed\-forward unit and attention channel has a signed push onXX, and the negative side of that accounting is set against three floors: the same accounting for a competitor the model merely considered, a random token from its top fifty; for a random vocabulary token; and forXX’s own contributions with their signs shuffled and their magnitudes kept, which is what picking the most negative of tens of thousands of terms produces on its own\. Table[50](https://arxiv.org/html/2609.10210#A3.T50)reports the comparison\. The token that lost carries the same negative side as a token nobody was deciding about\. The ratio of its negative to its positive mass matches the competitor’s to within0\.020\.02in every model, and a random token’s is one; reaching ninety percent of its negative mass takes the same share of the model as it takes with the signs shuffled, to within a point; and removing its eight most negative components lifts it by about as much as the same operation lifts the competitor, within a logit in every model and by less than it lifts a random token on two of them\. Attention channels carry the negative mass in proportion to their number, and no single component switches on to pushXXdown where it loses: the most negative push atBBamong components that were not pushingXXdown atAA, scored against the spread of its kind, is within a unit of the same statistic for the control tokens in every model, and on the bounded arms a unit cannot switch at all, since a nonnegative activation fixes the sign of its push on a token\. “Not this” is written onto the readout by thousands of components in every context and for every token, and it is the mirror of the positive side rather than a signal about the token\.

Much of that mass is a constant per component rather than anything the position decides: split each component’s activation into its mean over positions and the deviation from it, and the mean carries about three quarters of the mass on both sides of the baseline’s accounting, effectively all of it on the two bounded arms, half on GPT\-2 and a quarter on Gemma\-3 and Qwen\. Rerunning the comparison with that constant part removed and with ablation to the mean rather than to zero changes none of its conclusions: every entry of Table[50](https://arxiv.org/html/2609.10210#A3.T50)moves by a few hundredths or a few points and the circuits of Table[8](https://arxiv.org/html/2609.10210#S2.T8)stay at tens of components\. The one case where the constant part matters is the seed of the last row of Table[51](https://arxiv.org/html/2609.10210#A3.T51): four attention channels writing large constants at every position dominate its signed sum, and with the constant part removed its net denominator returns from one or two components to9494, the net is negative on33percent of pairs instead of3939, and its negative side sits at the floor like every other model’s\.

Table 50:The negative side of a token that lost, against its floor\. On the contrastive pairs of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5), at the position whereXXlost, the mass pushingXXdown over the mass pushing it up, forXX, for a competitor from the model’s top fifty and for a random vocabulary token; the share of the model’s components it takes to reach ninety percent ofXX’s negative mass, against the same count with the signs ofXX’s contributions shuffled; and the logit lift from removing the eight components pushing each token down hardest\. Medians over pairs\. Feed\-forward units and attention channels alike\.
### C\.9Conditions for trusting these numbers

Every measurement in Section[3](https://arxiv.org/html/2609.10210#S3)can be made to return a confident wrong answer\. Table[51](https://arxiv.org/html/2609.10210#A3.T51)states the conditions that separate the two outcomes, each with the number that moves when it is dropped, because a condition offered without its consequence is easy to read as fastidiousness and skip\.

Table 51:Conditions for reading a component\. The last column gives the measurement with the condition and without it\. None of these is a matter of taste, and a reading that drops one is not a weaker version of this one — it is capable of returning the opposite sign\.
### C\.10Fitting a unit’s drivers from its upstream components

Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)ranks upstream components rather than fitting them, because every fitted version failed, all of them in the same way, and the failure mode is invisible in the statistic normally used to guard against it\.

A unit’s pre\-activation is a sum of upstream write columns carried forward and read through its row, so the natural estimator selects a few upstream components and regresses the unit’s response on their activations, scoring on tokens held out from the selection\. Four candidate pools were tried\. The co\-members of a unit’s own sufficient set; the union of every component appearing in any sufficient set; the components that share sufficient sets with the target, against a control matched on how often a component is reused; and the twelve upstream components whose carried write columns align most strongly with the unit’s read row\. Every candidate was carried through the fitted layer maps, the carried embedding was forced into the design so that what was scored was explanation beyond the starting residual, and attention channels were included alongside feed\-forward units\.

All four returned*negative*held\-out variance — worse than predicting the mean — and the strongest pool returned the worst result,−0\.306\-0\.306against−0\.009\-0\.009for the carried embedding alone\.

Observations per parameter, the usual guard, says the design was adequate: three atoms fitted on thirty tokens is ten observations each\. It is the wrong count\. The atoms are selected as well as fitted, and selecting three columns from a dictionary of1,4401\{,\}440costslog2⁡\(14403\)≈29\\log\_\{2\}\\binom\{1440\}\{3\}\\approx 29bits against thirty observations\. The search consumes the sample before a coefficient is estimated, so the columns that survive are the ones correlated with noise in the training half, and those anti\-correlate out of sample\. That is why the scores are negative rather than small, and it is why raising the sample and lowering the atom count — which moved observations per parameter from1\.81\.8to1010— changed nothing\.

The reading that the model simply has no such structure is ruled out separately\. Split the sixteen contexts in half and the two halves agree on the anonymous profile at0\.6480\.648\(CI\[0\.643,0\.654\]\[0\.643,0\.654\],n=14,123n=14\{,\}123\), so roughly two thirds of the variance being fitted is reproducible signal\. A ranking, which selects nothing and fits nothing, reaches the always\-predictable target that the estimator never could\.

### C\.11Reading a write column against the vocabulary

The naming of Section[3\.1](https://arxiv.org/html/2609.10210#S3.SS1)works from predictions and never reads a column\. That was not the first approach tried, and the measurements that failed are collected here, because their failure has a single cause\.

A component’s contribution to a prediction is a product of two terms, how hard it fired and how well its column aligns with what the readout needs\. Measured on the baseline, the column puts the tokens it is credited for at the8686th percentile of its own ordering over the vocabulary, and a third of them inside its own top one percent — so the column is far from silent\. But the firing carries5252percent of the variance in the contribution\. The column narrows the field to roughly the top seventh of the vocabulary and the context chooses from inside it, so no measurement of the column alone can recover which token a component is credited for\. Reversing the question confirms it: given components already labeled by what they write, retrieving a label from column geometry alone beats chance by only1\.231\.23times on the baseline and1\.351\.35on the sigmoid model, real but far too weak to invert\.

What was tried, and what each returned\. The best cosine between a column and any single centered unembedding row sits at chance through two thirds of every model \(Table[52](https://arxiv.org/html/2609.10210#A3.T52)\)\. Splitting a layer’s write into the part a fixed map of the incoming state accounts for and the part left over does not rescue it \(Table[53](https://arxiv.org/html/2609.10210#A3.T53)\)\. The set a column points at is no tighter than the set an arbitrary direction picks out, once the comparison is made against a random*direction*rather than a random set of tokens\. Morphological axes exist in the embedding — plural, capitalization, tense, all consistent at0\.290\.29to0\.550\.55when each pair is scored against an axis built from the others — but write columns lie along them at chance, and a component’s contribution to a plural prediction is separable from a singular one only in the aggregate and not per column\. The readout’s own geometry is no help either: one direction holds most of its variance and none of the residual stream’s energy, and removing it costs nothing while removing the two behind it costs eleven points of top\-one accuracy, so there is no free correction to apply\.

Table 52:Where a write column sits relative to the readout, by depth\. Each entry is the median column’s best cosine with any single centered unembedding row, in excess of the same statistic for a random direction, so00is chance and the chance level itself is given for scale\. Feed\-forward and attention are the same kind of object here — one column of an output projection, scaled by a scalar the model computes — and are measured identically\. Bold marks the last third, the only band in which any model clears chance\. As a check that the statistic can see readout\-directed writing at all, the state entering the final norm scores0\.2490\.249on it against these chance levels, and an unembedding row scores1\.0001\.000\.Table 53:Where the token alignment of a layer’s write sits, once the write is split into the part a fixed map of the incoming state accounts for and the part left over \(Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)\)\. Both are scored with the same best cosine against any centered unembedding row used in Table[52](https://arxiv.org/html/2609.10210#A3.T52)\. “First layers” is the largest value over the opening quarter of the stack and “rest” the median over the remainder\. The alignment is in the fitted part, which is what carrying the embedding forward looks like; the leftover is at chance throughout\. Layers where the leftover is under a quarter of the write are excluded from its range, since the direction of a vector that small carries no information — this drops the sigmoid model’s first four layers and none of the baseline’s\.This is why the naming had to come from predictions\.

### C\.12Naming a unit by where it fires

The move that names the write side does not transfer, and the failure is about the architecture rather than about the measurement\. One could collect the positions in ordinary text where a unit fires hardest and name it by the token sitting there, and two units in three do acquire a name that way\. But the token at a position is present in the residual only at the very bottom: it starts at\+0\.45\+0\.45and is down to\+0\.003\+0\.003one layer later\. Above that a unit is reading what the model has built rather than the token that seeded it, and any agreement between the two is running through whatever that token caused\. Measured by depth, the agreement behaves accordingly — it is three times stronger at layer zero than at layer eleven, and falls the whole way\. Naming by firing position therefore describes the contexts a unit turns on in, a real fact about the unit rather than what it reads\.

The same objection disposes of naming a unit by the token some fixed distance*before*its firing positions\. That token is not in the residual either, and the instrument cannot be completed by sweeping the distance: whatever a unit keys on may lie hundreds of positions back, so any offset is a truncation\. The parameters read against the layer’s own table are the instrument that answers what a unit reads, and they answer it sharply for the head of the list and not below\.

Table 54:Attention alignment split by head, at the two ends of the stack\. Each entry is the range across the model’s twelve heads of that head’s write columns weighted by how hard they fire, scored by the same best cosine with any centered unembedding row but reported as an absolute cosine rather than as an excess, with the chance level beside it for scale\. Layer zero is aligned in every model because the residual there is the token; the last layer separates the baseline from the rest\.Attention was measured the same way and reads the same, with one difference of its own: it leans on few of its columns, the top one percent carrying up to a fifth of a block’s firing where a feed\-forward block’s carry at most a fortieth\. A median over all columns therefore describes a typical column of a block that mostly uses a few, which is why a channel’s contribution sits further above its median column than a unit’s does\. Whether the columns a block uses are also the readable ones is a separate question and the answer is mostly no: weight each column by how hard it fires and compare against the same weights permuted across columns, and only the first attention block shows a large gap, where the residual is still the token itself\.

One cross\-model difference does survive the measure\. At the last layer the baseline’s heads straddle chance while every other model’s sit above it \(Table[54](https://arxiv.org/html/2609.10210#A3.T54)\) — read through the typical column, the baseline holds its computation off the readout axis until later than the rest\.

### C\.13What the tokens outside the concept are

Section[3\.2](https://arxiv.org/html/2609.10210#S3.SS2.SSS0.Px2)splits a unit’s driver set into the part forming an embedding\-space concept and a remainder the cosine cannot speak to, and reports the second as7272percent of the response on the baseline, and6161to7272percent across the four models measured here\. Three explanations of that remainder can be tested without consulting the embedding for the verdict, and none of the three survives\. What the tests leave is a description rather than a name, and the description is what Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)then works from: they establish that the remainder is real and single before it is read in another basis\. They run over1,4441\{,\}444units drawn at random across four models and sixteen contexts, and every number is quoted beside the same statistic on the same unit’s coherent tokens \(Table[55](https://arxiv.org/html/2609.10210#A3.T55)\)\.

The first possibility is that the remainder is nothing, its tokens there by sampling accident\. The test is*reliability*: build the response profile on one half of the contexts, build it on the disjoint other half, and ask whether the remainder’s ordering reproduces\. Noise does not reproduce\. It reproduces\. The rank correlation between halves runs from0\.520\.52to0\.640\.64across the four models, against0\.590\.59to0\.750\.75for the coherent tokens of the same units\. For a minority the answer is the other way,1111to2020percent of units falling below0\.20\.2, and for those the remainder is sampling noise\. For the rest it is response the model reproduces on contexts it has not seen\.

The second is that the remainder is several concepts at once, one unit carrying three paradigm classes with no single coherent direction among them\. The test is*substructure*: count the eigen\-directions of the remainder’s own similarity matrix that exceed the largest eigenvalue of a size\-matched random set\. It carries none\. Not one unit of the1,4441\{,\}444shows more than a single direction above the null, and1515to4040percent show none at all\.

The third is that the remainder is relational, and a syntagmatic class has a signature: a paradigm unit fires on its token wherever the token appears, while a relational unit fires only in the right structure\. The test is*context\-dependence*, the coefficient of variation of the response across contexts\. The signature is absent\. The remainder varies slightly*less*across contexts than the coherent part on every model, and the share of units running the other way is3131to5050percent\.

What the three leave is a remainder that is real, single, and no more context\-bound than the tokens a concept does explain\. That is a sharper object than a blind spot: it is not an artifact to be explained away, and it is not several things that better tools would separate\. It is one thing, read in a basis it is not written in — which is what Section[3\.4](https://arxiv.org/html/2609.10210#S3.SS4)measures by changing the basis to the components upstream of the unit\.

One candidate can be ruled out, and it is the obvious one\. Most of what a transformer does between layers is carriage \(Section[2\.6](https://arxiv.org/html/2609.10210#S2.SS6)\), so the natural guess is that a unit’s unnameable response is that unit taking part in carriage rather than responding to anything\. Fitting the layer\-to\-layer map gives the subspace carriage works in, and each write column can be scored by how much of itself lies there\. Across326326baseline units and278278sigmoid units measured both ways, that score correlates with the nameable share at\+0\.14\+0\.14and\+0\.22\+0\.22: the wrong sign, and the wrong sign after controlling for depth and for membership in the residual’s own high\-variance subspace, with which the carriage span shares no more than a random overlap\. Units that write more into carriage are slightly*more*nameable\. There is also a reason the guess was unlikely to hold as stated\. A unit writes one direction whatever drives it, so how much of it is carriage is a property of its write, while how much of it is nameable is a property of its read\. The two need not have anything to say to each other\.

The read side admits the same test and fails it too\. A component that carries the stream has to*read*what is being carried, so its read row should lie in the carriage span; unlike the write column, the read row varies with what drives the unit, so this comparison is not ruled out by construction\. Read rows sit in that span at1\.261\.26and1\.341\.34times a matched random direction, about what write columns manage, and the lift predicts nothing about nameability:r=\+0\.007r=\+0\.007on the baseline and\+0\.177\+0\.177on the sigmoid model, the second again the wrong sign\. Reading the carried stream is not what the unnameable response is\.

##### The remainder against an untrained weight\.

A read row that never received a useful gradient would still look like its initialization, and a random direction has no token that names it, which would make a unit unnameable by default\. Comparing the checkpoint at50,00050\{,\}000steps against the end of training settles it without recourse to an initialization seed: not one read row of36,86436\{,\}864in either model is static\. The median cosine between the two checkpoints is0\.580\.58and0\.640\.64, each row traveling about four fifths of its own norm over that stretch, and no unit moves less than a tenth of it\. Nameability is flat across the range —r=\+0\.011r=\+0\.011on the baseline, and the least\-moved and most\-moved halves of the population are both at2828percent nameable to the point\.

Table 55:What the tokens outside the concept are\. “Coherent” is the part of a unit’s top forty that clears a cut calibrated against a random set of the same size, and the share is of the unit’s response\. Reliability is the rank correlation of a token ordering built on one half of the contexts against the disjoint other half\. Substructure counts eigen\-directions of the remainder above a size\-matched null\. Context variation is the coefficient of variation of the response across contexts\. Units are sampled uniformly at random per layer, and a unit is examined only where both sides have at least four tokens\.

## Appendix DThe dependency graph

Section[2](https://arxiv.org/html/2609.10210#S2)identifies the components that carry a prediction and the smallest set that builds it, and neither measurement needs to know where a component’s own input came from\. That question has an answer, and this appendix gives it: the edge relation between two components, the recursion that closes it into a graph, and what the resulting object is good for\.

Two results here are used elsewhere in the paper\. The edges are causal rather than merely attributed, which is what licenses Section[3](https://arxiv.org/html/2609.10210#S3)to describe a component by what its inputs carry\. And a graph’s second level cannot be estimated by multiplying its first level by a branching factor — the product overstates the measured count, by a factor that grows with the branching itself — which anyone measuring these graphs is likely to need\.

The rest is a route that does not arrive\. The closure, built to answer the sufficiency question, answers it worse than a set of the same size taken straight off the contribution ranking\. Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)reports what does work\.

### D\.1Edges, branching, and the closure

Ifn90n\_\{90\}is one hop, the natural next question is how quickly the structure widens below it\. For a nodexxin the one\-hop set, the share of its incoming drive owed to an upstream componentvvis

ev→x=av​⟨wv,γℓ⊙rx⟩/ρℓ,e\_\{v\\to x\}\\;=\\;a\_\{v\}\\,\\big\\langle w\_\{v\},\\;\\gamma\_\{\\ell\}\\odot r\_\{x\}\\big\\rangle\\big/\\rho\_\{\\ell\},\(49\)withrxr\_\{x\}the read row ofxxandγℓ\\gamma\_\{\\ell\}the gain of the norm feeding its sublayer\. The scaleρℓ\\rho\_\{\\ell\}is a single number for a givenxx, so it cancels from every share and the recursion needs no normalization of its own\.

The ninety percent here is a different quantity from the one of Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)\. That bar was on contribution to the readout, and there is one such total per prediction\. This one is on contribution to a single node’s pre\-activation, and there is a different total for every node in the graph\. What the same fraction is a fraction*of*differs\. With that fixed, define the*branching factor*as the number of upstream components needed to reach ninety percent of a node’s own incoming drive\.

Equation[6](https://arxiv.org/html/2609.10210#S2.E6)is already exact, so it is fair to ask what a second level adds\. The contributions sum to the logit, so the one\-hop set is not an approximation of the answer that a second level improves — it is a complete account of*the logit, given the activations*\. Every component in it writes onto the readout direction; that is what selects it; and no member depends on any other\. The one\-hop set is a star around the readout rather than a graph\.

What the accounting takes as given isaua\_\{u\}\. The write projection in Equation[6](https://arxiv.org/html/2609.10210#S2.E6)is a parameter and is fixed, but the activation is an observed number, and nothing so far says where it came from\. Equation[49](https://arxiv.org/html/2609.10210#A4.E49)is the first statement in this paper that relates two components to each other rather than each to the readout, and what it decomposes is not the logit but the pre\-activation behindaua\_\{u\}\. That is also the influence the exact decomposition structurally omits: a component in an early layer is credited for its own write landing on the readout, never for the fact that later layers read it and act on what they find\.

Nothing stops that step from repeating\. Every source it returns is itself a component with a read row, so the same question can be put to it, and to whatever it returns, until no node remains that has not been asked\. Call the fixed point the*closure*: the one\-hop set together with everything reachable from it by following ninety percent of each node’s own incoming drive\. It is the backward dependency graph of a single prediction, and every member of it is counted rather than estimated\.

Two properties keep the recursion from running away\. A component can draw only on components strictly below it, so the graph is acyclic and ordered by depth\. And the sets overlap heavily: different nodes draw on many of the same sources, so each level adds far less than the previous level’s size times the branching factor\. On the baseline the counts run5353,336336,736736over six levels, after which the frontier is empty —1\.61\.6percent of the model’s components\. Every model in this paper closes this way, at four to ten levels, and the closure runs three to six times the one\-hop set on the shaped arms, fourteen times on the baseline, and twelve to sixty on the models trained by other people\.

That overlap is why the product ofn90n\_\{90\}and the branching factor is never reported here as a stand\-in for the second level\. The product counts every component shared by two nodes once per node; across the eighteen models here it overstates the measured two\-level closure by1\.61\.6to9\.8×9\.8\\times, median3\.6×3\.6\\times, by a factor that grows with the branching factor itself and so distorts comparisons between models as well as the size of any one\.

A graph built this way can also be thinned, by lowering the share of a node’s incoming drive its sources have to account for\. Writing that shareθ\\theta, the closure above isθ=0\.9\\theta=0\.9, and smaller values keep only the strongest edges out of each node\. Both measurements that follow are read at several values ofθ\\theta, and they do not agree about what happens as it falls\.

Only feed\-forward nodes are expanded\. An attention component here is a named location — a layer, a head, and a channel within that head\. It is scored by Equation[49](https://arxiv.org/html/2609.10210#A4.E49)like any other component, it is ranked against them, it enters the one\-hop set and the closure on the same footing, and when a graph is ablated it is ablated by name\. Nothing about its contribution is unknown\.

What is not done is to ask what produced the value that channel read\. That question is about a different graph\. A feed\-forward unit at positionppdraws on the residual stream atpp, so its sources sit at the same position and lower layers, and the recursion stays inside a depth\-ordered acyclic graph\. An attention channel atppdraws on values written at earlier positions, so following it leaves the token whose prediction is being explained and opens a graph over the sequence\. This paper measures the graph at one position and treats attention as its boundary: between a third and three fifths of a closure is attention, identified and ablated but not followed\. What the graph is worth is an activation\-function question, taken up in Sections[5\.6](https://arxiv.org/html/2609.10210#S5.SS6)and[6\.2](https://arxiv.org/html/2609.10210#S6.SS2); here the point is only that these edges are real\.

These edges are causal rather than merely attributed\. Removing a node’s eight highest\-ranked sources leaves it at0\.340\.34of its original activation; removing eight components drawn at random from the same upstream pool leaves it at1\.00041\.0004\. The separation holds at every count tested, and the random control never moved a node by more than a few percent\.

Figure 97:One prediction’s edges, read one hop\. Bold marks the components whose removal changes the prediction; indented rows are what such a component draws on, with the share of its incoming drive they supply and, where writer and reader both rank a token in their first eight, that token\. The graph beneath draws the same components, depth running left to right, each joined to the answer with the rank the predicted token holds in what it promotes\.Figure[97](https://arxiv.org/html/2609.10210#A4.F97)is one prediction read this way, drawn from the same random sample as the gallery and under the same filters\. The context ends*a second Brexit referendum is looking more and more*and the model predicts*likely*atp=0\.58p=0\.58;5656components reach ninety percent of the net,9494percent of the contribution mass cancels, and two components are required\.

The four required and contributing components in it do not all relate to the answer the same way\. Unit19971997of layer88fires on*more*,*very*and*most*, and the three units feeding it hardest all sit in layer00and write exactly those degree words: the three edges are labeled with degree words,*more*,*less*,*most*and*very*, each a token that writer and this reader both rank in their first eight\. It promotes the predicted token tenth\. Unit772772of layer99answers to*no*and*serious*and writes modal verbs, promoting*likely*fourteenth\. A comparative detector fed by degree words, and a modal detector, are a reasonable account of why a sentence ending*more and more*continues with*likely*\.

Unit19681968of layer77is the other case\. It fires on past\-tense verbs and auxiliaries —*went*,*been*,*did*,*could*— and writes word\-continuation fragments —*es*,*ing*,*er*\. Both sides are legible\. What it does not do is promote the answer: it ranks*likely*past three thousand, so the predicted token is nowhere near what this unit specializes in\. It is required anyway\. Its contribution to that particular logit is large enough that removing it changes the prediction, which is a reminder that promoting a token and contributing to it are different things, and that the components a prediction cannot lose need not be the ones pushing toward it\.

### D\.2Aggregating influence along paths

The overlap half of that problem has a standard treatment\.[Abnar and Zuidema \[1\]](https://arxiv.org/html/2609.10210#bib.bib38)face it directly when aggregating attention across layers — paths through a transformer share edges, so summing path weights double\-counts — and answer it by treating the attention graph as a flow network with attention weights as capacities and taking a maximum flow\. That construction does not transfer here unaltered, for two reasons\. Capacities must be non\-negative, and attention weights are; the contributions of Equation[6](https://arxiv.org/html/2609.10210#S2.E6)are signed, and replacing them with magnitudes or with their positive parts is exactly the accounting this appendix argues against\. And a maximum flow scores a path by its bottleneck edge rather than by what it delivers, which answers how much could be routed rather than how much arrives\.

For a signed and locally linear quantity the overlap is in fact benign: contributions along different paths add, so a path\-sum over the graph is well defined and shared edges need no special handling\. What is*not*linear is the intervention — ablation removes a component and the survivors then compute something else — and the intervention rather than the attribution is what makesflip⁡\(k\)\\mathrm\{flip\}\(k\)non\-monotone\. A ranking by total path\-summed influence rather than by direct contribution is the natural next version of this measurement, and choosing a set by cut rather than by prefix is the version after that\. The prefix is reported because it is defined without further assumptions and can be checked directly by ablation, and it is stated as a bound rather than as a minimum\.

### D\.3A pruned graph is still the reason

Thinning the closure barely weakens it\. A tenth of it —5656components, about one part in a thousand of the model — still changes the prediction9595percent of the time, and a fifth of it changes every prediction tested \(Table[56](https://arxiv.org/html/2609.10210#A4.T56)\)\. Necessity survives pruning that removes most of the graph\.

Table 56:Pruning the graph by lowering the per\-node thresholdθ\\theta\. “Prediction changes” ablates the pruned graph; “preserved” keeps only it and replaces everything else\. The same predictions are used at everyθ\\theta, so the rows are paired\.
### D\.4Why sufficiency needs a search rather than a closure

The sufficiency test keeps a set, replaces every other component in the model, and asks whether the same token still comes out\. Put to the closure it is a fair question\. Put to the one\-hop set it is not, because replacing everything outside that set replaces the very drive that makes its members fire, so the graph is what the test needs\.

Two controls decide whether the recursion earns its place\. A size\-matched random set says whether the closure beats an arbitrary set of the same size, and the top\-\|C\|\|C\|components by direct contribution say whether it beats simply taking more of the one\-hop ranking\.

Table 57:Sufficiency against the one\-hop set and against its closure, with both controls at matched size\. “Keep\-more” is the validity check described below: how much preservation rises as the kept set grows from nothing to2,0482\{,\}048components\. Where that curve is flat the test discriminates nothing and no entry should be read as a measurement; Section[6\.5](https://arxiv.org/html/2609.10210#S6.SS5)reports a model where it is flat\.This measurement needs more predictions than it looks like it needs\. At2828of them \(Table[57](https://arxiv.org/html/2609.10210#A4.T57)\) the closure preserves the prediction on5454percent against3232for a size\-matched set taken further down the contribution ranking, which reads as twenty\-two points in the recursion’s favor\. At150150the sign reverses:6060percent against6565on the baseline,6363against7878on the sigmoid model, and6565against8787on GPT\-2\. A9595percent interval on a proportion near a half is±19\\pm 19points atn=28n=28and±8\\pm 8atn=150n=150, so twenty\-two points at the smaller sample sits inside its own noise\. What it buys on any model depends on that model’s branching factor: where the factor is small the closure is barely larger than the one\-hop set and there is little for the recursion to add, which is the case for the constructions of Section[5](https://arxiv.org/html/2609.10210#S5)\.

A keep\-only number means nothing unless keeping more helps, and that has to be checked rather than assumed\. Raising the kept set from nothing to2,0482\{,\}048components moves preservation by sixty\-three points, so the test has room to discriminate and the closure’s advantage over the controls is a real difference rather than noise around a ceiling\. Every model measured behaves this way but one \(Section[6\.5](https://arxiv.org/html/2609.10210#S6.SS5)\)\. Tightening the recursion does not repair it\. Raising the per\-node threshold from0\.90\.9to0\.990\.99and on to the point where the cut no longer binds leaves preservation flat or slightly lower on every model tested, while the set grows:5959to5656percent on the baseline as the closure goes from640640to786786components\. Ancestry is the wrong currency for this test: what preserves a prediction is components that write toward it, and the closure spends its budget on ancestry\.

Keeping*only*a pruned graph is not even monotone in how much is kept — preservation peaks atθ=0\.7\\theta=0\.7and falls atθ=0\.9\\theta=0\.9, where strictly more of the graph is retained \(Table[56](https://arxiv.org/html/2609.10210#A4.T56)\) — and Appendix[C\.6](https://arxiv.org/html/2609.10210#A3.SS6)shows that to be a property of the instrument rather than of this model\.

Both numbers in this appendix are necessity statements, and neither shows that the components in it can produce the answer on their own\. The search of Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)reaches eight components, two orders of magnitude below the closure, and a different set\.

### D\.5What the replacement choice does to a keep\-only number

Removing a component is not free of choices\. A transformer has no null value for a component’s output, so an ablation must write something in its place, and three are in common use: zero, the component’s mean output sampled over inputs, and a value resampled from a different input\. Which of these is chosen barely moves the measurement above; the three agree to within seven points\. The lens identifies the*necessary*parts of the transformer for that prediction, and replacing them quickly breaks the prediction\.

Run in reverse — keep only the components the lens found and replace everything else — the measurement changes character \(Table[58](https://arxiv.org/html/2609.10210#A4.T58)\)\. All three choices drastically impact the correctness: the prediction survives in3333percent of cases under zero replacement and33percent under resampling\. The components the lens identifies are thus necessary but*are not sufficient*on their own to make the prediction\. Section[2\.5](https://arxiv.org/html/2609.10210#S2.SS5)reports what does reproduce a prediction; the question here is the choice of zero against the mean and resample alternatives\. Zero replacement, the most flattering of the three, flatters for a reason that has nothing to do with the circuit\. Section[2\.3](https://arxiv.org/html/2609.10210#S2.SS3)showed that most of what a model writes at a position pushes*away*from the predicted token\. Zeroing every component outside the circuit deletes that opposing mass along with everything else, so the target logit can come out*higher*than the intact model produces\.

Table 58:Sufficiency under the three standard replacement baselines: the share of predictions preserved when only the circuit is kept and everything else is replaced\. The choice of replacement moves the answer by thirty points on the same circuit\. The necessity measurement, on the same components and the same three choices, moves by seven\. Zero, the most favorable replacement, is the least informative, for the reason given in the text\.

## References

- \[1\]\(2020\)Quantifying attention flow in transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4190–4197\.Cited by:[§D\.2](https://arxiv.org/html/2609.10210#A4.SS2.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px3.p1.1)\.
- \[2\]E\. Ameisen, J\. Lindsey, A\. Pearce, W\. Gurnee, N\. L\. Turner, B\. Chen, C\. Citro, D\. Abrahams, S\. Carter, B\. Hosmer, J\. Marcus, M\. Sklar, A\. Templeton, T\. Bricken, C\. McDougall, H\. Cunningham, T\. Henighan, A\. Jermyn, A\. Jones, A\. Persic, Z\. Qi, T\. B\. Thompson, S\. Zimmerman, K\. Rivoire, T\. Conerly, C\. Olah, and J\. Batson\(2025\)Circuit tracing: revealing computational graphs in language models\.Transformer Circuits Thread\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px5.p1.1)\.
- \[3\]A\. Arora, Z\. Wu, J\. Steinhardt, and S\. Schwettmann\(2026\)Language model circuits are sparse in the neuron basis\.InInternational Conference on Machine Learning \(ICML\),Note:Spotlight\. arXiv:2601\.22594Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px5.p1.1)\.
- \[4\]A\. Avrahamy, Y\. Gur\-Arieh, and M\. Geva\(2026\)Disentangling MLP neuron weights in vocabulary space\.InConference on Language Modeling \(COLM\),Note:arXiv:2604\.06005Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p2.1)\.
- \[5\]D\. Bau, S\. Liu, T\. Wang, J\. Zhu, and A\. Torralba\(2020\)Rewriting a deep generative model\.InEuropean Conference on Computer Vision \(ECCV\),Note:arXiv:2007\.15646Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p1.1)\.
- \[6\]N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. Steinhardt\(2023\)Eliciting latent predictions from transformers with the tuned lens\.arXiv preprint arXiv:2303\.08112\.Cited by:[§2\.2](https://arxiv.org/html/2609.10210#S2.SS2.SSS0.Px1.p3.1),[§2\.2](https://arxiv.org/html/2609.10210#S2.SS2.p3.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1)\.
- \[7\]S\. Bills, N\. Cammarata, D\. Mossing, H\. Tillman, L\. Gao, G\. Goh, I\. Sutskever, J\. Leike, J\. Wu, and W\. Saunders\(2023\)Language models can explain neurons in language models\.Note:OpenAICited by:[§C\.5](https://arxiv.org/html/2609.10210#A3.SS5.SSS0.Px4.p1.1)\.
- \[8\]R\. Bommasani, K\. Davis, and C\. Cardie\(2020\)Interpreting pretrained contextualized representations via reductions to static embeddings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4758–4781\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1)\.
- \[9\]T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn,et al\.\(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.html](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§C\.5](https://arxiv.org/html/2609.10210#A3.SS5.SSS0.Px4.p1.1),[§8\.3](https://arxiv.org/html/2609.10210#S8.SS3.p5.1)\.
- \[10\]P\. Chen, A\. Plaat, and N\. van Stein\(2026\)Every component is a lookup: token attribution and composition from a single decomposition\.arXiv preprint arXiv:2605\.23393\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p2.1)\.
- \[11\]A\. Conmy, A\. N\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-Alonso\(2023\)Towards automated circuit discovery for mechanistic interpretability\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[12\]H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey\(2023\)Sparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[§C\.5](https://arxiv.org/html/2609.10210#A3.SS5.SSS0.Px4.p1.1),[§8\.3](https://arxiv.org/html/2609.10210#S8.SS3.p5.1)\.
- \[13\]D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei\(2022\)Knowledge neurons in pretrained transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2104\.08696Cited by:[§4\.2](https://arxiv.org/html/2609.10210#S4.SS2.p2.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p3.1)\.
- \[14\]G\. Dar, M\. Geva, A\. Gupta, and J\. Berant\(2023\)Analyzing transformers in embedding space\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2209\.02535Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1)\.
- \[15\]Y\. Dong, J\. Cordonnier, and A\. Loukas\(2021\)Attention is not all you need: pure attention loses rank doubly exponentially with depth\.InInternational Conference on Machine Learning,Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p2.1)\.
- \[16\]J\. Dunefsky, P\. Chlenski, and N\. Nanda\(2024\)Transcoders find interpretable LLM feature circuits\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.11944Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p2.1)\.
- \[17\]N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan,et al\.\(2022\)Toy models of superposition\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2022/toy\_model/index\.html](https://transformer-circuits.pub/2022/toy_model/index.html)Cited by:[§1](https://arxiv.org/html/2609.10210#S1.p3.1),[§2\.6](https://arxiv.org/html/2609.10210#S2.SS6.p3.1),[§8\.3](https://arxiv.org/html/2609.10210#S8.SS3.p3.1)\.
- \[18\]N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2021/framework/index\.html](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p1.1)\.
- \[19\]J\. Ferrando and E\. Voita\(2024\)Information flow routes: automatically interpreting language models at scale\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px3.p2.1)\.
- \[20\]L\. Gao, A\. Rajaram, J\. Coxon, S\. V\. Govande, B\. Baker, and D\. Mossing\(2025\)Weight\-sparse transformers have interpretable circuits\.arXiv preprint arXiv:2511\.13653\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px9.p1.1)\.
- \[21\]M\. Geva, A\. Caciularu, K\. R\. Wang, and Y\. Goldberg\(2022\)Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1)\.
- \[22\]M\. Geva, R\. Schuster, J\. Berant, and O\. Levy\(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p1.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p2.1)\.
- \[23\]A\. D\. Hakimi, K\. Lei, I\. Augenstein, and H\. Schütze\(2026\)Output vector editing for memorization mitigation in large language models\.External Links:2606\.18767Cited by:[§4\.2](https://arxiv.org/html/2609.10210#S4.SS2.p2.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p3.1)\.
- \[24\]P\. Hase, M\. Bansal, B\. Kim, and A\. Ghandeharioun\(2023\)Does localization inform editing? surprising differences in causality\-based localization vs\. knowledge editing in language models\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),External Links:2301\.04213Cited by:[§4\.2](https://arxiv.org/html/2609.10210#S4.SS2.p3.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p4.1)\.
- \[25\]Z\. Huang, Y\. Shen, X\. Zhang, J\. Zhou, W\. Rong, and Z\. Xiong\(2023\)Transformer\-patcher: one mistake worth one neuron\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2301\.09785Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p1.1)\.
- \[26\]J\. Janiak, C\. Rager, J\. Dao, and Y\. Lau\(2024\)An adversarial example for direct logit attribution: memory management in GELU\-4L\.InProceedings of the 7th BlackboxNLP Workshop,pp\. 232–237\.Note:arXiv:2310\.07325Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px2.p1.1)\.
- \[27\]J\. Kramár, T\. Lieberum, R\. Shah, and N\. Nanda\(2024\)AtP\*: an efficient and scalable method for localizing LLM behaviour to components\.arXiv preprint arXiv:2403\.00745\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[28\]K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg\(2023\)Inference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2306\.03341Cited by:[§4\.4](https://arxiv.org/html/2609.10210#S4.SS4.p2.1)\.
- \[29\]M\. Li and L\. Janson\(2024\)Optimal ablation for interpretability\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2409\.09951Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px6.p1.1)\.
- \[30\]D\. Lindner, J\. Kramár, S\. Farquhar, M\. Rahtz, T\. McGrath, and V\. Mikulik\(2023\)Tracr: compiled transformers as a laboratory for interpretability\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2301\.05062Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p5.1)\.
- \[31\]A\. Liu, J\. Hayase, V\. Hofmann, S\. Oh, N\. A\. Smith, and Y\. Choi\(2025\)SuperBPE: space travel for language models\.InConference on Language Modeling \(COLM\),External Links:2503\.13423Cited by:[§8\.2](https://arxiv.org/html/2609.10210#S8.SS2.p3.1)\.
- \[32\]J\. Liu, Z\. Wang, K\. Zhao, and D\. Hu\(2025\)Balancing knowledge updates: toward unified modular editing in LLMs\.External Links:2510\.27400Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p4.1)\.
- \[33\]S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. Mueller\(2025\)Sparse feature circuits: discovering and editing interpretable causal graphs in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2403\.19647Cited by:[§1](https://arxiv.org/html/2609.10210#S1.p2.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[34\]C\. McDougall, A\. Conmy, C\. Rushing, T\. McGrath, and N\. Nanda\(2023\)Copy suppression: comprehensively understanding an attention head\.arXiv preprint arXiv:2310\.04625\.Note:verified 2026\-09\-09Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px2.p1.1)\.
- \[35\]T\. McGrath, M\. Rahtz, J\. Kramar, V\. Mikulik, and S\. Legg\(2023\)The hydra effect: emergent self\-repair in language model computations\.arXiv preprint arXiv:2307\.15771\.Cited by:[§C\.6](https://arxiv.org/html/2609.10210#A3.SS6.SSS0.Px1.p2.1),[§2\.4](https://arxiv.org/html/2609.10210#S2.SS4.p3.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px2.p1.1)\.
- \[36\]K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov\(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2202\.05262Cited by:[§4\.2](https://arxiv.org/html/2609.10210#S4.SS2.p1.1),[§4\.9](https://arxiv.org/html/2609.10210#S4.SS9.p1.1),[§5\.9](https://arxiv.org/html/2609.10210#S5.SS9.p1.2),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p1.1)\.
- \[37\]C\. Neo, S\. B\. Cohen, and F\. Barez\(2024\)Interpreting context look\-ups in transformers: investigating attention\-mlp interactions\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2402\.15055Cited by:[§4\.6](https://arxiv.org/html/2609.10210#S4.SS6.p4.1)\.
- \[38\]nostalgebraist\(2020\)Interpreting GPT: the logit lens\.Note:LessWrong[https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)Cited by:[§2\.2](https://arxiv.org/html/2609.10210#S2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2609.10210#S3.SS2.p2.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px1.p1.1)\.
- \[39\]M\. Oskin\(2026\)Explicit fuzzy logic in the feed\-forward layer: self\-forgetting quantifiers discover legible grammatical\-licensing detectors\.arXiv preprint arXiv:2606\.31845\.Cited by:[Appendix B](https://arxiv.org/html/2609.10210#A2.p1.1),[§5\.2](https://arxiv.org/html/2609.10210#S5.SS2.p4.1)\.
- \[40\]M\. Oskin\(2026\)Legible\-by\-construction: attention and end\-to\-end transformers\.Note:arXiv preprint arXiv:2607\.04319Cited by:[Appendix B](https://arxiv.org/html/2609.10210#A2.p1.1),[§5\.2](https://arxiv.org/html/2609.10210#S5.SS2.p4.1)\.
- \[41\]M\. Oskin\(2026\)Off\-axis, on purpose: where a transformer computes concepts and why it does so\.Note:arXiv preprint arXiv:2608\.10251Cited by:[§B\.3](https://arxiv.org/html/2609.10210#A2.SS3.p1.1),[§B\.3](https://arxiv.org/html/2609.10210#A2.SS3.p2.4),[§B\.3](https://arxiv.org/html/2609.10210#A2.SS3.p3.1),[Appendix B](https://arxiv.org/html/2609.10210#A2.p1.1),[§C\.5](https://arxiv.org/html/2609.10210#A3.SS5.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.10210#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.10210#S2.SS2.SSS0.Px1.p1.1),[§2\.6](https://arxiv.org/html/2609.10210#S2.SS6.p3.1),[§2\.6](https://arxiv.org/html/2609.10210#S2.SS6.p5.1),[§3\.4](https://arxiv.org/html/2609.10210#S3.SS4.SSS0.Px1.p8.1),[§4\.4](https://arxiv.org/html/2609.10210#S4.SS4.p6.1),[§5\.2](https://arxiv.org/html/2609.10210#S5.SS2.p5.1)\.
- \[42\]M\. Oskin\(2026\)Training, reading, and editing legible transformers\.Note:arXiv preprint arXiv:2607\.08946Cited by:[Appendix B](https://arxiv.org/html/2609.10210#A2.p1.1),[§5\.2](https://arxiv.org/html/2609.10210#S5.SS2.p4.1),[§5\.4](https://arxiv.org/html/2609.10210#S5.SS4.p3.1)\.
- \[43\]N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner\(2024\)Steering llama 2 via contrastive activation addition\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2312\.06681Cited by:[§4\.4](https://arxiv.org/html/2609.10210#S4.SS4.p2.1)\.
- \[44\]D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández\(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2\.1\.1](https://arxiv.org/html/2609.10210#S2.SS1.SSS1.p3.1)\.
- \[45\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.OpenAI technical report\.Cited by:[§2\.1](https://arxiv.org/html/2609.10210#S2.SS1.p1.1)\.
- \[46\]C\. Rushing and N\. Nanda\(2024\)Explorations of self\-repair in language models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:PMLR 235:42836–42855\. arXiv:2402\.15390Cited by:[§2\.4](https://arxiv.org/html/2609.10210#S2.SS4.p3.1),[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px2.p1.1)\.
- \[47\]S\. Salek and J\. Enguehard\(2025\)Using the path of least resistance to explain deep networks\.arXiv preprint arXiv:2502\.12108\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[48\]C\. W\. Schmidt, V\. Reddy, C\. Tanner, and Y\. Pinter\(2025\)Boundless byte pair encoding: breaking the pre\-tokenization barrier\.InConference on Language Modeling \(COLM\),External Links:2504\.00178Cited by:[§8\.2](https://arxiv.org/html/2609.10210#S8.SS2.p3.1)\.
- \[49\]S\. Sukhbaatar, E\. Grave, G\. Lample, H\. Jégou, and A\. Joulin\(2019\)Augmenting self\-attention with persistent memory\.arXiv preprint arXiv:1907\.01470\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px7.p1.1)\.
- \[50\]M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu\(2024\)Massive activations in large language models\.InFirst Conference on Language Modeling \(COLM\),Note:arXiv:2402\.17762Cited by:[§4\.6](https://arxiv.org/html/2609.10210#S4.SS6.p2.1),[§4\.6](https://arxiv.org/html/2609.10210#S4.SS6.p5.1)\.
- \[51\]M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu\(2024\)Massive activations in large language models\.arXiv preprint arXiv:2402\.17762\.Note:verified 2026\-09\-09Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px6.p1.1)\.
- \[52\]A\. Syed, C\. Rager, and A\. Conmy\(2023\)Attribution patching outperforms automated circuit discovery\.InNeurIPS 2023 Workshop on Attributing Model Behavior at Scale \(ATTRIB\),Note:arXiv:2310\.10348Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[53\]A\. Tamkin, M\. Taufeeque, and N\. D\. Goodman\(2023\)Codebook features: sparse and discrete interpretability for neural networks\.arXiv preprint arXiv:2310\.17230\.Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px9.p1.1)\.
- \[54\]W\. Timkey and M\. van Schijndel\(2021\)All bark and no bite: rogue dimensions in transformer language models obscure representational quality\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2109\.04404Cited by:[footnote 3](https://arxiv.org/html/2609.10210#footnote3)\.
- \[55\]A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid\(2023\)Steering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§4\.4](https://arxiv.org/html/2609.10210#S4.SS4.p2.1)\.
- \[56\]T\. Vergara\-Browne and Á\. Soto\(2025\)Tracr\-injection: distilling algorithms into pre\-trained language models\.InFindings of the Association for Computational Linguistics: ACL,Note:arXiv:2505\.10719Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p5.1)\.
- \[57\]K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt\(2023\)Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2211\.00593Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px4.p1.1)\.
- \[58\]Y\. Wei, X\. Yu, Y\. Weng, H\. Ma, Y\. Zhang, J\. Zhao, and K\. Liu\(2024\)Does knowledge localization hold true? surprising differences between entity and relation perspectives in language models\.InACM International Conference on Information and Knowledge Management \(CIKM\),Note:arXiv:2409\.00617Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p4.1)\.
- \[59\]G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis\(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2309\.17453; verified 2026\-09\-09Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px6.p1.1)\.
- \[60\]W\. Yang, F\. Sun, J\. Tan, X\. Ma, Q\. Cao, D\. Yin, H\. Shen, and X\. Cheng\(2025\)The mirage of model editing: revisiting evaluation in the wild\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2502\.11177Cited by:[§7](https://arxiv.org/html/2609.10210#S7.SS0.SSS0.Px8.p4.1)\.