Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation

# Attention Mean Fields Predict Average Representation Dynamics and Reveal Context-Specific Computation
Source: [https://arxiv.org/html/2609.16382](https://arxiv.org/html/2609.16382)
John W\. ByersAffiliation:Department of Computer Science, Boston Universityand Mark CrovellaAffiliation:Department of Computer Science, Boston UniversityAffiliation:Faculty of Computing & Data Sciences, Boston University

September 14, 2026

###### Abstract

A language model’s representation geometry is not predetermined; it evolves as the model runs, with each layer reshaping it through attention that depends upon the surrounding context\. A faithful account of that geometry must capture that dynamic process, and so cannot be based solely on model\-independent statistics such as co\-occurrence\. Here we introduce a mean\-field analysis of attention\. The average attention from one token to another defines a kernel that carries representations layer to layer and can be iterated through the network to model how the geometry is transformed\. We condition this average two ways\. Conditioned on a whole corpus, the kernel predicts the average\-case evolution of representation geometry\. Conditioned instead on a single context, it predicts the expected geometry for that context\. A head’s departure from that prediction, its*mean\-field deviation*, isolates the context\-specific computation that the mean field misses\.

Under the corpus\-conditional reading, the kernel yields an open\-loop model: from the input embeddings and the frozen weights alone, we can iterate the kernel and the model’s own MLPs over token representations, never consulting a measured deviation at any layer\. With no free parameters, the rollout reproduces the trajectory of each token’s mean representation across models from GPT\-2 to Qwen\-3\-14B\. We show that several alternatives, including one built from co\-occurrence, are all poorer predictors\.

In early training the model and its corpus mean field are indistinguishable\. Replace every attention head with its mean field, and across three scales of Pythia — 160m, 1\.4B, and 12B — the substitution leaves the loss on real text unchanged\. Around the onset of induction, the two diverge, and the gap widens as representations become contextualized and attention becomes correlated with the values it transports\. Recent theory characterizes early\-training weights as closed\-form compositions of corpus statistics; we give a mean\-field account of the computation, under which the young model’s behavior reduces to its corpus mean field until in\-context computation emerges\.

Under the context\-conditional reading, deviation from the mean field is a task\-agnostic measure of context\-specific computation\. The residual decomposes additively into unusual attention routing and contextualization of the transported values\. Across controlled induction and few\-shot settings, greater deviation tracks greater reliance on in\-context information\. Ranking heads by corpus\-level deviation recovers documented in\-context machinery while also revealing positional and content\-tracking heads\. On ICL tasks, task\-conditioned residuals act as function vectors; we show that selecting function\-vector heads by mean\-field deviation achieves transfer accuracy comparable to causal\-mediation selection, and only requires a single forward pass\.

## 1Introduction

What happens inside a language model,*on average*, as it processes a large text corpus? This is an important question from a model interpretability standpoint: when studying the behavior of an attention head, a natural baseline can be useful\. For example, when a head attends to a particular token and writes a particular vector, it is useful to know how much of that behavior is typical for those token types and how much is specific to the present context\.

In this paper we address this need by developing empirical methods for studying the average\-case behavior of attention heads\. We construct*mean\-field approximations*of a head’s behavior by averaging the attention of the head over a corpus\. Importantly, this is a corpus\-averaged statistic of the trained model’s behavior, not a statistic of the corpus alone\. Its entries summarize how the model routes attention when processing text\. The approximation yields a kernel at the*type*level – a characterization of a head’s behavior that is parameterized by pairs of types \(the entries in the model’s token vocabulary\)\. The kernel compresses a head’s context\-dependent routing into a type\-level operator that can be estimated in a single corpus pass, without optimization or fitted parameters\.

We show three uses for the resulting approximation\. First, it yields a surprisingly accurate, compact description of the evolution of average type representations\. This description allows a drastic simplification of the study of average type representations\. Considerable work has focused on understanding representation geometry\([Karkada et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib35);[Nava and Wyart, 2026](https://arxiv.org/html/2609.16382#bib.bib45);[Engels et al\., 2025](https://arxiv.org/html/2609.16382#bib.bib18);[Gurnee and Tegmark, 2024](https://arxiv.org/html/2609.16382#bib.bib29);[Park et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib52)\), including studies arguing that representation geometry is important for understanding model behavior and steering\([Lu et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib40);[Wurgaft et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib71)\)\. A simple description of how geometry evolves over the layers of a model can inform that work\.

Second, the mean\-field yields a highly accurate description of*model behavior in early training stages*\. We show that mean\-field substitution reveals an early training phase in which mean\-field attention can replace the model’s attention on frequent query types with almost no loss cost\. This is followed by a transition where context\-specific attention becomes behaviorally important under our intervention, and studying the evolution of the mean field reveals a major reorganization at the same point\.

Third, we use*deviation*from mean\-field predictions to explore*contextual computation*in the model\. Model behavior that is not well predicted by the mean field includes syntactic processing, positional computation, and, importantly, in\-context learning \(ICL\)\. Across controlled induction and few\-shot settings, prompts requiring more in\-context computation tend to exhibit larger mean\-field deviation\. Further, when we rank heads by their average mean field deviation, we find previously documented ICL heads clustered at the top of the ranking\. When we use mean\-field deviation to study ICL tasks, we find that task\-conditioned mean\-field deviations are themselves function vectors: selecting heads by deviation magnitude and injecting their residuals yields transfer competitive with established causal\-mediation methods, at a fraction of the computational cost\.

Thus, when the mean field succeeds in predicting representations, it gives a compressed description of average representation dynamics\. When it fails, the structured residual identifies computation that depends on the particular context\. The picture that emerges from our work is that a representation can be thought of as a corpus\-average component and a context\-supplied component: global structure plus local variation\. Mean\-field attention describes the global structure, while the residual measures local, contextual innovation\.

## 2Related Work

#### Corpus statistics and representation geometry\.

Much recent work has focused on the importance of understanding representation geometry as a factor in model behavior and steering\([Lu et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib40);[Wurgaft et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib71)\)\. To that end, a number of studies have derived representational geometry directly from corpus co\-occurrence statistics\.[Karkada et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib35)argue, theoretically and empirically, that symmetries in pairwise word co\-occurrence determine the circular and continuous geometries observed in word embeddings and language model representations, covering months, years and geographic coordinates;[Nava and Wyart \(2026\)](https://arxiv.org/html/2609.16382#bib.bib45)extend the account to hierarchical geometry, connecting the spectrum of the co\-occurrence matrix to coarse\-to\-fine conceptual organization\. These provide insights into phenomena described by[Engels et al\. \(2025\)](https://arxiv.org/html/2609.16382#bib.bib18), who find circular representations of days and months and give causal evidence that models use them, and by[Gurnee and Tegmark \(2024\)](https://arxiv.org/html/2609.16382#bib.bib29), who find linear spatial and temporal representations at multiple scales\.[Park et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib52)and[Park et al\. \(2025b\)](https://arxiv.org/html/2609.16382#bib.bib53)give formal accounts of that geometry, including the role of the inner product and the distinction between a representation and its linear accessibility, and[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib34)derive the emergence of linear semantic structure from a latent\-variable model of next\-token prediction\.

Those accounts relate corpus statistics to a*static*geometry, and papers deriving representations typically do so in simple models or abstractions of models\. The mean field studied here describes how representation geometry is*transformed*layer by layer in real language models\. At the same time, our paper documents the strong correlation between the corpus\-level attention mean\-field and co\-occurrence statistics\. We argue that this helps explain why, in real models, co\-occurrence appears to have a strong impact on representation geometry\.

#### Distributional semantics\.

The premise that learned geometry can reflect low\-order corpus statistics predates transformers\.[Pennington et al\. \(2014\)](https://arxiv.org/html/2609.16382#bib.bib55)learn embeddings directly from global co\-occurrence counts,[Levy and Goldberg \(2014\)](https://arxiv.org/html/2609.16382#bib.bib39)show that skip\-gram with negative sampling implicitly factorises a shifted PMI matrix, and[Arora et al\. \(2016\)](https://arxiv.org/html/2609.16382#bib.bib2)derive embedding geometry from a latent discourse process\.[Chan et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib12)show that distributional properties of the data — burstiness, rare classes, Zipfian frequency — govern the balance between in\-context and in\-weights learning, and[Park et al\. \(2025a\)](https://arxiv.org/html/2609.16382#bib.bib51)analyze competition among unigram, bigram, retrieval and inference strategies\. This body of work motivates the hypothesis that a low\-order corpus\-average operator should capture a substantial share of model behavior\. Our paper makes this hypothesis precise and then tests it directly, finding it borne out for centroid geometry and for early training, and informatively violated for in\-context learning\.

#### Training Dynamics\.

[Chang and Bergen \(2022\)](https://arxiv.org/html/2609.16382#bib.bib13)find that early predictions resemble unigram frequencies, then bigram probabilities, becoming contextually nuanced only later, and[Choshen et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib14)report consistent orders of acquisition across architectures, initializations and datasets\. Further,[Im et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib33)show analytically that early stages of learning in transformer models are driven primarily by corpus\-average statistics\. Our results are consistent with these papers, but go further by demonstrating that in the early stages of model training, a model’s attention mechanism is*fully replaceable*by its mean field without affecting model performance\.

#### Attention heads, corpus statistics, and contextual algorithms\.

[Elhage et al\. \(2021\)](https://arxiv.org/html/2609.16382#bib.bib17)analyze attention\-only transformers in closed form, decomposing a head into a QK circuit that determines where attention is sent and an OV circuit that determines what is written when it arrives\. Their analysis documents that this machinery can respond in ways that reflect corpus\-average bigram and skip\-trigram statistics — that attention is, among other things, a means by which a model can act on the aggregate statistics of its training distribution\. The present work is consistent with that observation and goes beyond it\. We show that corpus\-average statistics are sufficient to characterize substantial aspects of model behavior: the layerwise evolution of type centroids, and the trajectory a model follows early in training\. Further, what the mean\-field fails to capture is also informative, and we use its residual to*separate*the aspects of behavior that a second\-order corpus average cannot in principle supply — the prompt\-specific computation underlying in\-context learning\. Like our paper,[Vig and Belinkov \(2019\)](https://arxiv.org/html/2609.16382#bib.bib68)analyze average attention behavior at corpus scale\. However, their focus is on the relationship between average attention and syntax, dependency relations, distance and depth\. Finally,[Ethayarajh \(2019\)](https://arxiv.org/html/2609.16382#bib.bib19)measures how far representations vary across contexts, showing that contextuality increases in upper layers — a property that our mean\-field\-versus\-residual decomposition allows one to quantify\.

#### Input\-independent and fixed attention\.

One of the conclusions of our work is that certain aspects of model behavior are relatively unchanged if the attention mechanism is replaced by a fixed token\-level kernel\. Previous work has explored this notion in other contexts\.[Tay et al\. \(2021\)](https://arxiv.org/html/2609.16382#bib.bib64)replace token–token attention with synthetic and even random alignment matrices and find them competitive, concluding that learning attention from query–key interactions is “useful but not that important”;[Raganato et al\. \(2020\)](https://arxiv.org/html/2609.16382#bib.bib60)replace all but one attention head per encoder layer with fixed, non\-learnable positional patterns without loss of translation quality; and[You et al\. \(2020\)](https://arxiv.org/html/2609.16382#bib.bib74)replace learned self\-attention with fixed, input\-agnostic Gaussians at minimal cost, while finding cross\-attention less dispensable\. Taken together these results establish that a large part of attention’s input\-dependence is dispensable for task performance, which can be seen to motivate our work\. In other words, if a model can be trained to work with attention that never looks at its input, it is reasonable to ask how much of an existing model’s behavior a fixed, corpus\-averaged operator already accounts for\. We additionally go beyond those papers by pursuing the question not at the coarse level of overall loss, but at the finer grain of individual layerwise representations\.

#### Mean\-field theories of attention\.

The term “mean field” is used in several unrelated senses in the literature\.[Kim and Suzuki \(2024\)](https://arxiv.org/html/2609.16382#bib.bib36)take a mean\-field limit over a distribution of model*parameters*to study optimization dynamics\.[Poc\-López and Aguilera \(2024\)](https://arxiv.org/html/2609.16382#bib.bib56)use dynamical mean\-field theory to characterize expected self\-attention dynamics, and[Dohmatob \(2025\)](https://arxiv.org/html/2609.16382#bib.bib16)gives an exact mean\-field analysis of softmax attention on a toy problem\. Our approach has some conceptual similarities with[Burger et al\. \(2025\)](https://arxiv.org/html/2609.16382#bib.bib10),[Castin et al\. \(2025\)](https://arxiv.org/html/2609.16382#bib.bib11),[Geshkovski et al\. \(2025\)](https://arxiv.org/html/2609.16382#bib.bib25)and[Rigollet \(2025\)](https://arxiv.org/html/2609.16382#bib.bib61)\. Like those papers, we treat tokens as interacting particles driven by a mean\-field abstraction of the attention mechanism\. However, those papers are concerned with deriving continuum or infinite\-token and infinite\-depth limits, while our work is explicitly based on finite\-sample statistics\. To our knowledge, ours is the first mean\-field treatment measured directly on trained language models and validated against their behavior\.

In other words, previous uses of attention mean\-fields study limits in the statistical\-physics sense, and they illuminate what attention does asymptotically\. The sense used here is a different one: our mean field is an expectation over a*corpus distribution*, taken at fixed finite width, depth and token count with the model’s weights frozen\. Nothing is taken to a limit; the averaging is over contexts drawn from text\.

#### Rogue dimensions, attention sinks, and anisotropy\.

In studying the geometry of language model representations, there are certain phenomena that must be taken into account\. These generally relate to degenerate directions in transformer representations\.[Timkey and van Schijndel \(2021\)](https://arxiv.org/html/2609.16382#bib.bib66)show that a few dimensions dominate similarity measures while contributing little to behavior; together with the anisotropy documented by[Ethayarajh \(2019\)](https://arxiv.org/html/2609.16382#bib.bib19)and the common\-direction analyses of[Gao et al\. \(2019a\)](https://arxiv.org/html/2609.16382#bib.bib21)and[Mu et al\. \(2018\)](https://arxiv.org/html/2609.16382#bib.bib43), this is why every metric we report is computed*after centering*to remove these effects\.

Further,[Sun et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib63)document “massive activations” that are approximately context\-invariant and act as an implicit bias rather than carrying token content\. Related outlier phenomena are reported by[Kovaleva et al\. \(2021\)](https://arxiv.org/html/2609.16382#bib.bib37),[Puccetti et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib57)and[Dettmers et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib15)\. Finally,[Xiao et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib72)and[Gu et al\. \(2025\)](https://arxiv.org/html/2609.16382#bib.bib28)describe attention sinks — excess probability mass deposited on a few uninformative positions, canonically the first — while[Bondarenko et al\. \(2023\)](https://arxiv.org/html/2609.16382#bib.bib8)and[Franco and Crovella \(2025\)](https://arxiv.org/html/2609.16382#bib.bib20)connect that “do\-nothing” routing to the outlier activations themselves\.

Taken together, it becomes clear that these rogue dimensions are not conceptually part of the context\-dependent geometry we seek to predict\. We do not attempt to correct for these in the results reported in the paper body, but in Appendix[I](https://arxiv.org/html/2609.16382#A9), we analyze them and show that omitting them from analysis improves the predictions of our method\.

#### In\-context learning\.

Accounts of in\-context learning include induction heads\([Olsson et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib49)\), implicit gradient descent\([Oswald et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib50)\), preconditioned gradient descent\([Ahn et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib1)\), general statistical algorithms with algorithm selection\([Bai et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib4)\), and compressed task representations\([Hendel et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib31)\)\. We do not offer a competing theory of what in\-context learning implements\. The mean field is used instead as a diagnostic: it quantifies when a prediction requires prompt\-specific computation rather than a corpus\-average operator, and is therefore compatible with any of these accounts of what fills that gap\.

## 3Mean Field Analysis of Attention

### 3\.1Notation

In the attention mechanism, a head at any layer operates on an input\{𝐱i\}i=1m\\\{\\mathbf\{x\}\_\{i\}\\\}\_\{i=1\}^\{m\}, with𝐱i∈ℝD\\mathbf\{x\}\_\{i\}\\in\\mathbb\{R\}^\{D\}the head’s*representation*of theiith input token: a running vector that the head reads and updates\. The head computes a separate update for each position in the input, so we say that each position defines one*context*, denotedXX\. The context at positionj∈\{1,…,m\}j\\in\\\{1,\\dots,m\\\}is the prefixX=𝐱1:jX=\\mathbf\{x\}\_\{1:j\}of the input; the attention mechanism uses the representation𝐱j\\mathbf\{x\}\_\{j\}as the*query*and the representations𝐱1:j\\mathbf\{x\}\_\{1:j\}as the*keys*for that context\. The mechanism computes a distribution over the keys,pq​\(𝐱i\)≥0p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\\geq 0with∑i=1jpq​\(𝐱i\)=1\\sum\_\{i=1\}^\{j\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)=1, and reads a*value*v⁡\(𝐱i\)∈ℝDv\(\\mathbf\{x\}\_\{i\}\)\\in\\mathbb\{R\}^\{D\}from each key𝐱i\\mathbf\{x\}\_\{i\}\. The vector the head adds to𝐱j\\mathbf\{x\}\_\{j\}, the head’s*contribution*to𝐱j\\mathbf\{x\}\_\{j\}, is111The mean\-field approach does not depend on the specifics of howp⁡\(\)p\(\)andv⁡\(\)v\(\)are realized\. As an example, in a typical implementation, tokens are first passed through a layer norm, giving𝐱~=LN​\(𝐱\)\\tilde\{\\mathbf\{x\}\}=\\text\{LN\}\(\\mathbf\{x\}\), and thenp⁡\(\)p\(\)andv⁡\(\)v\(\)are computed asp⁡\(X~\)=Softmax⁡\(\{𝐱~j⊤​WQ⊤​WK​𝐱~i\}i=1j\)p\(\\tilde\{X\}\)=\\operatorname\{Softmax\}\(\\\{\\tilde\{\\mathbf\{x\}\}\_\{j\}^\{\\top\}W\_\{Q\}^\{\\top\}W\_\{K\}\\tilde\{\\mathbf\{x\}\}\_\{i\}\\\}\_\{i=1\}^\{j\}\)andv⁡\(𝐱~\)=WO​WV​𝐱~v\(\\tilde\{\\mathbf\{x\}\}\)=W\_\{O\}W\_\{V\}\\tilde\{\\mathbf\{x\}\}\. Different models will vary in these details, and we consider models with different implementations ofp⁡\(\)p\(\)andv⁡\(\)v\(\)in our experiments\.

𝐳j=∑i=1jpq​\(𝐱i\)​v​\(𝐱i\)\.\\mathbf\{z\}\_\{j\}\\;=\\;\\sum\_\{i=1\}^\{j\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\\,v\(\\mathbf\{x\}\_\{i\}\)\.\(1\)Figure[1](https://arxiv.org/html/2609.16382#S3.F1)illustrates this contribution geometrically: the value map defines a vector field over representation space, and the head delivers to the query an attention\-weighted sample of that field\.

Figure 1:A head’s contribution on a single context \(Equation[1](https://arxiv.org/html/2609.16382#S3.E1)\)\. The value mapv⁡\(⋅\)v\(\\cdot\)assigns a vector to every point of representation space \(light\-gray field\); each key𝐱i\\mathbf\{x\}\_\{i\}\(teal\) samples the field at its own position\. The attention weightspq​\(𝐱i\)p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\(purple rings, size proportional to weight\) set how much of each sampled value the head moves: the contribution𝐳j\\mathbf\{z\}\_\{j\}\(heavy purple arrow\) is the attention\-weighted sum of the value vectors, shown assembled term\-by\-term \(thin purple path\) at the query𝐱j\\mathbf\{x\}\_\{j\}\.Each of the model’s input tokens has a*token type*, its entry in the model’s*vocabulary*𝒯\\mathcal\{T\}\. The vocabulary𝒯\\mathcal\{T\}is the set of all possible input tokens — such as ‘October’, or ‘was’\. Even though representations are not in the model vocabulary, they still naturally associate with a token type\. SinceXXarises from a specific forward pass, each𝐱i\\mathbf\{x\}\_\{i\}traces back to the model’siith input token, and𝐱i\\mathbf\{x\}\_\{i\}inherits that token’s type\. For instance, given the input “October was cold,” the position\-1 representation has token type ‘October’ at every layer, but its representation vector differs from layer to layer\. Every other*occurrence*of ‘October’ in the corpus, each token of this type at its own position and context, shares that type but has its own representation vector\. We writeT⁡\(𝐱\)T\(\\mathbf\{x\}\)for the token type of𝐱\\mathbf\{x\}, and shorten*token type*to*type*where no ambiguity arises\.

Our analysis will focus primarily on the contribution𝐳\\mathbf\{z\}that a given head makes to a token’s representation\. We will also consider averages over various conditioning sets; we always denote an average by a bar with the conditioning set in the subscript, e\.g\.,𝐳¯s\\overline\{\\mathbf\{z\}\}\_\{s\}is the average contribution given query typess\. The same convention names other averaged quantities, such as𝐯¯t​X\\overline\{\\mathbf\{v\}\}\_\{tX\}, the average value over type\-ttkeys in a contextXX\. A corpus\-level average carries only type subscripts, while a context\-level average also carries the contextXX\. Our mean field estimates of𝐳¯\\overline\{\\mathbf\{z\}\}will always be denoted𝐳^\\hat\{\\mathbf\{z\}\}\. Finally, when the particular head\(ℓ,h\)\(\\ell,h\)matters, we will note the layer and the head in the superscript, e\.g\., as𝐳ℓ​h\\mathbf\{z\}^\{\\ell h\}\.

The contribution𝐳\\mathbf\{z\}\(Equation[1](https://arxiv.org/html/2609.16382#S3.E1)\) is what a single head computes on a single context\. The mean\-field approach asks what this contribution looks like*on average over a corpus*: not for one context, but averaged over a set of contexts that share some structure\. Different choices of that set give different mean fields, and we mainly develop three here\. The first averages over contexts holding only the query type fixed, and yields a description of how the model attends on average for queries of a given type\. We refer to this as the*corpus mean field*\(§[3\.2](https://arxiv.org/html/2609.16382#S3.SS2)\)\. The second averages over just the contexts that share a pair of token types: the query and the key, and we refer to this as the\(s,t\)\(s,t\)\-conditional mean field\. The third conditions on an individual context, yielding an average description fine enough to predict what the model does on a specific input\. We call this the*context\-conditional mean field*\(§[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)\)\. Each applies the same principle over different conditioning sets, and we show that they are each useful for a distinct set of analyses\.

### 3\.2The Corpus Mean Field

A corpus is a collection of inputs \(prompts\), each of which passes through the model independently\. Because many positions across the corpus may share the same type, each type is typically realized by many representations at every layer\. At a given layer, the type’s*centroid*,𝐜s=𝔼⁡\[𝐱∣T⁡\(𝐱\)=s\],\{\\mathbf\{c\}\}\_\{s\}=\\mathbb\{E\}\\\!\\left\[\\mathbf\{x\}\\mid T\(\\mathbf\{x\}\)=s\\right\],is the mean representation over all positions of typessacross the inputs of the corpus\. Our first mean field, defined in this section, is concerned with how each head updates the centroid for typess\. This is achieved by considering all contexts whose query type isss, ie,T⁡\(q\)=T⁡\(𝐱j\)=sT\(q\)=T\(\\mathbf\{x\}\_\{j\}\)=s, and applying\([1](https://arxiv.org/html/2609.16382#S3.E1)\)\. Averaging over the corpus, the amount the head adds to the type\-sscentroid is

𝐳¯s=𝔼s​\[∑i=1jpq​\(𝐱i\)​v​\(𝐱i\)\],\\overline\{\\mathbf\{z\}\}\_\{s\}=\\mathbb\{E\}\_\{s\}\\\!\\left\[\\sum\_\{i=1\}^\{j\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\\,v\(\\mathbf\{x\}\_\{i\}\)\\right\],\(2\)where𝔼s\\mathbb\{E\}\_\{s\}denotes the average over all corpus contexts whose query is typess, ie,𝔼s=def𝔼\{X:T⁡\(𝐱j\)=s\}\.\\mathbb\{E\}\_\{s\}\\stackrel\{\{\\scriptstyle\\text\{def\}\}\}\{\{=\}\}\\mathbb\{E\}\_\{\\\{X:T\(\\mathbf\{x\}\_\{j\}\)=s\\\}\}\.

Attention is per\-position, but several positions may share a type, so we can rewrite \([2](https://arxiv.org/html/2609.16382#S3.E2)\) as a sum over*token types*rather than positions, pooling positions of the same type\. For a query of typessand a contextXX, letAs​t=∑i:T⁡\(𝐱i\)=tpq\(𝐱i\)A\_\{st\}=\\sum\_\{i:\\,T\(\\mathbf\{x\}\_\{i\}\)=t\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\), the total attention placed on positions of typett\(defined to be 0 when a token type is absent\), and let

𝐯¯t=∑i:T⁡\(𝐱i\)=tpq\(𝐱i\)v\(𝐱i\)∑i:T⁡\(𝐱i\)=tpq\(𝐱i\)\\overline\{\\mathbf\{v\}\}\_\{t\}=\\frac\{\\sum\_\{i:\\,T\(\\mathbf\{x\}\_\{i\}\)=t\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\\,v\(\\mathbf\{x\}\_\{i\}\)\}\{\\sum\_\{i:\\,T\(\\mathbf\{x\}\_\{i\}\)=t\}p\_\{q\}\(\\mathbf\{x\}\_\{i\}\)\}be the attention\-weighted mean of the value vectorsv⁡\(𝐱i\)v\(\\mathbf\{x\}\_\{i\}\)over those positions\. Then the contribution the head makes to the type\-sscentroid is, by linearity of expectation,

𝐳¯s=𝔼s​\[∑t∈𝒯As​t​𝐯¯t\]=∑t∈𝒯𝔼s​\[As​t​𝐯¯t\]=∑t∈𝒯\[𝔼s​\[As​t\]​𝔼s​\[𝐯¯t\]\+Covs⁡\(As​t,𝐯¯t\)\]\.\\overline\{\\mathbf\{z\}\}\_\{s\}=\\mathbb\{E\}\_\{s\}\\left\[\\sum\_\{t\\in\\mathcal\{T\}\}A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\\right\]=\\sum\_\{t\\in\\mathcal\{T\}\}\\mathbb\{E\}\_\{s\}\\\!\\left\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\\right\]=\\sum\_\{t\\in\\mathcal\{T\}\}\\left\[\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]\\,\\mathbb\{E\}\_\{s\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\+\\operatorname\{Cov\}\_\{s\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)\\right\]\.\(3\)
#### Mean\-field approximation\.

To construct a mean\-field estimate𝐳^s\\hat\{\\mathbf\{z\}\}\_\{s\}for𝐳¯s\\overline\{\\mathbf\{z\}\}\_\{s\}, we introduce two*mean\-field assumptions\.*We emphasize that we do not assert that these assumptions hold*in general*; rather, they are useful in analyzing the behavior of the model’s centroids specifically\. Additionally, our subsequent analyses \(§[5](https://arxiv.org/html/2609.16382#S5)and §[6](https://arxiv.org/html/2609.16382#S6)\) will concern when mean\-field assumptions*do not*hold\.

Assumption 1\. Attention\-Value independence\.Covs⁡\(As​t,𝐯¯t\)≈0\\operatorname\{Cov\}\_\{s\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)\\approx 0\. This assumption posits that attention is approximately uncorrelated with the*value*a head moves\. Note, however, that attention may still correlate with the type’s*representation*\.

Assumption 2\. Negligible curvature\.𝔼s​\[𝐯¯t\]≈v⁡\(𝐜t\)\.\\mathbb\{E\}\_\{s\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\approx v\(\{\\mathbf\{c\}\_\{t\}\}\)\.This assumption posits that averaging the value map over a type’s occurrences is approximately the same as applying it once at the type’s centroid, as it would be if the value map were linear\.

Assumption[3\.2](https://arxiv.org/html/2609.16382#S3.SS2.SSS0.Px1)involves two subtleties\. First, the sets over which the expectation is taken differ on the two sides of the equation:𝔼s​\[𝐯¯t\]\\mathbb\{E\}\_\{s\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]is the mean type\-ttvalue across type\-sscontexts, whereas𝐜t\\mathbf\{c\}\_\{t\}is typett’s global centroid, based on all occurrences of typett\. Second, typical value maps are nonlinear \(e\.g\., due to the layer norm\), so the approximation does not hold exactly\. The discrepancy due to nonlinearity is the value map’s*Jensen gap\.*222The Jensen gap is a general phenomenon: for any nonlinear map, it is the difference between applying the map at an average and averaging the map’s outputs\. The same gap arises later for the MLP \(§[4](https://arxiv.org/html/2609.16382#S4)\)\.We evaluate the impact of the value map’s Jensen gap, and show that it is small, in Appendix[C](https://arxiv.org/html/2609.16382#A3)\.

Because of the central role of𝔼s​\[As​t\]\\mathbb\{E\}\_\{s\}\[A\_\{st\}\], we refer to it as the \(corpus\)*attention mean field*and denote it byPP, with entries

Ps​t=𝔼s​\[As​t\],P\_\{st\}=\\mathbb\{E\}\_\{s\}\[A\_\{st\}\],\(4\)the mean attention a type\-ssquery places on typettacross the corpus\.

Applying this substitution and the two assumptions to \([3](https://arxiv.org/html/2609.16382#S3.E3)\) yields the head’s*mean\-field contribution*𝐳^s\\hat\{\\mathbf\{z\}\}\_\{s\}, an approximation to the true averaged contribution𝐳¯s\\overline\{\\mathbf\{z\}\}\_\{s\}:

𝐳^s=∑t∈𝒯Ps​t​v​\(𝐜t\)≈𝐳¯s\.\\hat\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}P\_\{st\}\\,v\(\{\\mathbf\{c\}\_\{t\}\}\)\\;\\approx\\;\\overline\{\\mathbf\{z\}\}\_\{s\}\.\(5\)Figure[2](https://arxiv.org/html/2609.16382#S3.F2)illustrates the update: the per\-context attention of Figure[1](https://arxiv.org/html/2609.16382#S3.F1)is replaced by the fixed kernelPP, applied to the value map at the type centroids\.

Figure 2:The corpus mean\-field update \(Equation[5](https://arxiv.org/html/2609.16382#S3.E5)\)\. Each token typetthas a centroid𝐜t\\mathbf\{c\}\_\{t\}\(filled stars\) surrounded by its cloud of corpus occurrences \(small dots\)\. The mean field replaces per\-context attention with the fixed kernelPs​tP\_\{st\}\(gray edges, width proportional to the mean attention that a type\-ssquery places on typett\): the estimate𝐳^s\\hat\{\\mathbf\{z\}\}\_\{s\}\(heavy purple arrow\) applies the value map once at each centroid,v⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\)\(colored arrows\), and combines the results with weightsPs​tP\_\{st\}\. Summing the head contributions and applying the layer’s MLP yields the next\-layer centroid estimate𝐜^sℓ\+1\\hat\{\\mathbf\{c\}\}^\{\\,\\ell\+1\}\_\{s\}\(open star\); this is the update that the rollout of §[4](https://arxiv.org/html/2609.16382#S4)iterates across layers\.Equation \([5](https://arxiv.org/html/2609.16382#S3.E5)\) represents a drastic simplification of the model’s processing\. Rather than context\-specific attention applied to each representation over the entire corpus, the mean\-field estimate applies a single constant linear mapPs​tP\_\{st\}and the value maps of the type centroidsv⁡\(𝐜t\)v\(\{\\mathbf\{c\}\}\_\{t\}\)to derive𝐳^s\\hat\{\\mathbf\{z\}\}\_\{s\}\. This estimate only requires precomputingPPonce from a corpus\.

Each row ofPPis an average of probability distributions, soPPis row\-stochastic: the corpus\-conditional mean field is itself a valid attention pattern, one we can use in a replacement model\. This is the form the rollout in §[4](https://arxiv.org/html/2609.16382#S4)applies at each head and layer\.

### 3\.3Conditional Mean Fields

Next we introduce two more narrowly\-conditioned mean fields: the\(s,t\)\(s,t\)\-conditional mean field and the context\-conditional mean field\. The\(s,t\)\(s,t\)\-conditional mean field is of independent interest, and we use it to introduce certain key concepts\. Then we move to the context\-conditional mean field which is the basis for the results in §[5](https://arxiv.org/html/2609.16382#S5)and §[6](https://arxiv.org/html/2609.16382#S6)\.

For the purposes of exposition, in the remainder of this subsection, we make the simplifying assumption that a given typettcan only occur*at most once*in a context\. Making this assumption considerably simplifies the presentation compared to the case where a type can occur multiple times in a context\. We give full derivations that remove this assumption in Appendix[A](https://arxiv.org/html/2609.16382#A1)\.

#### The\(s,t\)\(s,t\)\-conditional mean field\.

We first show that the contribution to*each*centroid \(as a query\) from each token type \(as a key\) can be approximated under a mean\-field assumption\.

Start from the exact contribution to the centroid \([3](https://arxiv.org/html/2609.16382#S3.E3)\):

𝐳¯s=∑t∈𝒯𝔼s​\[As​t​𝐯¯t\]\\overline\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}\\mathbb\{E\}\_\{s\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]
We now condition on a typettbeing in the context\. We use𝔼s​t\\mathbb\{E\}\_\{st\}to denote the conditional expectation over contexts havingssas the query type and in which typettis present\. We usePrs⁡\(t\)\\Pr\_\{s\}\(t\)to denote the probability thatttis in a context chosen uniformly at random having query typess\.333Note the corner case fort=st=s:Prs⁡\(s\)=1\\Pr\_\{s\}\(s\)=1\.We then have

𝔼s​\[As​t​𝐯¯t\]=𝔼s​t​\[As​t​𝐯¯t\]​Prs⁡\(t\)\.\\mathbb\{E\}\_\{s\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\,\{\\Pr\}\_\{s\}\(t\)\.\(6\)This follows because when typettis not in the context,As​t=0A\_\{st\}=0\.

Denote𝔼s​t​\[As​t​𝐯¯t\]\\mathbb\{E\}\_\{st\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]as𝐳¯s​t\\overline\{\\mathbf\{z\}\}\_\{st\}\. This is the average contribution to thesstype centroid by typettwhenttis in the context\. This is a natural measure of how each type influences typesswhen it appears inss’s context\. Then

𝐳¯s=∑t∈𝒯𝐳¯s​t​Prs⁡\(t\)\\overline\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}\\overline\{\\mathbf\{z\}\}\_\{st\}\\,\{\\Pr\}\_\{s\}\(t\)\(7\)This is an exact decomposition of the corpus contribution in terms oftt\-conditional contributions\. We can rewrite𝐳¯s​t\\overline\{\\mathbf\{z\}\}\_\{st\}as:

𝐳¯s​t=𝔼s​t​\[As​t​𝐯¯t\]=𝔼s​t​\[As​t\]​𝔼s​t​\[𝐯¯t\]\+Covs​t⁡\(As​t,𝐯¯t\)\\overline\{\\mathbf\{z\}\}\_\{st\}=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\\,\\mathbb\{E\}\_\{st\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\+\\operatorname\{Cov\}\_\{st\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)Now we introduce two mean\-field assumptions: first, the\(s,t\)\(s,t\)\-conditional analog of Assumption[3\.2](https://arxiv.org/html/2609.16382#S3.SS2.SSS0.Px1), and second, that the average value mapped by a key of typettis independent ofss:

Assumption 3\. Attention\-value independence,\(s,t\)\(s,t\)\-conditional \.This posits thatCovs​t⁡\(As​t,𝐯¯t\)≈0\\operatorname\{Cov\}\_\{st\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)\\approx 0, now on the finer conditioning set of contexts having a query of typessand typettamong the keys\.

Next, define𝝁t\\boldsymbol\{\\mu\}\_\{t\}to be the corpus average value map of typettacting as a key, ie,𝝁t=𝔼⁡\[v⁡\(𝐱i\)∣T⁡\(𝐱i\)=t\]\\boldsymbol\{\\mu\}\_\{t\}=\\mathbb\{E\}\\\!\\left\[v\(\\mathbf\{x\}\_\{i\}\)\\mid T\(\\mathbf\{x\}\_\{i\}\)=t\\right\]\. This is computed directly from the corpus by the same recipe as the centroid𝐜t\{\\mathbf\{c\}\}\_\{t\}: condition on typettand average over its corpus positions, here averaging the frozen value map’s outputv⁡\(𝐱\)v\(\\mathbf\{x\}\)rather than the representation𝐱\\mathbf\{x\}\.444𝝁t\\boldsymbol\{\\mu\}\_\{t\}is the measured average of the value outputs, as against the mapped centroidv⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\)of Assumption[3\.2](https://arxiv.org/html/2609.16382#S3.SS2.SSS0.Px1); both quantities are used in the sections that follow\. Definingμt\\mu\_\{t\}also makes that assumption’s Jensen gap precise, as the difference𝝁t−v⁡\(𝐜t\)\\boldsymbol\{\\mu\}\_\{t\}\-v\(\\mathbf\{c\}\_\{t\}\)\.

Assumption 4\. Value\-query independence\.This assumption posits that𝔼s​t​\[𝐯¯t\]≈𝝁t\\mathbb\{E\}\_\{st\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\approx\\boldsymbol\{\\mu\}\_\{t\}, i\.e\., conditioning on the query typesshas little effect on the average value transmitted by typettkeys\.

As in the corpus mean field case, we define the\(s,t\)\(s,t\)\-conditional mean field

Ws​t=𝔼s​t​\[As​t\],W\_\{st\}=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\],\(8\)and note that \([6](https://arxiv.org/html/2609.16382#S3.E6)\) withAs​tA\_\{st\}alone in place ofAs​t​𝐯¯tA\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}givesPs​t=Ws​t​Prs⁡\(t\)P\_\{st\}=W\_\{st\}\\,\\Pr\_\{s\}\(t\)\. SoWWandPPare easily computed from one another\. We now apply Assumptions[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px1)and[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px1)to give our mean\-field approximation to the contribution to the centroid ofss, as a per\-ttcontribution, and show that it decomposes𝐳^\\hat\{\\mathbf\{z\}\}:

𝐳^s​t=Ws​t​𝝁t⟹𝐳^s=∑t∈𝒯𝐳^s​t​Prs⁡\(t\)≈𝐳¯s\\hat\{\\mathbf\{z\}\}\_\{st\}=W\_\{st\}\\boldsymbol\{\\mu\}\_\{t\}\\qquad\\Longrightarrow\\qquad\\hat\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}\\hat\{\\mathbf\{z\}\}\_\{st\}\\,\{\\Pr\}\_\{s\}\(t\)\\approx\\overline\{\\mathbf\{z\}\}\_\{s\}Substituting this back also confirms that the\(s,t\)\(s,t\)\-conditional mean field is consistent with the corpus mean field it refines:𝐳^s=∑t∈𝒯Ps​t​𝝁t\\hat\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}P\_\{st\}\\boldsymbol\{\\mu\}\_\{t\}, which is \([5](https://arxiv.org/html/2609.16382#S3.E5)\) with𝝁t\\boldsymbol\{\\mu\}\_\{t\}in place ofv⁡\(𝐜t\)v\(\{\\mathbf\{c\}\_\{t\}\}\)\. The two estimators differ by∑tPs​t​\(𝝁t−v⁡\(𝐜t\)\)\\sum\_\{t\}P\_\{st\}\\bigl\(\\boldsymbol\{\\mu\}\_\{t\}\-v\(\\mathbf\{c\}\_\{t\}\)\\bigr\), the value map’s Jensen gap of each key type weighted by the kernel, so they coincide when that gap is zero\. In our experiments the gap is small; we document the small Jensen gap of the value map and its small effect on predictions in Appendix[C](https://arxiv.org/html/2609.16382#A3)\. So we find that the two constructions agree in practice\.

#### The context\-conditional mean field\.

Next, we use a similar strategy to make an average case prediction for a particular contextXX\. We assume thatXXhas query typess\. We denote the contribution made to query typesswhen it has contextXXas𝐳s​X\{\\mathbf\{z\}\}\_\{sX\}, the value map from typettin that context as𝝁t​X\\boldsymbol\{\\mu\}\_\{tX\}, the attention paid bysstottconditioned on the context beingXXasWs​t​XW\_\{stX\}, and the analog ofPrs⁡\(t\)\\Pr\_\{s\}\(t\)for contextXXasPrX⁡\(t\)\\Pr\_\{X\}\(t\)\. Two of these quantities degenerate becauseXXis a single fixed context: the averageWs​t​XW\_\{stX\}is the actual attentionAs​t​XA\_\{stX\}the head pays fromsstottinXX, andPrX⁡\(t\)\\Pr\_\{X\}\(t\)is 1 for the types inXXand 0 otherwise\.

As in the\(s,t\)\(s,t\)\-conditional decomposition \([7](https://arxiv.org/html/2609.16382#S3.E7)\), these quantities give an exact per\-ttdecomposition of the contribution, now for the single contextXX:

𝐳s​X=∑t∈𝒯Ws​t​X​𝝁t​X​PrX⁡\(t\)=∑t∈XAs​t​X​𝝁t​X\{\\mathbf\{z\}\}\_\{sX\}=\\sum\_\{t\\in\\mathcal\{T\}\}W\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{tX\}\\,\{\\Pr\}\_\{X\}\(t\)\\,=\\sum\_\{t\\in X\}A\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{tX\}\(9\)
To construct a mean field analog, we introduce two mean\-field assumptions\.

Assumption 5\. Value\-context independence\.This posits that𝝁t​X≈𝝁t\\boldsymbol\{\\mu\}\_\{tX\}\\,\\approx\\,\\boldsymbol\{\\mu\}\_\{t\}\. That is, the value map of typettin contextXXis well\-approximated by its corpus average\. This is an important simplification; the difference between𝝁t​X\\boldsymbol\{\\mu\}\_\{tX\}and𝝁t\\boldsymbol\{\\mu\}\_\{t\}can be thought of as the*contextualization*of the value map ofttinduced by its contextXX\.

Assumption 6\. Proportionality of context attention\.This assumption posits that for eachss, the actual attention valuesAs​t​XA\_\{stX\}are proportional toWs​tW\_\{st\}across the types present inXX\. The constant of proportionality is then fixed by the requirement that a head’s attention over its context sums to one\.

Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px2)lets our mean\-field kernelA^s​t​X\\hat\{A\}\_\{stX\}estimate the real attentionAs​t​XA\_\{stX\}by renormalizingWs​tW\_\{st\}over the types inXX\(the general case of repeated types is treated in Appendix[A](https://arxiv.org/html/2609.16382#A1)\):

A^s​t​X=Ws​t∑t′∈XWs​t′≈Ws​t​X=As​t​X\.\\hat\{A\}\_\{stX\}=\\frac\{W\_\{st\}\}\{\\sum\_\{t^\{\\prime\}\\in X\}W\_\{st^\{\\prime\}\}\}\\,\\approx\\,W\_\{stX\}\\,=\\,A\_\{stX\}\.\(10\)
Continuing from \([9](https://arxiv.org/html/2609.16382#S3.E9)\), we substitute the corpus average𝝁t\\boldsymbol\{\\mu\}\_\{t\}for𝝁t​X\\boldsymbol\{\\mu\}\_\{tX\}\(Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px2)\) andA^s​t​X\\hat\{A\}\_\{stX\}for the attentionAs​t​XA\_\{stX\}\(Equation \([10](https://arxiv.org/html/2609.16382#S3.E10)\)\) to derive our*context\-conditional*mean field estimator:

𝐳^s​X=∑t∈XA^s​t​X​𝝁t≈𝐳s​X\.\\hat\{\\mathbf\{z\}\}\_\{sX\}=\\sum\_\{t\\in X\}\\hat\{A\}\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{t\}\\,\\approx\\,\{\\mathbf\{z\}\}\_\{sX\}\.\(11\)

#### Analysis ofA^\\hat\{A\}\.

Unlike the corpus mean field and the\(s,t\)\(s,t\)\-conditional mean field, the context\-conditional mean field computed here is a nonlinear function of model\-induced statistics \(because of the normalization used in \([10](https://arxiv.org/html/2609.16382#S3.E10)\)\)\. As such, a desirable property would be*unbiasedness*; specifically, we would like that𝔼s​t​\[As​t​X−A^s​t​X\]=0\.\\mathbb\{E\}\_\{st\}\[A\_\{stX\}\-\\hat\{A\}\_\{stX\}\]=0\.We analyze this question in Appendix[B](https://arxiv.org/html/2609.16382#A2), and show: \(1\)A^\\hat\{A\}is not, in general, unbiased by this definition; \(2\) the bias we observe inA^\\hat\{A\}in practice is quite small; and \(3\) removing the observed bias inA^\\hat\{A\}does not change any of our key results\.

## 4Mean Field Prediction of Representation Evolution

As a first application of mean\-field attention analysis, we use the mean field approximation to predict type centroid evolution across the layers of a language model\. The true model modifies the representation of each input token at each layer, given its context\. By tracking type centroids we can observe the evolution of type representation geometry layer by layer\.

Concretely, the mean\-field approximation \([5](https://arxiv.org/html/2609.16382#S3.E5)\) gives the average contribution to a type’s centroid for a single head from the static mean\-field attention and the frozen weights alone\. Iterating this through the layers, and through the MLP at each layer, predicts how a type’s centroid evolves across the whole network\.

### 4\.1Method: Open\-Loop Rollout

To construct the model’s mean\-field replacement, we run a purely open\-loop computation, analogous to a dynamical system, shown in Algorithm[1](https://arxiv.org/html/2609.16382#algorithm1)\. The process starts with input embeddings \(the layer\-0 centroids\),555In GPT\-2, a mean positional offset is added\. Initializing GPT\-2 with input embeddings rather than measured centroids has negligible effect\. Using measured centroids in GPT\-2 gives L1 centered cosine prediction of 0\.989, while initializing from rawWEW\_\{E\}embeddings gives 0\.984\.and then for each layer and each type, it first applies the mean\-field approximation \([5](https://arxiv.org/html/2609.16382#S3.E5)\) for each head \(line 4\), then sums head contributions and adds them to the current centroid \(line 5\), and then passes the resulting updated centroids through the layer’s MLP \(line 6\)\. The resulting centroid approximations are then passed to the next layer, where the process repeats\.

Algorithm 1Open\-loop mean\-field rolloutinput :layer\-0 centroid representations

\{𝐜^s0\}\\\{\\hat\{\\mathbf\{c\}\}^\{0\}\_\{s\}\\\}\(input embeddings\); mean fields

Pℓ​hP^\{\\ell h\}; frozen weights

output :predicted centroid representations

\{𝐜^sℓ\}\\\{\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{s\}\\\}for each type

ssand layer

ℓ\\ell
1for*each layerℓ=0\\ell=0toL−1L\-1*do

2for*each types∈𝒯s\\in\\mathcal\{T\}*do

3for*each headhh*do

4

𝐳^sℓ​h←∑t∈𝒯Ps​tℓ​h​v​\(𝐜^tℓ\)\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}\\leftarrow\\sum\_\{t\\in\\mathcal\{T\}\}P^\{\\ell h\}\_\{st\}\\,v\(\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{t\}\);//per\-head contribution, \([5](https://arxiv.org/html/2609.16382#S3.E5)\)

5

𝐦sℓ←𝐜^sℓ\+∑h𝐳^sℓ​h\\mathbf\{m\}^\{\\ell\}\_\{s\}\\leftarrow\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{s\}\+\\sum\_\{h\}\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\};//add attention contributions

6

𝐜^sℓ\+1←𝐦sℓ\+MLPℓ​\(LN⁡\(𝐦sℓ\)\)\\hat\{\\mathbf\{c\}\}^\{\\ell\+1\}\_\{s\}\\leftarrow\\mathbf\{m\}^\{\\ell\}\_\{s\}\+\\mathrm\{MLP\}^\{\\ell\}\(\\mathrm\{LN\}\(\\mathbf\{m\}^\{\\ell\}\_\{s\}\)\);//apply MLP to the average

The rollout is a self\-contained computation of the network’s average\-case behavior for each type, built only from mean\-field kernel and the frozen weights\. Each layer maps type centroids to type centroids, so one layer’s output is the next layer’s input, and the rollout iterates with no forward pass on any text\. Analogous to the mean\-field substitution for attention \([5](https://arxiv.org/html/2609.16382#S3.E5)\), we apply the MLP only once, at the post\-attention average\. This is again a drastic simplification of what the MLP does: one execution at the centroid average, rather than averaging many executions of the MLP over each specific type\-sscontext\. Ultimately, the centroid trajectory that the rollout produces is a prediction, whose accuracy we demonstrate in the remainder of this section\. §[4\.3](https://arxiv.org/html/2609.16382#S4.SS3)attributes the rollout’s error among these approximations\.

### 4\.2Basic Results and Ablations

We now evaluate the performance of mean\-field prediction of type centroids as obtained via Algorithm[1](https://arxiv.org/html/2609.16382#algorithm1)\. Our evaluation spans five models: GPT\-2 Small\([Radford et al\., 2019](https://arxiv.org/html/2609.16382#bib.bib58)\), Gemma\-2 2B\([Gemma Team, 2024](https://arxiv.org/html/2609.16382#bib.bib24)\), Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib27)\), Llama\-3\.2\-3B\([Meta AI, 2024](https://arxiv.org/html/2609.16382#bib.bib42)\), and Qwen\-3\-14B\([Yang et al\., 2025](https://arxiv.org/html/2609.16382#bib.bib73)\)\. Kernel and centroid statistics are estimated from OpenWebText\([Gokaslan et al\., 2019](https://arxiv.org/html/2609.16382#bib.bib26)\), an open reproduction of GPT\-2’s training corpus: text samples of 1\.5–2M tokens are tokenized with each model’s tokenizer and packed into 512\-token inputs\. The kernel and per\-type statistics on theN=1000N=1000most frequent types are estimated as defined in §[3\.2](https://arxiv.org/html/2609.16382#S3.SS2)\(the attention mean fieldPP, \([4](https://arxiv.org/html/2609.16382#S3.E4)\), and the type centroids\)\. The top 1000 tokens cover6464–68%68\\%of corpus positions and6767–72%72\\%of non\-BOS attention across all five models, a range nearly independent of vocabulary size:10001000types out of Gemma\-2’s256,000256\{,\}000cover as much text as10001000out of GPT\-2’s50,25750\{,\}257\(Appendix[E](https://arxiv.org/html/2609.16382#A5)\)\. Held\-out evaluation uses a document\-interleaved split: alternating documents are assigned to two disjoint halves of the corpus, the kernel and all corpus statistics are estimated on one half, and the true centroids used for scoring are measured on the other \(GPT\-2 and Gemma\-2; the Qwen\-3 and Llama results use a single split, as noted where they appear\)\. Appendix[D](https://arxiv.org/html/2609.16382#A4)gives the two extra kernel columns, one for BOS and one pooling every non\-tracked type, that keep each row ofPℓ​hP^\{\\ell h\}stochastic when only the topNNtypes are tracked\.

In Appendix[F](https://arxiv.org/html/2609.16382#A6)we perform additional evaluations on three different corpora: FineWeb\([Penedo et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib54)\), WikiText\([Merity et al\., 2017](https://arxiv.org/html/2609.16382#bib.bib41)\), and Pre\-1919 books\([Rae et al\., 2020](https://arxiv.org/html/2609.16382#bib.bib59)\)\. We find that the results in this section are not specific to the OpenWebText corpus, and in fact that results shown here even generalize across corpora when using one corpus for training and a different corpus for testing\.

We assess prediction quality with four metrics: centered cosine similarity, RSA, relative Euclidean error, and norm ratio \(defined in Appendix[G](https://arxiv.org/html/2609.16382#A7)\)\. Each is computed per type and then averaged\. Relative error is computed as a ratio of means \(mean Euclidean error over mean distance to the entire type set’s centroid\) so that types lying near the centroid do not arbitrarily inflate it\. All are computed after mean\-centering the representation cloud\. Centering is necessary because the representations are strongly anisotropic as discussed in Appendix[H](https://arxiv.org/html/2609.16382#A8)\. Further, we evaluate the effect of rogue dimensions \(aka ‘massive activations’, e\.g\.,[Timkey and van Schijndel \(2021\)](https://arxiv.org/html/2609.16382#bib.bib66);[Kovaleva et al\. \(2021\)](https://arxiv.org/html/2609.16382#bib.bib37)\) on our results in Appendix[I](https://arxiv.org/html/2609.16382#A9)\.

The results for cosine similarity are shown in Figure[3](https://arxiv.org/html/2609.16382#S4.F3)for GPT\-2, Gemma\-2 and Qwen\-3\. Results for these models on the other three metrics are consistent and are given in Appendix[G](https://arxiv.org/html/2609.16382#A7)\. The Llama models are the exception, and Appendix[J](https://arxiv.org/html/2609.16382#A10)treats them separately; §[4\.3](https://arxiv.org/html/2609.16382#S4.SS3)explains why\. Overall, the figures show that*mean field predictions of type centroids prove highly accurate*\. In GPT\-2, centered cosine similarity exceeds 0\.9 through the middle layers and remains above 0\.75 even in the last layer\. In Qwen\-3 and Gemma\-2, cosine similarities remain well above 0\.75 through most of the network\. Given the high dimension of the representation space, these reflect very close alignment of predictions with actual centroids\.666Why many of the curves are not monotone\.The rollout is open\-loop, so one expects error to compound and accuracy to fall steadily with depth\. However, as the curves in Figures[3](https://arxiv.org/html/2609.16382#S4.F3)and[4](https://arxiv.org/html/2609.16382#S4.F4)show, cosine is not always monotone\. This is explained by noting that cosine error improves when the prediction’s extent along the true direction grows faster than its sideways error\. For example, this can happen when the residual stream’s magnitude grows faster than the rollout accumulates new error\. As discussed in Appendix[H](https://arxiv.org/html/2609.16382#A8), residual stream magnitude typically grows steadily with depth\.

To assess the importance of the mean\-field kernel in producing the accurate results here, we compare against four alternatives\. Each alternative replaces one part of the construction with something simpler; their curves appear alongside the mean field in Figure[3](https://arxiv.org/html/2609.16382#S4.F3)\. Our first substitute iscontext\-free: a baseline that propagates each type through the network in isolation \(i\.e\., running the model on just \[BOS,𝐜t\\mathbf\{c\}\_\{t\}\]\)\. The mean field approach improves on this comparison at every layer, and increasingly so with depth\. Second, because the mean field is similar to a corpus\-average object, a natural question is whether we need to compute attention at all: can co\-occurrence statistics over the corpus suffice? In thecoocalternative, the type\-to\-type distributionSSis taken from a symmetric±\\pm5\-word co\-occurrence matrix over the corpus,CC, normalized to be row\-stochastic\. This is used to distribute attention across the centroids, correcting for the fraction of attention that is allocated to the top 1000 tokens\.777We normalizeC~s​t=Cs​t/∑t′Cs​t′\\tilde\{C\}\_\{st\}=C\_\{st\}/\\sum\_\{t^\{\\prime\}\}C\_\{st^\{\\prime\}\}\. We then form the row sumsTs=∑t=11000Ps​tT\_\{s\}=\\sum\_\{t=1\}^\{1000\}P\_\{st\}and use as the final attention\-substituteTs​C~s​t\.T\_\{s\}\\tilde\{C\}\_\{st\}\.Mean field outperformscoocas well\. Finally, we consider whether the kernel’s depth structure can be simplified\.P0onlyreuses the*layer\-0*kernel at every layer \(head by head\) testing whether depth evolution of the kernel matters\. The last alternative,Pavg, replaces it with the kernel averaged over layers\. Both are distinctly worse than the measured mean field in both models, meaning that the kernel’s variation across depth carries important content for prediction\.

Figure 3:Mean\-field predictions are accurate and superior to alternative kernels\. Average per\-type centered cosine distance\. Left: Gemma\-2, Middle: GPT\-2, Right: Qwen\-3\-14B\.We next ask whether the specific structure of attention mean fieldPPis necessary, or whether a structureless version could do as well\. For example, how important is it to distinguish between query types, as well as between key types? We study three cases: First,meanq\(query\-blind attention\): for each\(ℓ,h\)\(\\ell,h\), every query row inPPis replaced by the*average row*\. Every type then reads the same mixture of key types, so the attention term contributes an identical vector to every type — pure drift\. Second,uniform\(key\-blind attention\): each query’s total type\-to\-type budget∑tPs,t\\sum\_\{t\}P\_\{s,t\}is preserved but spread evenly over all key types\. Finally,rowshuffle\(negative control\): for each head, the query rows of the kernel are permuted randomly so typeqqreceives the actual attention profile of some*other*type\. This allows us to distinguish “the graph structure ofPPis uninformative” \(in which case shuffling would matchmeanq\) from “PP’s graph carries type\-specific information” \(where shuffling should do*worse*thanmeanq, because wrong type\-specific content is actively injected\)\. Figure[4](https://arxiv.org/html/2609.16382#S4.F4)shows that the full structure ofPPis important: none of the structureless approximations match the measured kernel, so accurate prediction of centroid evolution requires type\-level distinctions among both queries and keys, and the type\-resolved structure matters increasingly with depth\.

Figure 4:Mean\-field predictions are superior to query\-blind and key\-blind alternatives\. Average per\-type centered cosine distance\. Left: Gemma\-2, Middle: GPT\-2, Right: Qwen\-3\-14B\.Table[1](https://arxiv.org/html/2609.16382#S4.T1)reports the cross\-layer mean \(standard error over layers\) of the centered cosine and relative error \(lower is better\) for the mean field, the context\-free baseline,coocand the other ablations we consider\. The measured mean field is most accurate in both metrics and all three models\. However, among the alternatives, co\-occurrence \(cooc\) is closest to matching the mean\-field on GPT\-2 and Gemma\-2\. This is an interesting observation that*simple co\-occurrence can reproduce much \(but not all\) of the effects of mean\-field attention on centroid evolution\.*

Table 1:Cross\-layer fidelity of the mean\-field rollout, the context\-free baseline,cooc, and mean\-field ablations\.∗Note on the Qwen\-3\-14B ablations\.Qwen’scooc,uniformandPavgentries show large relative errors which are produced by a numerical instability of that model’s rollout concentrated at layer 0 and not a reflection of kernel quality; Appendix[I](https://arxiv.org/html/2609.16382#A9)gives the full accounting\.

### 4\.3Understanding Approximation Errors

Algorithm[1](https://arxiv.org/html/2609.16382#algorithm1)incorporates two approximations at every layer: the mean\-field prediction of each head’s contribution𝐳^sℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}, and the application of the MLP once at the resulting centroid\. Only the first is a property of mean\-field attention; the second is a property of the rollout as a way of propagating centroids\.

Accordingly, we first ask: how good is the mean\-field approximation of attention? We measure this by comparing𝐳¯\\overline\{\\mathbf\{z\}\}with its approximation𝐳^\\hat\{\\mathbf\{z\}\}\. Figure[5](https://arxiv.org/html/2609.16382#S4.F5)does this teacher\-forced: starting from the*true*centroids at each layer, it compares the predicted attention\-block output∑h𝐳^sℓ​h\\sum\_\{h\}\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}against the same quantity measured over the corpus,∑h𝐳¯sℓ​h\\sum\_\{h\}\\overline\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}, averaging across all typesss\.888Bootstrap CIs over types are less than±\\pm0\.01 and so not shown\.The prediction is good at every layer of every model: across all five models, cross\-layer mean centered cosine lies between0\.780\.78and0\.870\.87, and the norm ratio stays near one \(cross\-layer means0\.910\.91to1\.241\.24\), so the mean field does well at recovering both the direction and the scale of what each head writes\.

Figure 5:Teacher\-forced prediction of the head contribution𝐳¯\\overline\{\\mathbf\{z\}\}, by fractional depth\. Left: centered cosine; right: centered norm ratio, with the dotted line marking zero error\.We next decompose the per\-layer error using three increasingly approximate estimators of the layer\-ℓ\+1\\ell\{\+\}1centroid, measured for GPT\-2, Gemma\-2, and Qwen\-3\-14B; Llama is discussed below\. All three are applied to the true layer\-ℓ\\ellcentroids and end with the MLP applied once at the resulting centroid; each makes one more approximation in the attention block than the last\. Estimator \(a\) isolates the MLP gap by using exact attention for the rest of the prediction \(𝐳s\\mathbf\{z\}\_\{s\}, per \([3](https://arxiv.org/html/2609.16382#S3.E3)\)\), so that applying the MLP at the centroid is its only approximation\.999For Gemma\-2, this also includes the post\-attention norm at the centroid\.Estimator \(b\) adds the further approximation of taking the per\-type value at the centroid, i\.e\., replacing𝔼s​\[𝐯¯t\]\\mathbb\{E\}\_\{s\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]withv⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\)\(still keeping the attention\-covariance term\); and estimator \(c\) applies the full mean field of \([5](https://arxiv.org/html/2609.16382#S3.E5)\), which additionally drops the covariance term\.

The decomposition confirms that MLP\-introduced error is the main source of rollout inaccuracy \(Figure[6](https://arxiv.org/html/2609.16382#S4.F6)\): the dominant per\-layer source of error is the MLP evaluated at the centroid\. Estimator \(a\) \(exact attention at the centroid rather than per occurrence\) accounts for almost all of the per\-layer error, falling short of perfect prediction by approximately0\.02−0\.030\.02\-0\.03across all models in centered cosine similarity on average\. The two further approximations then add little: taking the value map at the centroid \(\(a\)→\\to\(b\)\) and dropping the covariance term \(\(b\)→\\to\(c\)\) each cost at most∼0\.005\{\\sim\}0\.005per layer\. The final error of the full open\-loop rollout \(Figure[3](https://arxiv.org/html/2609.16382#S4.F3)\) therefore does not arise from any single step being inaccurate, but from the accumulation of these small per\-layer errors across depth, with the MLP\-at\-centroid approximation the leading per\-step contributor\. Measured on its own, the MLP gap is less significant than this may suggest, as we explain in Appendix[K](https://arxiv.org/html/2609.16382#A11)\.

Figure 6:Per\-layer accounting of mean\-field approximation error\. Starting from the true centroids at layerℓ\\ell, three nested predictions of the layer\-ℓ\+1\\ell\{\+\}1centroid are compared to the true value \(centered cosine\): \(a\) exact attention with the MLP applied at the centroid; \(b\) additionally taking the value map at the centroid; \(c\) the full mean field of \([5](https://arxiv.org/html/2609.16382#S3.E5)\)\. Left: GPT\-2; middle: Gemma\-2; right: Qwen\-3\-14B\.The same decomposition explains the two Llama models, whose rollouts predict centroid evolution distinctly less well than the others \(full details in Appendix[J](https://arxiv.org/html/2609.16382#A10)\)\. The mean\-field*attention*prediction for Llama is as accurate as for the other models \(Figure[5](https://arxiv.org/html/2609.16382#S4.F5)\); the error is in the MLP\-at\-centroid step, whose Jensen gap is larger because many of Llama’s centroids, especially in early layers, sit where the MLP is more sharply curved\. Almost all of the Llama\-3\.2\-3B rollout error traces to poor MLP approximation at layer 1\. Because this is a property of the MLP rather than of the mean\-field attention, Llama’s centroid rollout requires a Jensen\-gap correction to be accurate\.

Finally, we note that the small error introduced via the value map suggests that, in general, representations are comparatively closely clustered around their type means\. In Appendix[C](https://arxiv.org/html/2609.16382#A3)we confirm this property quantitatively\. And we note that the covariance term is small in magnitude but not in importance: it contributes little to centroid prediction, yet is precisely what carries context\-specific computation \(§[6](https://arxiv.org/html/2609.16382#S6)\)\.

## 5The Mean Field as a Training Waypoint

We now show how mean\-field analysis can provide insight into the training dynamics of a model\. In this section we show that early in training, the model’s computation*reduces to*its mean field, then deviates as in\-context computation arrives\. Analyzing the resulting dynamics shows which changes in model weights drive early\-stage learning\.

To do so, we run Pythia\([Biderman et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib6)\), replacing every attention head’s contribution with the checkpoint’s context\-conditional mean field and measuring the resulting change in next\-token loss\. We perform these measurements across three scales of Pythia \(160m, 1\.4B, and 12B parameters\) at 19 checkpoints each, spanning each model’s full training run\. Because the mean field ignores token order, zero difference in loss would mean the model’s context\-dependence adds nothing beyond its corpus average\.

*Near\-zero difference in loss is in fact what we find early in training\.*After the mean\-field\-equivalent period, the difference rises sharply; the departure from the mean field and the arrival of in\-context computation coincide\.[Im et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib33)characterize the early weights as closed\-form compositions of corpus statistics; our mean\-field account is the computational side of the same picture, showing that the young model’s behavior reduces to its corpus mean field until in\-context computation emerges\.

### 5\.1Replacing Attention with its Mean Field

The experiment described above requires replacing the attention computed by the model for each specific contextXXwith a mean field approximation\. For that purpose we use the context\-conditional mean field of §[3\.3](https://arxiv.org/html/2609.16382#S3.SS3), Equation \([11](https://arxiv.org/html/2609.16382#S3.E11)\)\. Specifically, we substitute𝐳^s​Xℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}for each𝐳s​Xℓ​h\\mathbf\{z\}^\{\\ell h\}\_\{sX\}computed by the model, leaving the rest of the model’s computation unchanged\. We note an important property of𝐳^s​X\\hat\{\\mathbf\{z\}\}\_\{sX\}: it is unaffected by the ordering of the tokens in the contextXX\(𝐳^s​X\\hat\{\\mathbf\{z\}\}\_\{sX\}is ‘order\-free’\)\. If performing the substitution does not change the model’s loss, then what the model was computing was reproducible without considering token order\.

We computeA^s​tℓ​h\\hat\{A\}^\{\\ell h\}\_\{st\}and𝝁tℓ\\boldsymbol\{\\mu\}^\{\\ell\}\_\{t\}\(the ingredients of Equation[11](https://arxiv.org/html/2609.16382#S3.E11)\) from the same corpus as used in §[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)\. On a disjoint held\-out sample \(100,000 tokens\) we then run the model’s forward pass with every head’s contribution at every layer replaced by its per\-context contribution𝐳^s​Xℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}\. The substitution is made for the top 1000 types, for whichWWcan be estimated reliably from a 2M token pass of each model\. For Pythia, these 1000 types constitute 65% of positions in the corpus \(Appendix[E](https://arxiv.org/html/2609.16382#A5)\); remaining query types keep the model’s own computation\. In Appendix[M](https://arxiv.org/html/2609.16382#A13)we show parallel results for the topN=2000N=2000andN=4000N=4000types \(covering 72% and 80% of corpus positions, respectively\); the effects in this section are unchanged over this range ofNN\.

We report two diagnostics:Δ​L\\Delta LandDMFD\_\{\\mathrm\{MF\}\}\. The paired loss gapΔ​L=LMF−Lmodel\\Delta L=L\_\{\\mathrm\{MF\}\}\-L\_\{\\mathrm\{model\}\}measures how much prediction is contextual: the loss the model gives up when its attention computation is replaced by the mean field\. Writep\(⋅∣X\)p\(\\cdot\\mid X\)for the model’s next\-token distribution on contextXX, andpMF\(⋅∣X\)p\_\{\\mathrm\{MF\}\}\(\\cdot\\mid X\)for the distribution obtained when the attention output of every head at the query position is replaced by its context\-conditional mean\-field contribution \([11](https://arxiv.org/html/2609.16382#S3.E11)\), with the rest of the forward pass left intact\. Writingyyfor the token that actually followsXX,

Lmodel=−log⁡p⁡\(y∣X\),LMF=−log⁡pMF​\(y∣X\),Δ​L=LMF−Lmodel\.L\_\{\\mathrm\{model\}\}=\-\\log p\(y\\mid X\),\\qquad L\_\{\\mathrm\{MF\}\}=\-\\log p\_\{\\mathrm\{MF\}\}\(y\\mid X\),\\qquad\\Delta L=L\_\{\\mathrm\{MF\}\}\-L\_\{\\mathrm\{model\}\}\.\(12\)Δ​L\\Delta Lis the loss the model avoids by doing something that the mean field does not\.

Both losses are next\-token cross\-entropies averaged over the100,000100\{,\}000held\-out tokens above, so the singleΔ​L\\Delta Lwe report per checkpoint is a mean over that evaluation sample;DMFD\_\{\\mathrm\{MF\}\}likewise averages \([13](https://arxiv.org/html/2609.16382#S5.E13)\) over the sampled positions of the same text\. The*mean deviation*at a sampled position\(s,X\)\(s,X\):

Ds​X=meanℓ,h⁡\(1−cos⁡\(𝐳s​Xℓ​h,𝐳^s​Xℓ​h\)\),D\_\{sX\}=\\operatorname\{mean\}\_\{\\ell,h\}\\,\\bigl\(1\-\\cos\(\\mathbf\{z\}^\{\\ell h\}\_\{sX\},\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}\)\\bigr\),\(13\)measures how far the per\-head contributions themselves move from their corpus\-averaged counterparts\. The per\-checkpoint diagnostic we report is its average over sampled positions,DMF=means,X⁡Ds​XD\_\{\\mathrm\{MF\}\}=\\operatorname\{mean\}\_\{s,X\}\\,D\_\{sX\}101010DMFD\_\{\\mathrm\{MF\}\}is an un\-centered cosine distance between individual head contributions, distinct from the centered cosine over layers that Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)uses to score centroid prediction\.

### 5\.2Models Reduce to Their Mean Field, Then Depart from It

Figure 7:Pythia models side by side \(160m, 1\.4B, 12B, left to right\)\.Top:the model’s loss \(blue\) and the loss under the iterated mean field \(red\) across training steps \(log scale\); the yellow band marks the induction\-formation window\. The curves are visually indistinguishable through step10210^\{2\}and separate permanently shortly thereafter\.Bottom:the gapΔ​L\\Delta L\(green, left axis\) and the deviationDMFD\_\{\\mathrm\{MF\}\}\(purple, right axis\)\.Figure[7](https://arxiv.org/html/2609.16382#S5.F7)plots the experimental results for Pythia\-160m \(left\), Pythia\-1\.4B \(middle\), and Pythia\-12B \(right\)\. The top row of plots depicts the next\-token loss actually incurred by the model \(blue\) and by the mean\-field replacement \(red\), as training elapses from left to right\. The bottom two plots depict our corresponding measures over this experiment: the gapΔ​L\\Delta L\(green\) between the two upper curves and the deviationDMFD\_\{\\mathrm\{MF\}\}\(purple, right axis\)\. The yellow shading shows the induction band \(steps 1000\-2000\), the window in which induction heads are reported to form in Pythia\([Tigges et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib65)\)\.111111According to[Tigges et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib65), roughly22–55billion tokens, or steps954954–23842384at Pythia’s2\.12\.1M tokens per step\.

#### In early training, neither measure moves\.

Up through training step10210^\{2\}, both bottom panels showΔ​L\\Delta Lnegligible andDMFD\_\{\\mathrm\{MF\}\}also near its floor\. These measures show that departure from the mean field has not yet occurred: replacing the entire attention computation with the iterated mean field costs nothing in terms of loss\. TheΔ​L\\Delta Lgap is within a few millinats of zero, and the attention contributions of the heads sit, on average, close to their corpus means\. Yet the*losses themselves are falling fast over this stretch,*by more than three nats, so this regime by no means experiences an absence of learning\. However, what the model has learned so far depends only on which tokens are present, not their order, which is exactly what the mean field captures\. §[5\.3](https://arxiv.org/html/2609.16382#S5.SS3)gives the mechanistic reason\.

#### Next, across the induction window, both measures rise in parallel\.

Starting around step 128 and continuing through the induction band,Δ​L\\Delta LandDMFD\_\{\\mathrm\{MF\}\}both rise sharply\. Of note is that the rise occurs at the same token count in all three models despite a75×75\\timesdifference in parameters \(160m to 12B\), an indication that the transition is set by data seen rather than model size\. Departure from the mean field here reflects context\-dependent computation the mean field does not represent, a topic §[6](https://arxiv.org/html/2609.16382#S6)takes up in detail\.

#### After the induction window, the two measures diverge\.

After the induction band, the two measures behave differently: the per\-head deviationDMFD\_\{\\mathrm\{MF\}\}plateaus in the 160m case and eases down from a mid\-training peak in the 1\.4B and 12B cases \(most sharply at 12B\), while the loss cost of imposing the mean fieldΔ​L\\Delta Lcontinues to rise\. The top panels show why: the mean\-field loss stops tracking the model loss\. Imposing the mean field now actively harms the model, not because the mean field intrinsically became worse, but because the model has learned to rely on something the mean field does not supply\. Comparing the final performance across the three models, their true losses are smaller for larger models, while their mean\-field losses remain quite similar\. This shows that the performance benefits obtained by larger models are related to context\-dependent processing\.

#### Mean\-field loss augments the weight\-space account\.

[Im et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib33)show early attention weights to be closed\-form compositions of corpus statistics \(verified on Pythia\-1\.4B at the earliest checkpoints\)\. The mean\-field results here are the computational counterpart, but they extend further because similar weights need not mean similar behavior\. And the correspondence of the model with its mean field does not fade gradually: functionally it holds almost exactly, then fails over a narrow interval, so the drift seen in weight space corresponds to a specific event, the arrival of in\-context learning\.

#### The effect replicates in OLMo\-2\.

The only other model family in which training checkpoints exist in early \(pre\-1B\-token\) stages is OLMo\([OLMo Team, 2024](https://arxiv.org/html/2609.16382#bib.bib48)\)\. Even in that model, there are limited data points available in the key region\. However, there is enough data to confirm that the same effect appears to take place in OLMo\-2, at roughly the same point in the training process\. Details are in Appendix[L](https://arxiv.org/html/2609.16382#A12)\(Figure[22](https://arxiv.org/html/2609.16382#A12.F22)\)\.

### 5\.3Kernel Dynamics Across Training

For a deeper understanding of why the model reduces to its mean field and then departs, we study the evolution of the mean field kernelWW\([8](https://arxiv.org/html/2609.16382#S3.E8)\)\. Figure[8](https://arxiv.org/html/2609.16382#S5.F8)shows how muchWWchanges across training, measured using linear CKA, averaged over heads\. \(Estimation details in Appendix[N](https://arxiv.org/html/2609.16382#A14)\)\. The left three panels give the kernel similarity between every pair of checkpoints; the right panel gives the similarity between adjacent checkpoints, depicting the rate at whichWWchanges\.

Three regimes appear in all three models, aligned with those in §[5\.2](https://arxiv.org/html/2609.16382#S5.SS2)\. Up through step 128 the kernel barely moves: adjacent\-checkpoint similarity sits at 1\.0, even as the model’s loss falls steeply over the same checkpoints \(Figure[7](https://arxiv.org/html/2609.16382#S5.F7)\)\.*Learning in this regime is in the values, MLPs, and embeddings, not the attention pattern\.*After step 128 the kernel reorganizes, at the same point where the two departure measures begin to rise: adjacent\-checkpoint similarity falls to its minimum leading up to the induction band, then recovers and stays high afterward\. So mean\-field attentionWWis not arranged ahead of in\-context learning: it holds flat early, reorganizes rapidly leading up to and across the induction band, and stabilizes afterward\. The reorganization is much more pronounced for larger models\.

The fact that mean\-field attention is not changing much in the early stage of learning suggests that the attention heads are still behaving as if randomly initialized\. To confirm this, we measure how evenly each head divides its attention among the types present\. Early in training a type draws attention in proportion to how often it appears, and not otherwise: at initialization a head’s attention varies by only about a factor of1\.51\.5across types, which is what random projections give\. By the end of training the same heads vary by more than a factor of2020\. Heads begin to attend selectively when the kernel reorganizes, and per\-head spread increases through the rest of training\. Heads also differ sharply: some develop large spread while others barely change from their near\-uniform initialization\. Appendix[N](https://arxiv.org/html/2609.16382#A14)defines the row\-spread measure, tabulates the raw values, and plots its trajectory across training for all three models \(Figure[25](https://arxiv.org/html/2609.16382#A14.F25)\)\.

![Refer to caption](https://arxiv.org/html/2609.16382v1/w_cka_compare.png)Figure 8:How much the kernel changes across training, all three models, measured as similarity between checkpoints: a value of 1\.0 is no change\.Left three panels:kernel similarity between every pair of checkpoints \(160m, 1\.4B, 12B\), averaged over heads\.Right:adjacent\-checkpoint kernel similarity, with the induction window shaded for reference\.#### When the mean field holds in practice\.

The measurements in this section support the following conclusion: early in training a head’s corpus\-mean attention to a type is fixed by how often the type occurs, independent of the values those occurrences carry\. Attention and value are therefore uncorrelated across types, which is precisely Assumption[3\.2](https://arxiv.org/html/2609.16382#S3.SS2.SSS0.Px1), attention\-value independence for the corpus mean field\. The assumption thus holds closely early in training as an empirical fact about Pythia \(not a simplification\) and stops holding once the kernel reorganizes and heads acquire contextual sensitivity\.

## 6In\-Context Learning and Contextual Computation

So far, §[5](https://arxiv.org/html/2609.16382#S5)has shown that deviation from the mean field identifies*when*, during training, the model begins doing context\-specific work that the mean field cannot represent\. We now apply this principle more broadly, to locate*where*that extra computation lives: in which heads, on which prompts, and at which tokens of a prompt\. We take the context\-conditional mean field as a*null model for contextual processing*\. Because a head’s mean\-field prediction models what its output would be in the absence of contextual computation, the deviation from it estimates the contextual computation present\. This makes it a*principled*null: the baseline it sets is the model’s own average behavior, not an external reference\.

Thus, even though to this point in the paper, the mean\-field residual was considered a source of error, in this section we treat the residual as a useful signal\. The residual has two sources: the correlation between where the head attends and the values it moves, and the contextualization of those values \(the attended tokens carrying content specific to this context rather than their corpus average\)\. We make these two concepts more mathematically precise in \([15](https://arxiv.org/html/2609.16382#S6.E15)\) below\. Each makes a distinct contribution to the residual\.

Results in this section are based on the deviation of a head’s actual output𝐳s​Xℓ​h\\mathbf\{z\}^\{\\ell h\}\_\{sX\}from the context\-conditional prediction𝐳^s​Xℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}of \([11](https://arxiv.org/html/2609.16382#S3.E11)\)\. We compute two measures: the*context\-dependent deviation*of head\(ℓ,h\)\(\\ell,h\)onXX, and its per\-context average \(over all heads\):

Ds​Xℓ​h=1−cos⁡\(𝐳s​Xℓ​h,𝐳^s​Xℓ​h\),DMF​\(X\)=meanℓ,h⁡Ds​Xℓ​h\.D^\{\\ell h\}\_\{sX\}=1\-\\cos\\\!\\big\(\\mathbf\{z\}^\{\\ell h\}\_\{sX\},\\,\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}\\big\),\\qquad D\_\{\\mathrm\{MF\}\}\(X\)=\\operatorname\{mean\}\_\{\\ell,h\}\\,D^\{\\ell h\}\_\{sX\}\.\(14\)
Note thatssdoes not appear in the average to produceDMFD\_\{\\mathrm\{MF\}\}in \([14](https://arxiv.org/html/2609.16382#S6.E14)\), asXXis associated with a unique query typess\.

In this section, we will show the following:

Mean field deviation rises with in\-context learning\.We show three experiments supporting the strong association of mean\-field deviation with ICL \(§[6\.1](https://arxiv.org/html/2609.16382#S6.SS1)\)\.

Deviation decomposes additively into two contributors\.For each context, the deviation splits exactly into an attention\-pattern covariance and a contextualization of the attended values, each of which carries information about ICL in both GPT\-2 and Gemma\-2\-2B \(§[6\.2](https://arxiv.org/html/2609.16382#S6.SS2)\)\. Averaged over contexts per\-head, the split characterizes every head’s deviation across those two axes, and we find that deviation in general involves both axes\.

Mean field deviation organizes an unsupervised survey of head activity\.Ranking GPT\-2 heads by mean field deviation on generic text surfaces the known in\-context heads; further analyses confirm the documented heads’ roles and characterize previously undocumented GPT\-2 heads\. \(Method in Appendix[P\.1](https://arxiv.org/html/2609.16382#A16.SS1), findings in §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3)\.\)

Mean field deviation isolates function vectors\.Aggregating the mean field residual on a task set, a head’s deviation is causally a function vector, and the corpus null lets it be extracted from as little as one behavior\-exhibiting prompt \(§[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)\)\.

### 6\.1Mean\-Field Deviation Rises with In\-Context Learning

Our first set of experiments shows that in\-context learning has a strong effect on mean\-field deviation\. Note that we do not claim that this goes both ways; mean\-field deviation can be large for other well\-understood reasons; we study examples in §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3)\.

We present three experiments intended to illustrate the relationship between mean\-field deviation and in\-context learning at different scales\. Experiment 1 asks, within a single prompt where ICL is present, whether deviation localizes ICL to the token\(s\) where a known induction head fires\. Experiment 2 asks, across whole prompts, whether deviation succeeds at separating prompts by the in\-context learning they do, from induction down to random text\. And Experiment 3 asks, across training at the corpus level, whether deviation emerges during training at the point where induction is known to arise\.

#### Experiment 1: Mean\-field deviation rises with induction as defined by[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49)\.

In order to study a controlled in\-context signal, we adapt the induction setup of[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49): a real OpenWebText chunkccof128128tokens is presented as\[c;c\]\[c;c\]\(BOS\-prepended\), so that in the second copy every token has a distant earlier occurrence and in\-context \(induction\) prediction becomes available\. While[Olsson et al\.](https://arxiv.org/html/2609.16382#bib.bib49)score induction heads on repeated*random*tokens, to isolate the mechanism from content, we use real text so that the loss drop at the copy boundary is a quantity the model would actually realize on corpus data\. \(Our Experiment 3 will use their random\-token form\.\) We treat each sequence as 128 contexts, by considering each token as a queryss, and we measure the context\-conditional mean\-field deviationDMFD\_\{\\mathrm\{MF\}\}\([14](https://arxiv.org/html/2609.16382#S6.E14)\) for that position\. Over500500such sequences we record, per query position, the mean next\-token loss and the meanDMFD\_\{\\mathrm\{MF\}\}; and, for each pair of matched copy\-1 / copy\-2 positions of the same token, an*in\-context loss reduction*\(loss⁡\(copy​1\)−loss⁡\(copy​2\)\\mathrm\{loss\}\(\{\\text\{copy \}1\}\)\-\\mathrm\{loss\}\(\{\\text\{copy \}2\)\}\) against the mean deviation \(DMF​\(copy​2\)−DMF​\(copy​1\)D\_\{\\mathrm\{MF\}\}\(\{\\text\{copy \}2\}\)\-D\_\{\\mathrm\{MF\}\}\(\{\\text\{copy \}1\}\)\)\.

Figure[9](https://arxiv.org/html/2609.16382#S6.F9)\(left\) shows that at the copy boundary the next\-token loss drops sharply while the mean\-field deviation sharply rises\. Among GPT\-2 heads, deviation rise is strongest in L5H1, a known induction head \(Figure[9](https://arxiv.org/html/2609.16382#S6.F9), middle\)\. The right side of the figure shows that*mean\-field deviation has a sensitive response to induction*– the mean\-field deviation increases monotonically with induction as measured by in\-context loss reduction\.

Figure 9:Instance\-level context\-conditional mean\-field deviation on GPT\-2 small\.Left:per\-query\-position next\-token loss \(red\) and all\-head mean\-field deviationDMFD\_\{\\mathrm\{MF\}\}\(blue\) on repeated text\[c;c\]\[c;c\]; the dashed line marks the copy boundary at position128128, separating the two parts of the prompt\.Middle:Ds​Xℓ​hD^\{\\ell h\}\_\{sX\}averaged over promptsXXfor the single head that rises most across the boundary, GPT\-2 L5H1, a known induction head\.Right:per\-token in\-context loss reduction \(losscopy​1−losscopy​2\\mathrm\{loss\}\_\{\\text\{copy \}1\}\-\\mathrm\{loss\}\_\{\\text\{copy \}2\}\) binned by the induced deviation \(DMF​\(copy​2\)−DMF​\(copy​1\)D\_\{\\mathrm\{MF\}\}\(\{\\text\{copy \}2\}\)\-D\_\{\\mathrm\{MF\}\}\(\{\\text\{copy \}1\}\)\); error bars are standard errors of the mean\.
#### Experiment 2: Mean\-field deviation tracks the amount of ICL across prompt classes\.

Next, we ask whether mean\-field deviation distinguishes different types of ICL\. We study four prompt classes to vary the kind of ICL expected to take place\.

- •induction\(planted ICL\) Each prompt consists of 12 random tokens, followed by a \(random\-length\) repeated subsequence of those tokens, and the query is the last token in the repeated subsequence\.
- •few\-shot\(planted ICL\) Each prompt gives fiveinput:outputexemplars of one task \(e\.g\.hot:cold\), followed by a held\-out query input and a colon\. The correct completion is determined only by the rule the exemplars establish, so predicting it requires inferring the task from the context\. We use five tasks: antonym, capital, past tense, plural, and capitalization\.
- •natural\(background ICL\) Prompts are sequences from OpenWebText, whose last token is one of the tracked types \(so that a type specific mean\-field prediction can be made\)\. Each prompt is taken from a different OWT document\. This represents the ICL occurring naturally in typical text\.
- •random\(matched non\-ICL control\) Prompts are the same random sequences and query tokens as used in ‘induction’ but the query is restricted to only be in the first copy, so the next token is unpredictable\. This makes it a matched\-control for the ‘induction’ case\.

For each prompt we computeΔ​L\\Delta L\([12](https://arxiv.org/html/2609.16382#S5.E12)\) and the mean\-field deviationDMFD\_\{\\mathrm\{MF\}\}\. Additionally, we shuffle the context positions using a random permutationπ\\pi, and measure the resulting change in model loss \(Δ​Lperm=Lmodel​\(π​X\)−Lmodel​\(X\)\\Delta L\_\{\\text\{perm\}\}=L\_\{\\text\{model\}\}\(\\pi X\)\-L\_\{\\text\{model\}\}\(X\)\)\. To verify the mean\-field’s order\-invariance we measure relative change in the mean\-field contribution \(Δ​𝐳^perm=‖𝐳^​\(π​X\)−𝐳^​\(X\)‖/‖𝐳^​\(X\)‖\\Delta\\hat\{\\mathbf\{z\}\}\_\{\\text\{perm\}\}=\\\|\\hat\{\\mathbf\{z\}\}\(\\pi X\)\-\\hat\{\\mathbf\{z\}\}\(X\)\\\|/\\\|\\hat\{\\mathbf\{z\}\}\(X\)\\\|\)\.

Table[2](https://arxiv.org/html/2609.16382#S6.T2)reports statistics when attention contribution at every head and layer of the model is replaced with the context\-conditional mean field \([11](https://arxiv.org/html/2609.16382#S3.E11)\) for each context\. In the table,Δ​LMF\\Delta L\_\{\\mathrm\{MF\}\}is the loss difference when attention is replaced by the mean field, andΔ​Lperm\\Delta L\_\{\\mathrm\{perm\}\}is the loss difference when the context is randomized\. Both are computed per prompt from \([12](https://arxiv.org/html/2609.16382#S5.E12)\) and reported as the mean over thennprompts of the class,nnas given in the table\.121212Loss increases are meaningful, because the models are able to perform the tasks\. Gemma\-2’s top\-1 accuracy is≥98%\\geq 98\\%on all induction and few\-shot prompts; GPT\-2’s is95%95\\%on induction and73%73\\%on few\-shot, the lower figure consistent with its weaker few\-shot behavior in Figure[10](https://arxiv.org/html/2609.16382#S6.F10)\.Figure[10](https://arxiv.org/html/2609.16382#S6.F10)shows detailed results for every prompt in the dataset, plottingΔ​Lperm\\Delta L\_\{\\text\{perm\}\}against mean\-field deviation\.

The table and figure show that larger mean\-field deviations correspond to the context categories expected to have larger ICL components, and to larger changes in loss when the context is permuted\. Both substituting the mean field and permuting the context are catastrophic when the next token is in\-context\-determined \(induction, few\-shot\) and harmless when it is not \(random\)\. These facts all suggest that mean\-field deviation is responsive to the presence of ICL\. We note that this result is about class differences\.DMFD\_\{\\mathrm\{MF\}\}orders the classes as their permutation lossΔ​Lperm\\Delta L\_\{\\text\{perm\}\}does, a separation driven by the designed differences between classes and not by any within\-class relationship\. We therefore report it as a standardized class separation \(Appendix[O](https://arxiv.org/html/2609.16382#A15)\), not as a correlation over the pooled prompts\. The random control’sΔ​Lperm\\Delta L\_\{\\text\{perm\}\}is zero in expectation, its context being exchangeable by construction, and the tabled values are consistent with zero \(p=0\.09p=0\.09/0\.290\.29\)\. The mean\-field read is essentially unchanged under permutation \(Δ​𝐳^perm∼10−7\\Delta\\hat\{\\mathbf\{z\}\}\_\{\\text\{perm\}\}\\sim 10^\{\-7\}\), confirming it supplies an order\-free null\.

Table 2:Per prompt class: mean\-field deviationDMFD\_\{\\mathrm\{MF\}\}; loss increaseΔ​L\\Delta Lfrom replacing attention by the mean field \(all layers\); model loss change under context permutationΔ​Lperm\\Delta L\_\{\\text\{perm\}\}; and permutation sensitivityΔ​𝐳^perm\\Delta\\hat\{\\mathbf\{z\}\}\_\{\\text\{perm\}\}\(maximum over all228228prompts\)\. All columns are averaged over heads and over the prompts of the class,nnper class as shown\.Figure 10:Mean\-field deviation and in\-context learning: per\-prompt permutation loss changeΔ​Lperm\\Delta L\_\{\\mathrm\{perm\}\}against mean\-field deviation, all228228prompts\. The prompt classes separate along both axes \(shown in §[O](https://arxiv.org/html/2609.16382#A15)\)\.
#### Experiment 3: Mean\-field deviation emerges during training simultaneously with ICL\.

§[5](https://arxiv.org/html/2609.16382#S5)showed that the deviation, summed over heads, rises at the induction transition\. Here we look head by head: does the deviation emerge in*precisely the heads*that become induction heads, and at the same checkpoint? We test this on nineteen Pythia\-160m training checkpoints \(steps00to143,000143\{,\}000\), spanning the induction\-formation transition that[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49)and[Tigges et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib65)place shortly after2×1092\\times 10^\{9\}tokens\. At each checkpoint we measure, for all144144heads, the per\-head mean\-field deviationS¯ℓ​h\\bar\{S\}^\{\\ell h\}, the average over query types of the per\-type deviation \(§[6\.3](https://arxiv.org/html/2609.16382#S6.SS3.SSS0.Px1), \([17](https://arxiv.org/html/2609.16382#S6.E17)\)\), from that checkpoint’s own corpus pass, and a standard behavioral induction score\.

The induction score is measured independently of the mean field, from the head’s attention pattern alone: we feed batches of*randomly*generated token sequences repeated back\-to\-back,\[BOS;c;c\]\[\\mathrm\{BOS\};\\,c;\\,c\], and score each head by the average attention a token in the second copy places on the position immediately after that token’s first occurrence — the induction target\([Olsson et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib49)\)\. In contrast to Experiment 1, this probe uses*random*repeated tokens \(rather than repeated text from OpenWebText\)\. The reason for this difference is to isolate the induction circuit itself and avoid any content\-based regularities\. Figure[11](https://arxiv.org/html/2609.16382#S6.F11)plots the cross\-head Spearman correlation betweenS¯ℓ​h\\bar\{S\}^\{\\ell h\}and the induction score across the144144heads over training\.S¯ℓ​h\\bar\{S\}^\{\\ell h\}is defined in Equation[17](https://arxiv.org/html/2609.16382#S6.E17)below, and represents the average over all types in the corpus of a head’s deviation\. Among the ten checkpoints before the transition \(through step 256\), no correlations are significantly different from zero \(allp≥0\.16p\\geq 0\.16\)\. It rises at step512512\(0\.370\.37\), peaks at the documented transition \(0\.710\.71\), and then declines somewhat as later, non\-induction contextual heads accumulate and dilute the pure induction signal\. Mean\-field deviation thus appears in the heads that become induction heads, at the checkpoint they do — further evidence that the deviation rises with the onset of in\-context computation\.

Figure 11:Correlation between mean\-field deviationS¯ℓ​h\\bar\{S\}^\{\\ell h\}\(see Eq\.[17](https://arxiv.org/html/2609.16382#S6.E17)\) and behavioral induction score across training, Pythia\-160m, over the nineteen checkpoints of Figure[7](https://arxiv.org/html/2609.16382#S5.F7)\. The yellow band marks the induction\-formation window \(steps10001000–20002000,≈2×109\{\\approx\}2\\times 10^\{9\}tokens\), as in that figure\.

### 6\.2Mean Field Deviation Combines Two Contributors

In order to probe the relationship between mean\-field deviation and in\-context learning, we decompose the deviation into its two context\-dependent components, and consider how each relates to contextual processing\.

#### Decomposing the deviation: covariance and contextualization terms\.

The deviation \([14](https://arxiv.org/html/2609.16382#S6.E14)\) compares the actual contribution in a given contextXXto the mean field approximation𝐳^X=∑tA^s​t​X​𝝁t\\hat\{\\mathbf\{z\}\}\_\{X\}=\\sum\_\{t\}\\hat\{A\}\_\{stX\}\\boldsymbol\{\\mu\}\_\{t\}, withA^s​t​X\\hat\{A\}\_\{stX\}the context\-conditional kernel \([10](https://arxiv.org/html/2609.16382#S3.E10)\)\.

The difference splits into two additive terms:

𝐳X−𝐳^X=∑t\(As​t​X−A^s​t​X\)​𝐯¯t​X⏟\(a\) covariance term\+∑tA^s​t​X​\(𝐯¯t​X−𝝁t\)\.⏟\(b\) contextualization term\\mathbf\{z\}\_\{X\}\-\\hat\{\\mathbf\{z\}\}\_\{X\}=\\underbrace\{\\textstyle\\sum\_\{t\}\(A\_\{stX\}\-\\hat\{A\}\_\{stX\}\)\\,\\overline\{\\mathbf\{v\}\}\_\{tX\}\}\_\{\\textbf\{\(a\) covariance term\}\}\+\\underbrace\{\\textstyle\\sum\_\{t\}\\hat\{A\}\_\{stX\}\\,\\big\(\\overline\{\\mathbf\{v\}\}\_\{tX\}\-\\boldsymbol\{\\mu\}\_\{t\}\\big\)\.\}\_\{\\textbf\{\(b\) contextualization term\}\}\(15\)
Term \(a\) is the part of the deviation due to variation in attention: each summand is zero ifAs​t​XA\_\{stX\}, the attention the head actually places on typett, matches the mean\-field kernelA^s​t​X\\hat\{A\}\_\{stX\}\. Because the kernel is approximately unbiased for the attention over the ensemble of contexts sharing the query type \(𝔼s​t​\[A^s​t\]≈𝔼s​t​\[As​t\]\\mathbb\{E\}\_\{st\}\[\\hat\{A\}\_\{st\}\]\\approx\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\), averaging term \(a\) over that ensemble gives the residual attention–value*covariance*∑tCovs​t⁡\(As​t−A^s​t,𝐯¯t\)\\sum\_\{t\}\\operatorname\{Cov\}\_\{st\}\\\!\\big\(A\_\{st\}\-\\hat\{A\}\_\{st\},\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\)\(full derivation is in Appendix[B\.5](https://arxiv.org/html/2609.16382#A2.SS5)\.\) Term \(b\) is the part of the deviation due to variation in the values, the*contextualization*of these values: each summand is zero exactly when𝐯¯t​X\\overline\{\\mathbf\{v\}\}\_\{tX\}, the value typettcarries in this context, matches its corpus type\-mean𝝁t\\boldsymbol\{\\mu\}\_\{t\}\.131313Note that because𝐳^\\hat\{\\mathbf\{z\}\}uses the measured type\-mean𝝁t\\boldsymbol\{\\mu\}\_\{t\}rather than the centroid valuev⁡\(𝐜tℓ\)v\(\\mathbf\{c\}^\{\\ell\}\_\{t\}\), the value\-map Jensen gap does not appear in the deviation at all\. This is deliberate: that gap is a static property of the value map’s curvature at the centroid, independent ofXX, so it cannot carry in\-context information — and in any case it is small in all of our experiments \(Appendix[C](https://arxiv.org/html/2609.16382#A3)\)\.

These two terms describe different*mechanisms*behind contextual processing\. In short, contextual processing can arise either because \(a\) the attention provided to certain tokens is driven to move specific values into the query, or \(b\) because the tokens attended to in the context are highly contextualized\. What \([15](https://arxiv.org/html/2609.16382#S6.E15)\) allows us to do is to separate these two sources into*additive*terms that can be measured separately\.

#### How do the deviation components vary across tasks?

Since the two terms describe different mechanisms, we measure each separately on the prompts of Table[2](https://arxiv.org/html/2609.16382#S6.T2), and ask how they vary across tasks\. Each contribution is formed teacher\-forced, from the model’s true residuals at layerℓ\\elland the measured corpus value means𝝁tℓ\\boldsymbol\{\\mu\}^\{\\ell\}\_\{t\}\. We summarize each per context as a scale\-free*metric*\. Writing𝐚\\mathbf\{a\}and𝐛\\mathbf\{b\}for the two terms of \([15](https://arxiv.org/html/2609.16382#S6.E15)\), so that𝐚\+𝐛=𝐳−𝐳^\\mathbf\{a\}\+\\mathbf\{b\}=\\mathbf\{z\}\-\\hat\{\\mathbf\{z\}\}exactly, let𝐞=𝐳−𝐳^\\mathbf\{e\}=\\mathbf\{z\}\-\\hat\{\\mathbf\{z\}\}denote the total error and𝐞⟂\\mathbf\{e\}\_\{\\perp\}be its component that is orthogonal to𝐳^\\hat\{\\mathbf\{z\}\}, ie,𝐞⟂=𝐞−𝐞⋅𝐳^‖𝐳^‖2​𝐳^\\mathbf\{e\}\_\{\\perp\}=\\mathbf\{e\}\-\\frac\{\\mathbf\{e\}\\cdot\\hat\{\\mathbf\{z\}\}\}\{\\\|\\hat\{\\mathbf\{z\}\}\\\|^\{2\}\}\\hat\{\\mathbf\{z\}\}\. We measure each term’s contribution to𝐞⟂\\mathbf\{e\}\_\{\\perp\}and normalize:

σcov=𝐚⋅𝐞⟂∥𝐞⟂∥2,σctx=𝐛⋅𝐞⟂∥𝐞⟂∥2,σcov\+σctx=𝐞⋅𝐞⟂∥𝐞⟂∥2=1,\\sigma\_\{\\mathrm\{cov\}\}=\\frac\{\\mathbf\{a\}\\cdot\\mathbf\{e\}\_\{\\perp\}\}\{\\lVert\\mathbf\{e\}\_\{\\perp\}\\rVert^\{2\}\},\\qquad\\sigma\_\{\\mathrm\{ctx\}\}=\\frac\{\\mathbf\{b\}\\cdot\\mathbf\{e\}\_\{\\perp\}\}\{\\lVert\\mathbf\{e\}\_\{\\perp\}\\rVert^\{2\}\},\\qquad\\sigma\_\{\\mathrm\{cov\}\}\+\\sigma\_\{\\mathrm\{ctx\}\}=\\frac\{\\mathbf\{e\}\\cdot\\mathbf\{e\}\_\{\\perp\}\}\{\\lVert\\mathbf\{e\}\_\{\\perp\}\\rVert^\{2\}\}=1,The metrics141414When𝐞⟂=𝟎\\mathbf\{e\}\_\{\\perp\}=\\mathbf\{0\},𝐳\\mathbf\{z\}is collinear with𝐳^\\hat\{\\mathbf\{z\}\}, so the deviation is either exactly 0 \(aligned\) or 2 \(anti\-aligned\), and the perpendicular shares are undefined\. This case does not occur in our measurements\.are

Mcov​\(X\)=meanℓ,h⁡σcovℓ​h​Ds​Xℓ​h,Mctx​\(X\)=meanℓ,h⁡σctxℓ​h​Ds​Xℓ​h\.M\_\{\\mathrm\{cov\}\}\(X\)=\\operatorname\{mean\}\_\{\\ell,h\}\\,\\sigma\_\{\\mathrm\{cov\}\}^\{\\ell h\}\\,D^\{\\ell h\}\_\{sX\},\\qquad M\_\{\\mathrm\{ctx\}\}\(X\)=\\operatorname\{mean\}\_\{\\ell,h\}\\,\\sigma\_\{\\mathrm\{ctx\}\}^\{\\ell h\}\\,D^\{\\ell h\}\_\{sX\}\.\(16\)Becauseσcov\+σctx=1\\sigma\_\{\\mathrm\{cov\}\}\+\\sigma\_\{\\mathrm\{ctx\}\}=1per head, the metrics sum to the deviationDs​Xℓ​hD^\{\\ell h\}\_\{sX\}of \([14](https://arxiv.org/html/2609.16382#S6.E14)\) exactly\. Note that the shares are allocated along𝐞⟂\\mathbf\{e\}\_\{\\perp\}only: the component of𝐞\\mathbf\{e\}parallel to𝐳^\\hat\{\\mathbf\{z\}\}rescales the contribution without rotating it, and the cosine\-based deviation is insensitive to that rescaling, so it is deliberately excluded from the split\. The shares are signed, so one mechanism can partially cancel the other\.

Table 3:Table[2](https://arxiv.org/html/2609.16382#S6.T2)augmented with the covariance and contextualization metricsMcov,MctxM\_\{\\mathrm\{cov\}\},M\_\{\\mathrm\{ctx\}\}of \([16](https://arxiv.org/html/2609.16382#S6.E16)\), which sum toDMFD\_\{\\mathrm\{MF\}\}\.
#### Results\.

Table[3](https://arxiv.org/html/2609.16382#S6.T3)shows how the two components of mean\-field deviation vary across experiments: the covariance termMcovM\_\{\\mathrm\{cov\}\}\(roughly, the ‘unusualness of attention’\), and the contextualization termMctxM\_\{\\mathrm\{ctx\}\}\(roughly, the ‘unusualness of representations’\)\. The covariance term is consistently the larger part of mean\-field deviation\. But a large covariance term is not by itself an ICL signal: the random control shows highMcovM\_\{\\mathrm\{cov\}\}yet the lowestMctxM\_\{\\mathrm\{ctx\}\}of any class and near\-zeroΔ​Lperm\\Delta L\_\{\\text\{perm\}\}\. This is because its random tokens are out\-of\-distribution for the corpus statisticsWs​tW\_\{st\}thatA^\\hat\{A\}is built from\. Among the genuine ICL classes, the dominant mechanism is model\-dependent: in Gemma\-2 the attended tokens are more contextualized than in GPT\-2 \(comparing metrics to random\), while GPT\-2’s ICL deviation is driven mainly by unusual attention\. Appendix[O](https://arxiv.org/html/2609.16382#A15)separates the two contributions more finely, putting them on a common scale \(Table[16](https://arxiv.org/html/2609.16382#A15.T16)\) as the separation of each class from the random control\. This confirms the model\-dependent dominance, and shows the deviation ranks the designed conditions rather than individual prompts within them\.

Overall, the picture that emerges is that ICL, as characterized by mean\-field deviation, is not purely a function of covariance or contextualization alone, but in general involves both mechanisms\.

### 6\.3Mean Field Deviation Recovers Contextual Heads and Circuitry

We next show how mean\-field deviation can be used to identify heads that carry contextual computation\. As a first cut, we rank all heads by their context\-conditional deviation, averaged across query types\. We report our findings on GPT\-2 throughout, because the circuit literature has cataloged it in enough detail to inform our readings\.151515The head survey is repeated on Gemma\-2\-2B in Appendix[P\.5](https://arxiv.org/html/2609.16382#A16.SS5)\. Some of the same machinery appears, but with no comparable catalog to check it against, our survey can only test the reproducibility of our methods, not whether its readings agree with an independent account\.A number of heads of GPT\-2 that the circuit literature has documented appear near the top of the rankings, and so do other contextual heads those methods miss, including positional and content\-moving heads which we newly identify\. We additionally place heads onto theMcovM\_\{\\mathrm\{cov\}\}/MctxM\_\{\\mathrm\{ctx\}\}plane of §[6\.2](https://arxiv.org/html/2609.16382#S6.SS2), which visually localizes both high and low\-deviation groups of heads that share a common documented role\. Finally, we investigate how known circuit pairs \(e\.g\., a previous\-token head with an induction head\) align with respect to these measures\.

#### Ranking heads by corpus deviation\.

We rank all 144 GPT\-2 heads by their*corpus deviation*: how far the head’s corpus\-average behavior departs from the mean\-field account, at the level of query types\. For query typess, the head’s measured contribution is𝐳¯sℓ​h=𝔼s​\[∑tAs​t​𝐯¯t\]\\overline\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}=\\mathbb\{E\}\_\{s\}\\big\[\\sum\_\{t\}A\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\]\(Eq\.[3](https://arxiv.org/html/2609.16382#S3.E3)\), and its context\-averaged mean\-field prediction is𝐳^¯sℓ​h=𝔼s​\[𝐳^s​Xℓ​h\]=∑t𝔼s​\[A^s​tℓ​h\]​𝝁t,\\overline\{\\hat\{\\mathbf\{z\}\}\}^\{\\ell h\}\_\{s\}\\;=\\;\\mathbb\{E\}\_\{s\}\[\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}\]=\\textstyle\\sum\_\{t\}\\mathbb\{E\}\_\{s\}\\big\[\\hat\{A\}^\{\\ell h\}\_\{st\}\\big\]\\,\\boldsymbol\{\\mu\}\_\{t\},which uses the context\-averaged context\-conditional kernel of \([10](https://arxiv.org/html/2609.16382#S3.E10)\)\.161616Both quantities are accumulated in one corpus pass over 2M OpenWebText tokens, the same pass that measures the kernels of §[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)\. Tokens outside the top\-1000 contribute through BOS and “other” buckets in both terms\.

The per\-type deviation and the head’s corpus score are

Ssℓ​h=1−cosc⁡\(𝐳¯sℓ​h,𝐳^¯sℓ​h\),S¯ℓ​h=means⁡Ssℓ​h,S^\{\\ell h\}\_\{s\}=1\-\\cos\_\{c\}\\\!\\big\(\\overline\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\},\\,\\overline\{\\hat\{\\mathbf\{z\}\}\}^\{\\ell h\}\_\{s\}\\big\),\\qquad\\bar\{S\}^\{\\ell h\}=\\operatorname\{mean\}\_\{s\}\\,S^\{\\ell h\}\_\{s\},\(17\)wherecosc\\cos\_\{c\}is the centered cosine \(the mean over query types is removed from both arguments, so the score measures how the head’s output*varies*with the query type, not a shared offset\) and the mean is over the types with at least 50 corpus occurrences\. It is centered because across query types, the contributions share a large common offset that would dilute the deviation\. Note that it is not possible to do such a centering on a single context\.

Note thatS¯ℓ​h\\bar\{S\}^\{\\ell h\}is based on mean field deviation likeDℓ​h,D^\{\\ell h\},but unlikeDℓ​hD^\{\\ell h\}, the head’s contribution is averaged over contexts*before*the deviation is taken, signified by the bar on𝐳^¯sℓ​h\\overline\{\\hat\{\\mathbf\{z\}\}\}^\{\\ell h\}\_\{s\}\. Per\-context errors that are unbiased cancel in𝐳^¯sℓ​h\\overline\{\\hat\{\\mathbf\{z\}\}\}^\{\\ell h\}\_\{s\}\. For example, the mean field badly predicts the attention of a previous\-token head in*any single context*\(that attention is one\-hot whereas the kernel spreads over the composition\), but its errors average out against its own bigram statistics, so by \([17](https://arxiv.org/html/2609.16382#S6.E17)\) it scores low\. The score isolates the*systematic*, type\-conditioned part of a head’s context sensitivity, which is what makes it useful for detecting in\-context learning\.

#### What the ranking recovers\.

We compute the corpus deviationS¯ℓ​h\\bar\{S\}^\{\\ell h\}for all 144 GPT\-2 heads and rank them\. Figure[12](https://arxiv.org/html/2609.16382#S6.F12)shows all heads on theMcovM\_\{\\mathrm\{cov\}\}vs\.MctxM\_\{\\mathrm\{ctx\}\}axes, and Table[4](https://arxiv.org/html/2609.16382#S6.T4)shows the top twenty, with deviation also broken down byMcovM\_\{\\mathrm\{cov\}\}vs\.MctxM\_\{\\mathrm\{ctx\}\}\([33](https://arxiv.org/html/2609.16382#A16.E33)\), along with the behavioral induction score of[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49)\. In the final two columns, we document the role of each head from the literature and from our own studies\. Roles are attributed to the most authoritative source that names the head: the IOI circuit\([Wang et al\., 2023a](https://arxiv.org/html/2609.16382#bib.bib69)\), the induction\-head behavioral class defined by the test in[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49), and this paper\. Failing all of those, we use the SAE\-based head survey\([Krzyzanowski et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib38)\), whose annotations are unrefereed and should be read as indicative; those entries are given in quotation marks to mark them as that survey’s characterizations rather than established roles\.

We find that the top heads include many heads with a known in\-context learning role\. For example,[Wang et al\.](https://arxiv.org/html/2609.16382#bib.bib69)identify four induction heads, three of which are in the top\-20 here; and they identify 11 name movers, 5 of which are in the top\-20 here\. However, the top of the list is not limited to ICL heads\. The top twenty also include heads that move absolute position, and heads whose role we cannot document\. We walk through the full top\-20 list of heads below, with additional per\-head detail in Appendix[P](https://arxiv.org/html/2609.16382#A16)\.

#### Position movers \(L1H8,L2H7,L1H2,L1H3,L1H10\)\.

The first example of contextual processing we illustrate is*positional*transport\. For these heads,9090–99%99\\%of their within\-type output variance is contained in GPT\-2’s positional subspace\. We find that for each head, what is moved is the*absolute position*of the attended\-to key token\. Of these heads,L1H2attends locally \(usually to the token immediately before the query\),L1H10attends widely across the context, and the remaining three heads attend to BOS\. These latter three therefore move the position of the BOS into the query token\. We perform a detailed analysis of the mechanism, explore why this is useful and what it can accomplish in Appendix[P\.4](https://arxiv.org/html/2609.16382#A16.SS4)\.

#### Induction heads \(L5H5,L6H9,L7H10\)\.

These heads show high induction scores on the metric of\([Olsson et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib49)\)\. They are highly covariance\-dominated, reflecting the fact that their contextual processing arises from unusual attention\. This is because an induction head attends to an earlier token that is selected based on a previous pattern in the context, rather than being predictable based on the earlier token’s type\.

#### Name movers \(L9H9,L9H6,L10H6,L10H0,L10H10\)\.

Three name moversL9H9,L9H6,L10H0and backup name moversL10H6andL10H10were previously characterized as part of the IOI circuit in[Wang et al\. \(2023a\)](https://arxiv.org/html/2609.16382#bib.bib69)\. These heads show intermediate induction scores \(0\.15\-0\.45\), showing that they are sensitive to preceding patterns in the input, but do not perfectly copy them\. These heads are contextualization dominated; they move contextualized representations from tokens representing non\-repeated names in IOI prompts, a kind of content\-driven data movement\.

#### Content Movers \(L9H8,L5H2,L8H3,L6H8\)\.

A number of heads at the top of the list are not previously characterized in the literature\. To gain some insight into their potential role, we attempt to interpret aspects of their function\. This succeeds to varying degrees, and we do not claim that these characterizations are complete or highly accurate\. Since they show high mean\-field deviation, we try to interpret these heads by examining contexts and in some cases decoding the content that they move\. We do this by extracting per\-context deviation vectors𝚫i=𝐳i−𝐳^i\\mathbf\{\\Delta\}\_\{i\}=\\mathbf\{z\}\_\{i\}\-\\hat\{\\mathbf\{z\}\}\_\{i\}over a corpus, and selecting contexts containing the highest\-norm \(largest deviation\) instances\. For each, we examine the context around the max\-deviation token, and attempt to decode the deviation vector using the Logit Lens[nostalgebraist \(2020\)](https://arxiv.org/html/2609.16382#bib.bib46)and the J\-lens[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib30)\. We generically call these heads ‘content movers\.’

L9H8\.Decoding its high\-scoring deviation vectors, we find that in all 8 out of the top 8 cases, the deviation vector decodes via J\-lens to the document’s topic \(Table[17](https://arxiv.org/html/2609.16382#A16.T17)\), suggesting that what makes this head’s behavior not type\-predictable is its movement of topic information into the query\.

L5H2\.J\-lens decoding suggests that this head moves an abstract completion of degree or limitation computed from the local construction \(Table[18](https://arxiv.org/html/2609.16382#A16.T18)\)\.

L8H3\.High\-deviation contexts show movement of content in parallel and enumerated frames — repeated constructions, itemized lists, paired names — attending to the frame’s anchor \(Table[19](https://arxiv.org/html/2609.16382#A16.T19)\)\.

L6H8\.High\-deviation contexts show movement of content within idiomatic comparative frames \(“as …as one can,” “to say the least”; Table[20](https://arxiv.org/html/2609.16382#A16.T20)\)\.

#### Uncharacterized heads \(L9H4,L6H2, andL0H11\)\.

Three heads remain uncharacterized among the top 20\.L9H4is characterized in\([Krzyzanowski et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib38)\)as ‘particle/pronoun bigram\.’ We were not able to reach a confident characterization ofL6H2orL0H11\.

#### Heads at the bottom of the ranking\.

We study the low end of the ranking in Figure[12](https://arxiv.org/html/2609.16382#S6.F12)and Table[21](https://arxiv.org/html/2609.16382#A16.T21)\(Appendix[P\.3](https://arxiv.org/html/2609.16382#A16.SS3.SSS0.Px4)\)\. These are heads whose corpus\-average behavior is indistinguishable from a head that only uses type and co\-occurrence to determine attention\. We note that a number of heads with known functions show up there as well\. Four heads are duplicate\-token heads \(L0H1,L0H10, andL3H0were identified in[Wang et al\. \(2023a\)](https://arxiv.org/html/2609.16382#bib.bib69);L0H2was detected by our own inspection of attention\)\. These heads have near\-zero mean\-field deviation, which is because their attention patterns are*entirely type\-predictable*: they always attend to themselves\. The other set isL0H7,L1H0,L2H3, andL2H5\. These are previous\-token heads that always attend one position back\. For such a head its corpus average is just a bigram statistic – which the mean\-field kernel consequently encodes\. Mean\-field deviation is again small as a result\.

Table 4:The twenty GPT\-2 heads with the largest mean\-field deviation, rank\-ordered byS¯ℓ​h\\bar\{S\}^\{\\ell h\}\([17](https://arxiv.org/html/2609.16382#S6.E17)\), decomposed intoMcovM\_\{\\mathrm\{cov\}\}vs\.Mctx\\,M\_\{\\mathrm\{ctx\}\}\([33](https://arxiv.org/html/2609.16382#A16.E33)\)\.†induction is the behavioral class of[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.16382#bib.bib49);‡unresolved by our analysis, label from the SAE survey at5/105/10confidence\.Figure 12:All144144GPT\-2 heads on theMcovM\_\{\\mathrm\{cov\}\}/MctxM\_\{\\mathrm\{ctx\}\}plane \([33](https://arxiv.org/html/2609.16382#A16.E33)\), natural OpenWebText\. Large markers: the top twenty heads of Table[4](https://arxiv.org/html/2609.16382#S6.T4), colored by functional group, and the named heads from the*bottom*of the ranking \(Table[21](https://arxiv.org/html/2609.16382#A16.T21)\)\.L1H3\(rank 3\) is omitted: at\(\+0\.34,\+0\.05\)\(\+0\.34,\+0\.05\)it is a far outlier\. The faint dashed diagonal isy=xy=x: heads below it are covariance\-dominated, heads above it contextualization\-dominated\.
#### Circuits straddle the ranking\.

A*circuit*is a group of heads that compose across layers to carry out a function: an earlier head writes some structure into the residual stream, and a later head reads it back to produce its output\. An induction circuit composes a previous\-token head with an induction head\([Elhage et al\., 2021](https://arxiv.org/html/2609.16382#bib.bib17);[Olsson et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib49)\), and the IOI circuit composes duplicate\-token heads with name movers\([Wang et al\., 2023a](https://arxiv.org/html/2609.16382#bib.bib69)\); in each the first head writes and the second reads\. Both pairings straddle our ranking\. Threeinduction headssit in the top twenty while the previous\-token headsL2H5,L2H3andL1H0that feed them sit near the bottom; thename moversrank high while the duplicate\-token headsL3H0andL0H1that feed them rank near last\. We hypothesize this as a principle needed for ICL\. In\-context retrieval is a*write\-then\-read*computation: a first stage lays down retrievable structure by applying the same local rule at every position \(tag each token with its predecessor; flag duplicates\), and a second stage dereferences that structure conditionally\. The first stage must be uniform to be reliable, since it runs before the query is known, and uniform behavior conditioned on type is what the mean\-field kernel predicts, so a tagging head’s write deviation is small\. The second stage’s output is the novel in\-context binding, which corpus statistics cannot supply, so its deviation is necessarily large\.DMFD\_\{\\mathrm\{MF\}\}thus meters where contingency enters a computation, and a lookup circuit appears in the ranking twice: its invariant half at the bottom, its contingent half at the top\. This hypothesis accords with existing two\-stage readings of in\-context learning\. Induction is described as a paired write/read, or associative\-memory, mechanism\([Olsson et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib49);[Bietti et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib7)\)\. In label\-anchor accounts, shallow layers aggregate context into anchor positions that deeper layers then read out\([Wang et al\., 2023b](https://arxiv.org/html/2609.16382#bib.bib70)\)\. And in the same Gemma\-2\-2B model studied here, the contextualize\-then\-aggregate circuit account has early layers write contextual information into exemplar positions that later layers aggregate into the task representation\([Bakalova et al\., 2025](https://arxiv.org/html/2609.16382#bib.bib5)\)\. While these accounts all describe the same division of labor, what mean\-field analysis adds is a principle, grounded in a corpus\-level statistic, that separates the two roles without specifying a task\.

### 6\.4Task\-Specific Deviations Are Function Vectors

So far, we have been measuring a head’s contribution on a specific prompt against a corpus average, and using the deviation as a diagnostic; in §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3), we used it to locate, decompose, and characterize contextual\-processing heads\. However, the mean field deviation becomes something we can actively inject when it is measured against a*task*\.

[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)showed that in large models, GPT\-J among them, a small set of attention heads carries a transportable ‘function vector’: the sum of those heads’ outputs on few\-shot prompts of a task, added to the residual stream of an unrelated prompt, causes the model to perform the task\. For example, an antonym vector can be extracted from few\-shot prompts that cue the task with an answer\-cuing token ‘:’ \(‘hot: cold, fast: slow, …’\), then injected into a zero\-shot prompt with no examples \(‘good:’\), where it triggers the model answer ‘bad’\.[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)find attention heads through causal mediation, a supervised search over patching interventions\. We construct an alternative function vector extraction pipeline using mean\-field deviation, and compare the two methods in depth\.

#### Measuring deviation against a task\.

As in §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3.SSS0.Px1), a head’s contribution is averaged over contexts*before*the deviation is taken\. Here, however, we average the head’s contribution over the task’s few\-shot prompts rather than over the corpus, and keep the difference as a vector instead of reducing it to the scalar deviation of \([17](https://arxiv.org/html/2609.16382#S6.E17)\)\. For a prompt ending in the answer\-cuing token ‘:’, we define for each head its*task\-specific deviation*

rℓ​h=𝔼X∈task prompts\[𝐳‘:’,Xℓ​h\]−𝐳^‘:’ℓ​h,r^\{\\ell h\}\\;=\\;\\mathbb\{E\}\_\{X\\in\\text\{task prompts\}\}\\Big\[\\mathbf\{z\}^\{\\ell h\}\_\{\\text\{\`:'\},X\}\\Big\]\\;\-\\;\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{\\text\{\`:'\}\},\(18\)the average over the task’s promptsXXof the head’s contribution at the final ‘:’, minus the*corpus*mean\-field contribution𝐳^sℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}evaluated ats=‘:’s=\\text\{\`:'\}\.

An extraction pipeline has two components:*head selection*and*vector construction*\. For head selection, mean\-field analysis ranks heads by the norm of the deviation vector‖rℓ​h‖\\\|r^\{\\ell h\}\\\|; those whose task\-prompt output departs most from their corpus expectation are ranked highest\.[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)select heads by causal mediation\. They take a head’s average activation on clean task prompts, patch it into label\-corrupted prompts, and score the head by how much that restores the answer\. They call this theaverage indirect effect\(AIE\)\. For vector construction, mean\-field analysis injects the deviation vectorrℓ​hr^\{\\ell h\}itself, while[Todd et al\.](https://arxiv.org/html/2609.16382#bib.bib67)take the raw sum of the chosen heads’ outputs\. Unlike the patching search used by[Todd et al\.](https://arxiv.org/html/2609.16382#bib.bib67), the mean\-field measurement needs no contrast prompts and just one forward pass per prompt\.

These two differences raise two questions\. \(1\) For either piece of the pipeline, does one approach dominate the other? \(2\) Which full pipeline transports function vectors better: \(AIE \+ raw averages\) or \(MFA residual norms \+ deviation vectors\)? We answer these below, with full results in Appendix[Q](https://arxiv.org/html/2609.16382#A17)\.

#### Setup\.

We consider Gemma\-2\-2B and GPT\-J\-6B, with six tasks spanning a range of difficulty: the antonym, capital and past\-tense tasks of §[6\.1](https://arxiv.org/html/2609.16382#S6.SS1), plus country→\\tolanguage, country→\\tocurrency, and English→\\toFrench word translation\. Each task draws on6060cue–answer pairs, split three times into different2020extraction and4040held\-out evaluation pairs\. We score a method by the fraction of the held\-out queries the model answers correctly when its extracted vector is injected into a zero\-shot prompt, averaged over the three splits \(n=120n=120; binomial95%95\\%confidence intervals roughly±0\.03\\pm 0\.03–0\.090\.09\)\. The MFA head selection is recomputed from each split’s extraction prompts \(it is stable:77–1010of ten heads shared across splits\), while the AIE selection is held fixed from a single mediation sweep per task\. Injection follows[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67): the selected heads’ vectors are summed into one vector and added to the residual stream at a single layer and the final token position \(their Eqs\. 5 and 14\)\. We fix the injection layer and scale by one sweep and hold them fixed across all methods, so none is favored by tuning\. Appendix[Q](https://arxiv.org/html/2609.16382#A17)gives the sweep, the per\-head\-injection and GPT\-J protocols, and confirms that the protocol choice does not affect the comparison\.

Table 5:Gemma\-2\-2B: zero\-shot transfer for each combination of head selection and vector construction\. AIE∩\\capMFA is the number of heads the selections share; a range means it varies across the three splits\. The best result per task is in bold\.
#### Results\.

Table[5](https://arxiv.org/html/2609.16382#S6.T5)reports zero\-shot transfer for each combination of head selection \(AIE vs\. MFA\) and vector construction \(raw sum vs\. MFArr\) for Gemma\-2\-2B\. In this model, the overlap between the two head selection methods is low and neither consistently performs best\. The heads selected also sit at different depths: AIE in a compact mid\-stack group \(L12–L15 with head L14H0 rank 1 on all six tasks\) and MFA in an earlier, more task\-varying L6–L9 band\. On GPT\-J, the two selections more closely coincide \(MFA selects44–77of Todd’s ten heads\) and transfer rates tie on each of the tasks considered \(Table[24](https://arxiv.org/html/2609.16382#A17.T24)in Appendix[Q](https://arxiv.org/html/2609.16382#A17)\)\.

For vector construction, an initial observation is that again neither method is superior, with an even split across tasks\. But, closer inspection shows the surprising result that the best pipeline for a task almost always*mixes*the two methods, taking the head selection from one method and the injection vector from another\. On antonym, for instance, AIE selection coupled with the MFA deviation vector reaches0\.820\.82, above either aligned pairing \(0\.570\.57for AIE,0\.760\.76for MFA\)\. Further study of these apparent complementarities could be a useful direction for future work\.

#### GPT\-2: category vectors\.

At GPT\-2 small’s scale, injecting either construction into a zero\-shot prompt never gets the model to produce that prompt’s own answer, and thus Todd et al\. did not evaluate function vectors in models at GPT\-2’s scale\. What the deviation vector transports instead is the task’s*category*; §[Q\.3](https://arxiv.org/html/2609.16382#A17.SS3)gives the measurements behind the following results\. On capital, the injected model names a capital city on all120120held\-out queries, but essentially never the one the query asks for\. The raw average produces no task answer at all, on any query\.

We can see that the deviation vector’s content nonetheless indicates the whole category rather than the remembered answers, so it is not simply parroting\. Decoding the injected vector with the J\-lens of[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib30), on GPT\-2’s best capital head, the capitals seen in the prompts, the held\-out capitals and*capitals belonging to neither set*all rank high in the vocabulary \(median ranks99,3737and3434, against∼21,000\{\\sim\}21\{,\}000for unrelated words\)\. The small advantage that observed capitals hold translates into their being output as the top prediction\. Gemma\-2’s vectors do not have this behavior\. At the scale of GPT\-2, then, the vector fixes the*kind*of answer the model should give but not the correct answer, and leans towards categorical responses it has previously seen\.

## 7Conclusions

Mean\-field analysis describes a model’s average behavior relative to a chosen corpus, so our results are necessarily conditional on the corpora studied\. While we compare results across corpora \(and find little difference in results among them\), there are many diverse corpora that may be considered as a basis for mean\-field analysis\. A fruitful direction for future work is understanding how different corpora induce different attention mean\-fields, and how an appropriate corpus for training the mean field can be chosen in any given analysis setting\.

To summarize, we make four contributions\. First, we develop the conceptual framework needed to precisely describe the average\-case attention of a head\. Second, we show that, using this operator, one can make accurate predictions of average representation evolution through the layers of a model\. Third, we show that in early training stages, models behave like their mean fields\. Finally, mean\-field deviation provides a task\-agnostic measure of context\-specific computation: it recovers known ICL machinery and yields task residuals that act as transferable function vectors\.

#### Acknowledgements\.

Mark Crovella was supported by NSF grants CNS\-2312711 and CNS\-2319369, and by a grant from Coefficient Giving\.

## Use of AI

The authors used various versions of Anthropic’s Claude and OpenAI’s ChatGPT models throughout this work\. An explicit goal of this research project was to explore the limits of publicly available generative AI capabilities \(circa 2026\) for conducting research in mechanistic interpretability\. Under the authors’ guidance, Claude wrote the experimental code, ran and re\-ran numerous experiments, drafted preliminary analyses of experimental results, and suggested follow\-on experiments\. In many cases, validated findings from those experiments appear in this manuscript\. In close collaboration with the authors, Claude and ChatGPT reviewed and copy\-edited portions of the manuscript for correctness, readability and concision\. The research questions, experimental designs, and the interpretations reported here are the authors’\. The authors take full responsibility for the content of this paper\.

## References

- Ahn et al\. \(2023\)Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra\.Transformers learn to implement preconditioned gradient descent for in\-context learning\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/hash/8ed3d610ea4b68e7afb30ea7d01422c6\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/8ed3d610ea4b68e7afb30ea7d01422c6-Abstract-Conference.html)\.
- Arora et al\. \(2016\)Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski\.A latent variable model approach to PMI\-based word embeddings\.*Transactions of the Association for Computational Linguistics*, 4:385–399, 2016\.doi:10\.1162/tacl\_a\_00106\.URL[https://aclanthology\.org/Q16\-1028/](https://aclanthology.org/Q16-1028/)\.
- Ba et al\. \(2016\)Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E\. Hinton\.Layer normalization\.*arXiv preprint arXiv:1607\.06450*, 2016\.
- Bai et al\. \(2023\)Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei\.Transformers as statisticians: Provable in\-context learning with in\-context algorithm selection\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/hash/b2e63e36c57e153b9015fece2352a9f9\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/b2e63e36c57e153b9015fece2352a9f9-Abstract-Conference.html)\.
- Bakalova et al\. \(2025\)Aleksandra Bakalova, Yana Veitsman, Xinting Huang, and Michael Hahn\.Contextualize\-then\-aggregate: Circuits for in\-context learning in gemma\-2 2b\.*arXiv preprint arXiv:2504\.00132*, 2025\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purber, USVSN Sai Prashanth, Edward Raff, et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning*, 2023\.
- Bietti et al\. \(2023\)Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Hervé Jégou, and Léon Bottou\.Birth of a transformer: A memory viewpoint\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- Bondarenko et al\. \(2023\)Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort\.Quantizable transformers: Removing outliers by helping attention heads do nothing\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Brody et al\. \(2023\)Shaked Brody, Uri Alon, and Eran Yahav\.On the expressivity role of LayerNorm in transformers’ attention\.In*Findings of the Association for Computational Linguistics: ACL 2023*, 2023\.
- Burger et al\. \(2025\)Martin Burger, Samira Kabri, Yury Korolev, Tim Roith, and Lukas Weigand\.Analysis of mean\-field models arising from self\-attention dynamics in transformer architectures with layer normalization, 2025\.URL[https://arxiv\.org/abs/2501\.03096](https://arxiv.org/abs/2501.03096)\.
- Castin et al\. \(2025\)Valérie Castin, Pierre Ablin, José Antonio Carrillo, and Gabriel Peyré\.A unified perspective on the dynamics of deep transformers, 2025\.URL[https://arxiv\.org/abs/2501\.18322](https://arxiv.org/abs/2501.18322)\.
- Chan et al\. \(2022\)Stephanie C\. Y\. Chan, Adam Santoro, Andrew K\. Lampinen, Jane X\. Wang, Aaditya K\. Singh, Pierre H\. Richemond, James L\. McClelland, and Felix Hill\.Data distributional properties drive emergent in\-context learning in transformers\.In*Advances in Neural Information Processing Systems*, volume 35, pages 18878–18891, 2022\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/hash/77c6ccacfd9962e2307fc64680fc5ace\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/77c6ccacfd9962e2307fc64680fc5ace-Abstract-Conference.html)\.
- Chang and Bergen \(2022\)Tyler A\. Chang and Benjamin K\. Bergen\.Word acquisition in neural language models\.*Transactions of the Association for Computational Linguistics*, 10:1–16, 2022\.doi:10\.1162/tacl\_a\_00444\.URL[https://doi\.org/10\.1162/tacl\_a\_00444](https://doi.org/10.1162/tacl_a_00444)\.
- Choshen et al\. \(2022\)Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend\.The grammar\-learning trajectories of neural language models\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 8281–8297, Dublin, Ireland, 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.acl\-long\.568\.URL[https://aclanthology\.org/2022\.acl\-long\.568/](https://aclanthology.org/2022.acl-long.568/)\.
- Dettmers et al\. \(2022\)Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer\.LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Dohmatob \(2025\)Elvis Dohmatob\.Understanding softmax attention layers: Exact mean\-field analysis on a toy problem\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.URL[https://papers\.neurips\.cc/paper\_files/paper/2025/hash/2cf63608d264a3c65e68e3cff860c152\-Abstract\-Conference\.html](https://papers.neurips.cc/paper_files/paper/2025/hash/2cf63608d264a3c65e68e3cff860c152-Abstract-Conference.html)\.
- Elhage et al\. \(2021\)Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield\-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah\.A mathematical framework for transformer circuits\.*Transformer Circuits Thread*, 2021\.URL[https://transformer\-circuits\.pub/2021/framework/index\.html](https://transformer-circuits.pub/2021/framework/index.html)\.
- Engels et al\. \(2025\)Joshua Engels, Eric J\. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark\.Not all language model features are one\-dimensionally linear\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=d63a4AM4hb](https://openreview.net/forum?id=d63a4AM4hb)\.
- Ethayarajh \(2019\)Kawin Ethayarajh\.How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 55–65, 2019\.
- Franco and Crovella \(2025\)Gabriel Franco and Mark Crovella\.Pinpointing attention\-causal communication in language models\.In*Proceedings of NeurIPS*, San Diego, CA, 2025\.doi:TBD\.URL[http://www\.cs\.bu\.edu/faculty/crovella/paper\-archive/attention\-causal\-neurips25\.pdf](http://www.cs.bu.edu/faculty/crovella/paper-archive/attention-causal-neurips25.pdf)\.Also appeared in Mechanistic Interpretability Workshop at NeurIPS 2025\.
- Gao et al\. \(2019a\)Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie\-Yan Liu\.Representation degeneration problem in training natural language generation models\.In*International Conference on Learning Representations \(ICLR\)*, 2019a\.
- Gao et al\. \(2019b\)Xiang Gao, Meera Sitharam, and Adrian E\. Roitberg\.Bounds on the Jensen gap, and implications for mean\-concentrated distributions\.*The Australian Journal of Mathematical Analysis and Applications*, 16\(2\), 2019b\.
- García\-Carrasco et al\. \(2024\)Jorge García\-Carrasco, Alejandro Maté, and Juan Trujillo\.How does GPT\-2 predict acronyms? Extracting and understanding a circuit via mechanistic interpretability\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, volume 238, pages 3322–3330, 2024\.arXiv:2405\.04156\.
- Gemma Team \(2024\)Gemma Team\.Gemma 2: Improving open language models at a practical size\.*arXiv preprint arXiv:2408\.00118*, 2024\.
- Geshkovski et al\. \(2025\)Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet\.A mathematical perspective on transformers\.*Bulletin of the American Mathematical Society*, 62\(3\):427–479, 2025\.
- Gokaslan et al\. \(2019\)Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex\.Openwebtext corpus\.[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus), 2019\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Gu et al\. \(2025\)Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin\.When attention sink emerges in language models: An empirical study\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Gurnee and Tegmark \(2024\)Wes Gurnee and Max Tegmark\.Language models represent space and time\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=jE8xbmvFin](https://openreview.net/forum?id=jE8xbmvFin)\.
- Gurnee et al\. \(2026\)Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey\.Verbalizable representations form a global workspace in language models\.Transformer Circuits Thread, 2026\.URL[https://transformer\-circuits\.pub/2026/workspace/index\.html](https://transformer-circuits.pub/2026/workspace/index.html)\.
- Hendel et al\. \(2023\)Roee Hendel, Mor Geva, and Amir Globerson\.In\-context learning creates task vectors\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 9318–9333, Singapore, 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.findings\-emnlp\.624\.URL[https://aclanthology\.org/2023\.findings\-emnlp\.624/](https://aclanthology.org/2023.findings-emnlp.624/)\.
- Idel \(2016\)Martin Idel\.A review of matrix scaling and Sinkhorn’s normal form for matrices and positive maps\.*arXiv preprint arXiv:1609\.06349*, 2016\.
- Im et al\. \(2026\)Shawn Im, Changdae Oh, Zhen Fang, and Sharon Li\.How do transformers learn to associate tokens: Gradient leading terms bring mechanistic interpretability\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.
- Jiang et al\. \(2024\)Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch\.On the origins of linear representations in large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 21879–21911\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/jiang24d\.html](https://proceedings.mlr.press/v235/jiang24d.html)\.
- Karkada et al\. \(2026\)Dhruva Karkada, Daniel J\. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri\.Symmetry in language statistics shapes the geometry of model representations, 2026\.URL[https://arxiv\.org/abs/2602\.15029](https://arxiv.org/abs/2602.15029)\.
- Kim and Suzuki \(2024\)Juno Kim and Taiji Suzuki\.Transformers learn nonlinear features in context: Nonconvex mean\-field dynamics on the attention landscape\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 24527–24561\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/kim24af\.html](https://proceedings.mlr.press/v235/kim24af.html)\.
- Kovaleva et al\. \(2021\)Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky\.BERT busters: Outlier dimensions that disrupt transformers\.In*Findings of the Association for Computational Linguistics: ACL\-IJCNLP 2021*, 2021\.
- Krzyzanowski et al\. \(2024\)Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda\.We inspected every head in GPT\-2 small using SAEs so you don’t have to\.AI Alignment Forum, 2024\.Accessed 2026\-08\-05; per\-head annotations in the authors’ companion spreadsheet\.
- Levy and Goldberg \(2014\)Omer Levy and Yoav Goldberg\.Neural word embedding as implicit matrix factorization\.In*Advances in Neural Information Processing Systems*, volume 27, pages 2177–2185, 2014\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4\-Abstract\.html](https://proceedings.neurips.cc/paper_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html)\.
- Lu et al\. \(2026\)Jennifer Meng Lu, Ruochen Zhang, Isabelle Lee, David Alvarez\-Melis, Ellie Pavlick, and Naomi Saphra\.Adversarial concept search: Predicting compositional errors from feature geometry, 2026\.URL[https://arxiv\.org/abs/2606\.13934](https://arxiv.org/abs/2606.13934)\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Meta AI \(2024\)Meta AI\.Llama 3\.2: Revolutionizing edge AI and vision with open, customizable models\.[https://ai\.meta\.com/blog/llama\-3\-2\-connect\-2024\-vision\-edge\-mobile\-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/), 2024\.Accessed 2026\-08\-03\.
- Mu et al\. \(2018\)Jiaqi Mu, Suma Bhat, and Pramod Viswanath\.All\-but\-the\-top: Simple and effective postprocessing for word representations\.In*International Conference on Learning Representations \(ICLR\)*, 2018\.
- Nanda and Bloom \(2022\)Neel Nanda and Joseph Bloom\.Transformerlens\.[https://github\.com/TransformerLensOrg/TransformerLens](https://github.com/TransformerLensOrg/TransformerLens), 2022\.
- Nava and Wyart \(2026\)Andres Nava and Matthieu Wyart\.Hierarchical concept geometry in language models emerges from word co\-occurrence, 2026\.URL[https://arxiv\.org/abs/2605\.23821](https://arxiv.org/abs/2605.23821)\.
- nostalgebraist \(2020\)nostalgebraist\.Interpreting GPT: the logit lens\.LessWrong, 2020\.
- Oehlert \(1992\)Gary W\. Oehlert\.A note on the delta method\.*The American Statistician*, 46\(1\):27–29, 1992\.
- OLMo Team \(2024\)OLMo Team\.2 OLMo 2 Furious, 2024\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al\.In\-context learning and induction heads\.*Transformer Circuits Thread*, 2022\.
- Oswald et al\. \(2023\)Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov\.Transformers learn in\-context by gradient descent\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 35151–35174\. PMLR, 2023\.URL[https://proceedings\.mlr\.press/v202/von\-oswald23a\.html](https://proceedings.mlr.press/v202/von-oswald23a.html)\.
- Park et al\. \(2025a\)Core Francisco Park, Ekdeep Singh Lubana, and Hidenori Tanaka\.Competition dynamics shape algorithmic phases of in\-context learning\.In*The Thirteenth International Conference on Learning Representations*, 2025a\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/a635abfdd007c9a698410c8bbe5ebc33\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/a635abfdd007c9a698410c8bbe5ebc33-Abstract-Conference.html)\.
- Park et al\. \(2024\)Kiho Park, Yo Joong Choe, and Victor Veitch\.The linear representation hypothesis and the geometry of large language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 39643–39666\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/park24c\.html](https://proceedings.mlr.press/v235/park24c.html)\.
- Park et al\. \(2025b\)Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch\.The geometry of categorical and hierarchical concepts in large language models\.In*The Thirteenth International Conference on Learning Representations*, 2025b\.URL[https://openreview\.net/forum?id=bVTM2QKYuA](https://openreview.net/forum?id=bVTM2QKYuA)\.
- Penedo et al\. \(2024\)Guilherme Penedo, Hynek Kydlí𝐜\\mathbf\{c\}ek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf\.The fineweb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks*, 2024\.
- Pennington et al\. \(2014\)Jeffrey Pennington, Richard Socher, and Christopher Manning\.GloVe: Global vectors for word representation\.In*Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing*, pages 1532–1543, Doha, Qatar, 2014\. Association for Computational Linguistics\.doi:10\.3115/v1/D14\-1162\.URL[https://aclanthology\.org/D14\-1162/](https://aclanthology.org/D14-1162/)\.
- Poc\-López and Aguilera \(2024\)Ángel Poc\-López and Miguel Aguilera\.Dynamical mean\-field theory of self\-attention neural networks, 2024\.URL[https://arxiv\.org/abs/2406\.07247](https://arxiv.org/abs/2406.07247)\.
- Puccetti et al\. \(2022\)Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell’Orletta\.Outlier dimensions that disrupt transformers are driven by frequency\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, 2022\.
- Radford et al\. \(2019\)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.*OpenAI Technical Report*, 2019\.
- Rae et al\. \(2020\)Jack W\. Rae, Anna Potapenko, Siddhant M\. Jayakumar, Chloe Hillier, and Timothy P\. Lillicrap\.Compressive transformers for long\-range sequence modelling\.In*International Conference on Learning Representations \(ICLR\)*, 2020\.
- Raganato et al\. \(2020\)Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann\.Fixed encoder self\-attention patterns in transformer\-based machine translation\.In*Findings of the Association for Computational Linguistics: EMNLP 2020*, 2020\.
- Rigollet \(2025\)Philippe Rigollet\.The mean\-field dynamics of transformers, 2025\.URL[https://arxiv\.org/abs/2512\.01868](https://arxiv.org/abs/2512.01868)\.
- Sinkhorn and Knopp \(1967\)Richard Sinkhorn and Paul Knopp\.Concerning nonnegative matrices and doubly stochastic matrices\.*Pacific Journal of Mathematics*, 21\(2\):343–348, 1967\.
- Sun et al\. \(2024\)Mingjie Sun, Xinlei Chen, J\. Zico Kolter, and Zhuang Liu\.Massive activations in large language models\.In*Conference on Language Modeling \(COLM\)*, 2024\.
- Tay et al\. \(2021\)Yi Tay, Dara Bahri, Donald Metzler, Da\-Cheng Juan, Zhe Zhao, and Che Zheng\.Synthesizer: Rethinking self\-attention for transformer models\.In*Proceedings of the 38th International Conference on Machine Learning \(ICML\)*, volume 139 of*PMLR*, 2021\.
- Tigges et al\. \(2024\)Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman\.LLM circuit analyses are consistent across training and scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Timkey and van Schijndel \(2021\)William Timkey and Marten van Schijndel\.All bark and no bite: Rogue dimensions in transformer language models obscure representational quality\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 4527–4546, 2021\.
- Todd et al\. \(2024\)Eric Todd, Millicent L\. Li, Arnab Sen Sharma, Aaron Mueller, Byron C\. Wallace, and David Bau\.Function vectors in large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Vig and Belinkov \(2019\)Jesse Vig and Yonatan Belinkov\.Analyzing the structure of attention in a transformer language model\.In*Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, pages 63–76\. Association for Computational Linguistics, 2019\.doi:10\.18653/v1/W19\-4808\.URL[https://aclanthology\.org/W19\-4808/](https://aclanthology.org/W19-4808/)\.
- Wang et al\. \(2023a\)Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt\.Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.In*International Conference on Learning Representations*, 2023a\.
- Wang et al\. \(2023b\)Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun\.Label words are anchors: An information flow perspective for understanding in\-context learning\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9840–9855, 2023b\.
- Wurgaft et al\. \(2026\)Daniel Wurgaft, Can Rager, Matthew Kowal, Vasudev Shyam, Sheridan Feucht, Usha Bhalla, Tal Haklay, Eric Bigelow, Raphael Sarfati, Thomas McGrath, Owen Lewis, Jack Merullo, Noah Goodman, Thomas Fel, Atticus Geiger, and Ekdeep Singh Lubana\.Manifold steering reveals the shared geometry of neural network representation and behavior, 2026\.URL[https://arxiv\.org/abs/2605\.05115](https://arxiv.org/abs/2605.05115)\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- You et al\. \(2020\)Weiqiu You, Simeng Sun, and Mohit Iyyer\.Hard\-coded gaussian attention for neural machine translation\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2020\.

## Appendix AConditional Mean Fields with Repeated Types

Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)derived thes,ts,t\-conditional and context\-conditional mean fields under the simplifying assumption that a type occurs at most once in a context\. This appendix removes that assumption\. The structure of the derivation is unchanged: the same two mean fields are reached by the same two steps, but one additional assumption is required, governing how the attention a query places on a type grows with the number of copies of that type present\. Setting every count to zero or one recovers Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)exactly\.

#### Notation for repeated types\.

Recall from Section[3\.2](https://arxiv.org/html/2609.16382#S3.SS2)thatnt​\(X\)n\_\{t\}\(X\)is the number of positions of typettavailable to the query inXX, and writentn\_\{t\}for the corresponding random variable whenXXis drawn from the corpus\. We writePrs⁡\(nt=n\)\\Pr\_\{s\}\(n\_\{t\}=n\)for the probability that a random context with querysscontains exactlynncopies oftt, and𝔼s​t​n\\mathbb\{E\}\_\{stn\}for the conditional expectation over such contexts\. The expected number of type\-ttpositions in a type\-ssquery’s context is

n¯s​t=𝔼s​\[nt\]=∑n≥1n​Prs⁡\(nt=n\)\.\\overline\{n\}\_\{st\}\\;=\\;\\mathbb\{E\}\_\{s\}\[n\_\{t\}\]\\;=\\;\\sum\_\{n\\geq 1\}n\\,\{\\Pr\}\_\{s\}\(n\_\{t\}=n\)\.When a type can occur at most once,nt∈\{0,1\}n\_\{t\}\\in\\\{0,1\\\}, son¯s​t=Prs⁡\(t\)\\overline\{n\}\_\{st\}=\\Pr\_\{s\}\(t\)and𝔼s​t​1=𝔼s​t\\mathbb\{E\}\_\{st1\}=\\mathbb\{E\}\_\{st\}; every expression below then reduces to its counterpart in Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)\.

#### Thes,ts,t\-conditional mean field\.

Start from the exact contribution to the centroid \([3](https://arxiv.org/html/2609.16382#S3.E3)\):

𝐳¯s=∑t∈𝒯𝔼s​\[As​t​𝐯¯t\]\\overline\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}\\mathbb\{E\}\_\{s\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]
We now condition on the*number*of copies ofttin the context rather than merely on the presence oftt\. Partitioning on the value ofntn\_\{t\},

𝔼s​\[As​t​𝐯¯t\]=∑n≥0𝔼s​t​n​\[As​t​𝐯¯t\]​Prs⁡\(nt=n\)=∑n≥1𝔼s​t​n​\[As​t​𝐯¯t\]​Prs⁡\(nt=n\),\\mathbb\{E\}\_\{s\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]=\\sum\_\{n\\geq 0\}\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\,\{\\Pr\}\_\{s\}\(n\_\{t\}=n\)\\;=\\;\\sum\_\{n\\geq 1\}\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\,\{\\Pr\}\_\{s\}\(n\_\{t\}=n\),where then=0n=0term drops out becauseAs​t=0A\_\{st\}=0whenttis absent from the context, exactly as before\.

Denote𝔼s​t​n​\[As​t​𝐯¯t\]\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\\overline\{\\mathbf\{v\}\}\_\{t\}\]as𝐳¯s​t​n\\overline\{\\mathbf\{z\}\}\_\{stn\}\. This is the average contribution to thesstoken centroid by tokenttwhen exactlynncopies ofttare in the context: the influence measure𝐳¯s​t\\overline\{\\mathbf\{z\}\}\_\{st\}of Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3), resolved by multiplicity\. Then

𝐳¯s=∑t∈𝒯∑n≥1𝐳¯s​t​n​Prs⁡\(nt=n\)\\overline\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{n\\geq 1\}\\overline\{\\mathbf\{z\}\}\_\{stn\}\\,\{\\Pr\}\_\{s\}\(n\_\{t\}=n\)which is again an exact decomposition of the corpus contribution, now refined by multiplicity\. We rewrite𝐳¯s​t​n\\overline\{\\mathbf\{z\}\}\_\{stn\}as

𝐳¯s​t​n=𝔼s​t​n​\[As​t\]​𝔼s​t​n​\[𝐯¯t\]\+Covs​t​n⁡\(As​t,𝐯¯t\)\\overline\{\\mathbf\{z\}\}\_\{stn\}=\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\]\\,\\mathbb\{E\}\_\{stn\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\+\\operatorname\{Cov\}\_\{stn\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)and impose Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px1)on this finer conditioning set,Covs​t​n⁡\(As​t,𝐯¯t\)≈0\\operatorname\{Cov\}\_\{stn\}\(A\_\{st\},\\overline\{\\mathbf\{v\}\}\_\{t\}\)\\approx 0, giving

𝐳¯s​t​n≈𝔼s​t​n​\[As​t\]​𝔼s​t​n​\[𝐯¯t\]\.\\overline\{\\mathbf\{z\}\}\_\{stn\}\\approx\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\]\\,\\mathbb\{E\}\_\{stn\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\.\(19\)
Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px1)carries over with one addition: the average value map of tokenttis taken to be independent not only of the query type but also of how many copies ofttare present\.

Assumption 7\. Value\-query independence \(extended to repeated types\)\.𝔼s​t​n​\[𝐯¯t\]≈𝝁t\\mathbb\{E\}\_\{stn\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\\approx\\boldsymbol\{\\mu\}\_\{t\}, the corpus average value map of tokentt, for everyssand everynn\.

The quantity𝔼s​t​n​\[𝐯¯t\]\\mathbb\{E\}\_\{stn\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]is an average over type\-ssquery contexts, and𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}is itself attention\-weighted within the context, so it differs from𝝁t\\boldsymbol\{\\mu\}\_\{t\}by the degree to which typett’s representation is contextualized in the company ofss\. The other factor of \([19](https://arxiv.org/html/2609.16382#A1.E19)\),𝔼s​t​n​\[As​t\]\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\], is where repeated types enter: it is a separate quantity for each multiplicitynn, whereas Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)had the single numberWs​t=𝔼s​t​\[As​t\]W\_\{st\}=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\. Here, we recastWs​tW\_\{st\}as a as a rate*per key*rather than per context,

Ws​t:=𝔼s​\[As​t\]𝔼s​\[nt\]=Ps​tn¯s​t,W\_\{st\}\\;:=\\;\\frac\{\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]\}\{\\mathbb\{E\}\_\{s\}\[n\_\{t\}\]\}\\;=\\;\\frac\{P\_\{st\}\}\{\\overline\{n\}\_\{st\}\},the total attention a type\-ssquery places on typett, divided by the expected number of type\-ttkeys that mass was spread over\. This is a definition rather than an assumption, and it is directly measurable: numerator and denominator are each a single corpus pass\. When a type occurs at most once it reduces toWs​t=Ps​t/Prs⁡\(t\)=𝔼s​t​\[As​t\]W\_\{st\}=P\_\{st\}/\\Pr\_\{s\}\(t\)=\\mathbb\{E\}\_\{st\}\[A\_\{st\}\], the quantity of Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)\. Our final assumption relates how attention mass grows with multiplicity:

Assumption 8\. Homogeneity of attention\.𝔼s​t​n​\[As​t\]=n​Ws​t\\mathbb\{E\}\_\{stn\}\[A\_\{st\}\]=n\\,W\_\{st\}\. The attention a type\-ssquery places on typettis proportional to the number of copies ofttpresent, at a per\-key rate that does not otherwise depend on the context\.

Assumption[A](https://arxiv.org/html/2609.16382#A1.SS0.SSS0.Px2)is purely a linearity claim: averaging it back overnnreturns∑nn​Ws​t​Prs⁡\(nt=n\)=Ws​t​n¯s​t=Ps​t\\sum\_\{n\}n\\,W\_\{st\}\\,\\Pr\_\{s\}\(n\_\{t\}=n\)=W\_\{st\}\\,\\overline\{n\}\_\{st\}=P\_\{st\}, which holds identically by the definition ofWs​tW\_\{st\}\. What is being assumed is therefore not the overall level of the attention but its*distribution across multiplicities*: that a context holding five copies ofttdraws five times the mass of one holding a single copy, rather than, say, saturating\.

Under the three assumptions,

𝐳¯s≈𝐳^s=∑t∈𝒯Ws​t​𝝁t​∑n≥1n​Prs⁡\(nt=n\)=∑t∈𝒯Ws​t​𝝁t​n¯s​t,\\overline\{\\mathbf\{z\}\}\_\{s\}\\approx\\hat\{\\mathbf\{z\}\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\}W\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}\\sum\_\{n\\geq 1\}n\\,\{\\Pr\}\_\{s\}\(n\_\{t\}=n\)\\;=\\;\\sum\_\{t\\in\\mathcal\{T\}\}W\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}\\,\\overline\{n\}\_\{st\},the repeated\-type counterpart of thes,ts,t\-conditional mean field of Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3): the presence probabilityPrs⁡\(t\)\\Pr\_\{s\}\(t\)is replaced by the expected countn¯s​t\\overline\{n\}\_\{st\}, and nothing else changes\.

#### The context\-conditional mean field\.

We now repeat the second construction of Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)for a particular contextXXwith queryss, again allowing a type to occur any number of times inXX\. As before,Ws​t​XW\_\{stX\}denotes the attention paid bysstottin contextXX, and𝝁t​X\\boldsymbol\{\\mu\}\_\{tX\}the average value map of tokenttin that context\.

Note thatWs​t​XW\_\{stX\}is the*total*attention the query places on typett, summed over allnt​\(X\)n\_\{t\}\(X\)of its positions in the prefix; it isAs​tA\_\{st\}of \([3](https://arxiv.org/html/2609.16382#S3.E3)\) evaluated on this particular context\. Likewise𝝁t​X\\boldsymbol\{\\mu\}\_\{tX\}is the attention\-weighted mean of the value vectors at those sament​\(X\)n\_\{t\}\(X\)positions, which is𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}evaluated on this context\. Both therefore already account for every copy ofttinXX\.

Conditioning onXXfixes every count, soPrX⁡\(t\)\\Pr\_\{X\}\(t\)is again11for the tokens insupp⁡\(X\)\\supp\(X\)and00otherwise, and

𝐳s​X=∑t∈𝒯Ws​t​X​𝝁t​X​PrX⁡\(t\)=∑t∈supp⁡\(X\)Ws​t​X​𝝁t​X\.\\mathbf\{z\}\_\{sX\}=\\sum\_\{t\\in\\mathcal\{T\}\}W\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{tX\}\\,\{\\Pr\}\_\{X\}\(t\)\\;=\\;\\sum\_\{t\\in\\supp\(X\)\}W\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{tX\}\.
Next, as before, we make theassumptionthat𝝁t​X=𝝁t\\boldsymbol\{\\mu\}\_\{tX\}=\\boldsymbol\{\\mu\}\_\{t\}, again noting that the difference between them is the*contextualization*ofttinduced by its contextXX, leaving

𝐳^s​X=∑t∈supp⁡\(X\)Ws​t​X​𝝁t\.\\hat\{\\mathbf\{z\}\}\_\{sX\}=\\sum\_\{t\\in\\supp\(X\)\}W\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{t\}\.
And as before, sinceXXis a fixed context,Ws​t​XW\_\{stX\}is simply the attentionAs​t​XA\_\{stX\}that the head pays fromsstottin this context\. To construct a mean\-field analog we form anapproximationtoAs​t​XA\_\{stX\}using renormalization, now weighting each type by the number of copies it contributes:

Ws​t​X≈A^s​t​X=nt​\(X\)​Ws​t∑t′∈supp⁡\(X\)nt′​\(X\)​Ws​t′,W\_\{stX\}\\approx\\hat\{A\}\_\{stX\}=\\frac\{n\_\{t\}\(X\)\\,W\_\{st\}\}\{\\sum\_\{t^\{\\prime\}\\in\\supp\(X\)\}n\_\{t^\{\\prime\}\}\(X\)\\,W\_\{st^\{\\prime\}\}\},\(20\)the numerator being the mass that Assumption[A](https://arxiv.org/html/2609.16382#A1.SS0.SSS0.Px2)predicts for a context holdingnt​\(X\)n\_\{t\}\(X\)copies oftt\. This yields the*context\-conditional*mean field approximation for contexts with repeated types,

𝐳^s​X=∑t∈supp⁡\(X\)A^s​t​X​𝝁t,\\hat\{\\mathbf\{z\}\}\_\{sX\}=\\sum\_\{t\\in\\supp\(X\)\}\\hat\{A\}\_\{stX\}\\,\\boldsymbol\{\\mu\}\_\{t\},which reduces to the expression of Section[3\.3](https://arxiv.org/html/2609.16382#S3.SS3)when everynt​\(X\)n\_\{t\}\(X\)equals11\. Note that the multiplicities do not cancel from the renormalization: they reweight the types relative to one another, so the prediction depends onXXthrough the full multiset of types it contains and not merely throughsupp⁡\(X\)\\supp\(X\)\.

## Appendix BAnalysis of the Context\-conditional Mean Field

### B\.1The KernelA^\\hat\{A\}as a Mean Field

The context\-conditional mean field underlies §[5](https://arxiv.org/html/2609.16382#S5)and §[6](https://arxiv.org/html/2609.16382#S6)\. Its kernel renormalizes the corpus kernel over the types a context actually contains \([20](https://arxiv.org/html/2609.16382#A1.E20)\):

A^s​t​X=nt​\(X\)​Ws​tNX,NX=∑t′∈supp⁡\(X\)nt′​\(X\)​Ws​t′,\\hat\{A\}\_\{stX\}\\;=\\;\\frac\{n\_\{t\}\(X\)\\,W\_\{st\}\}\{N\_\{X\}\},\\qquad N\_\{X\}\\;=\\;\\sum\_\{t^\{\\prime\}\\in\\supp\(X\)\}n\_\{t^\{\\prime\}\}\(X\)\\,W\_\{st^\{\\prime\}\},\(21\)whereAs​t​XA\_\{stX\}is the*total*attention the head pays from a query of typessto allnt​\(X\)n\_\{t\}\(X\)positions of typett, andWs​tW\_\{st\}defined asPs​t/n¯s​tP\_\{st\}/\\bar\{n\}\_\{st\}\(Appendix[A](https://arxiv.org/html/2609.16382#A1)\) withPs​t=𝔼s​\[As​t\]P\_\{st\}=\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]andn¯s​t=𝔼s​\[nt​\(X\)\]\\bar\{n\}\_\{st\}=\\mathbb\{E\}\_\{s\}\[n\_\{t\}\(X\)\]\.

Renormalization is a natural way to turn a corpus\-average propensity into something that sums to one over a particular context, but it introduces a nonlinearity into the definition ofA^\\hat\{A\}\. Nonetheless, the choice ofA^\\hat\{A\}can be justified ifA^\\hat\{A\}is an*unbiased estimator*, that is, if

𝔼s​t​\[As​t​X−A^s​t​X\]=0\.\\mathbb\{E\}\_\{st\}\\\!\\left\[A\_\{stX\}\-\\hat\{A\}\_\{stX\}\\right\]=0\.\(22\)
HereA^s​t​X\\hat\{A\}\_\{stX\}varies with the context through its token counts, while𝔼s​t​\[As​t​X\]\\mathbb\{E\}\_\{st\}\[A\_\{stX\}\]is fixed, so \([22](https://arxiv.org/html/2609.16382#A2.E22)\) asks the random quantity to match the constant in expectation\.

The remainder of this section shows the following:

1. 1\.Equation \([22](https://arxiv.org/html/2609.16382#A2.E22)\) does*not*hold in general; we show why and we show how to estimate its error\.
2. 2\.Equation \([22](https://arxiv.org/html/2609.16382#A2.E22)\)*does*hold approximately in practice for the data we study in this paper\.
3. 3\.It is possible to define an*unbiased*replacement forWW\. Specifically, we show how to construct a matrixUU, such that if we useUUin place ofWWin the definition ofA^\\hat\{A\}, then \([22](https://arxiv.org/html/2609.16382#A2.E22)\) holds by construction\. When we replaceWWwith its unbiased versionUUin our experiments,*our key results do not change\.*
4. 4\.Since Equation \([22](https://arxiv.org/html/2609.16382#A2.E22)\) holds approximately in practice for our datasets, that justifies the splitting of context\-conditional mean field deviation into the components given in \([15](https://arxiv.org/html/2609.16382#S6.E15)\), which we call*covariance*and*contextualization*\.

In the remainder of this appendix we typically omit the contextXXand abbreviate the per\-context quantitiesAs​t​XA\_\{stX\},A^s​t​X\\hat\{A\}\_\{stX\}andnt​\(X\)n\_\{t\}\(X\)asAs​tA\_\{st\},A^s​t\\hat\{A\}\_\{st\}andntn\_\{t\}\. We restore the explicitXX\(as inAs​t​XA\_\{stX\},nt​\(X\)n\_\{t\}\(X\),NXN\_\{X\}\) at the steps where the dependence on the particular context is the point\. The expectations𝔼s\\mathbb\{E\}\_\{s\}and𝔼s​t\\mathbb\{E\}\_\{st\}average over the contextsXX, and the corpus quantitiesPs​tP\_\{st\},Ws​tW\_\{st\},n¯s​t\\bar\{n\}\_\{st\}andπs​t\\pi\_\{st\}carry no context argument\.

### B\.2Analysis of Bias inA^\\hat\{A\}

First, we give some insight into the nature of the bias inA^\\hat\{A\}\. BecauseAs​t​X=0A\_\{stX\}=0andnt​\(X\)=0n\_\{t\}\(X\)=0together wheneverttis absent, writingπs​t=Prs⁡\(t∈supp⁡\(X\)\)\\pi\_\{st\}=\\Pr\_\{s\}\(t\\in\\supp\(X\)\)gives𝔼s​\[As​t\]=πs​t​𝔼s​t​\[As​t\]\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]=\\pi\_\{st\}\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]and𝔼s​\[nt\]=πs​t​𝔼s​t​\[nt\]\\mathbb\{E\}\_\{s\}\[n\_\{t\}\]=\\pi\_\{st\}\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\. The presence probability therefore cancels from the definition ofWW:

Ws​t=Ps​tn¯s​t=𝔼s​\[As​t\]𝔼s​\[nt\]=𝔼s​t​\[As​t\]𝔼s​t​\[nt\]W\_\{st\}\\;=\\;\\frac\{P\_\{st\}\}\{\\bar\{n\}\_\{st\}\}\\;=\\;\\frac\{\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]\}\{\\mathbb\{E\}\_\{s\}\[n\_\{t\}\]\}\\;=\\;\\frac\{\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\}\{\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\}\(23\)which means that

𝔼s​t​\[As​t\]=Ws​t​𝔼s​t​\[nt\]\.\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\\;=\\;W\_\{st\}\\,\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\.\(24\)From \([21](https://arxiv.org/html/2609.16382#A2.E21)\),𝔼s​t​\[A^s​t\]=Ws​t​𝔼s​t​\[nt/NX\]\\mathbb\{E\}\_\{st\}\[\\hat\{A\}\_\{st\}\]=W\_\{st\}\\,\\mathbb\{E\}\_\{st\}\[n\_\{t\}/N\_\{X\}\]\. Subtracting from \([24](https://arxiv.org/html/2609.16382#A2.E24)\),

𝔼s​t​\[As​t−A^s​t\]=Ws​t​\(𝔼s​t​\[nt\]−𝔼s​t​\[nt/NX\]\)\.\\mathbb\{E\}\_\{st\}\\\!\\left\[A\_\{st\}\-\\hat\{A\}\_\{st\}\\right\]\\;=\\;W\_\{st\}\\Big\(\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\\;\-\\;\\mathbb\{E\}\_\{st\}\\\!\\left\[n\_\{t\}/N\_\{X\}\\right\]\\Big\)\.\(25\)Writing1/NX=1\+\(1−NX\)/NX1/N\_\{X\}=1\+\(1\-N\_\{X\}\)/N\_\{X\}in \([25](https://arxiv.org/html/2609.16382#A2.E25)\),

𝔼s​t​\[nt/NX\]−𝔼s​t​\[nt\]=−𝔼s​t​\[nt​\(NX−1\)NX\],\\mathbb\{E\}\_\{st\}\[n\_\{t\}/N\_\{X\}\]\-\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\\;=\\;\-\\,\\mathbb\{E\}\_\{st\}\\\!\\left\[\\frac\{n\_\{t\}\\,\(N\_\{X\}\-1\)\}\{N\_\{X\}\}\\right\],\(26\)
More precisely, the deficit splits into two terms of opposite sign,

𝔼s​t​\[nt/NX\]−𝔼s​t​\[nt\]=𝔼s​t​\[nt\]​\(𝔼s​t​\[1/NX\]−1\)\+Covs​t⁡\(nt,1/NX\)\.\\mathbb\{E\}\_\{st\}\[n\_\{t\}/N\_\{X\}\]\-\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\\;=\\;\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]\\big\(\\mathbb\{E\}\_\{st\}\[1/N\_\{X\}\]\-1\\big\)\\;\+\\;\\operatorname\{Cov\}\_\{st\}\\\!\\big\(n\_\{t\},\\,1/N\_\{X\}\\big\)\.\(27\)The first is a Jensen term:1/NX1/N\_\{X\}is convex, so𝔼s​\[1/NX\]\>1/𝔼s​\[NX\]≈1\\mathbb\{E\}\_\{s\}\[1/N\_\{X\}\]\>1/\\mathbb\{E\}\_\{s\}\[N\_\{X\}\]\\approx 1, pushing toward*over*\-prediction\. For example, in GPT\-2 measured𝔼s​\[1/NX\]\\mathbb\{E\}\_\{s\}\[1/N\_\{X\}\]is1\.361\.36while𝔼s​\[NX\]\\mathbb\{E\}\_\{s\}\[N\_\{X\}\]is0\.990\.99\. The second is negative: bothnt​\(X\)n\_\{t\}\(X\)andNXN\_\{X\}grow with the length of the context, sontn\_\{t\}and1/NX1/N\_\{X\}are anticorrelated, pushing toward*under*\-prediction\. For every bucket whose multiplicity grows with length \(all tracked types, and the “other” bucket\), the covariance term is larger, andA^s​t\\hat\{A\}\_\{st\}under\-predicts\. The one exception is the position\-00bucket, whose multiplicity is identically11: its covariance term vanishes exactly, leaving the pure Jensen term, and it is over\-predicted\.

BothAAandA^\\hat\{A\}are distributions over the same context:∑tAs​t​X=∑tA^s​t​X=1\\sum\_\{t\}A\_\{stX\}=\\sum\_\{t\}\\hat\{A\}\_\{stX\}=1, hence

∑t𝔼s​\[As​t−A^s​t\]=0\\sum\_\{t\}\\mathbb\{E\}\_\{s\}\\\!\\left\[A\_\{st\}\-\\hat\{A\}\_\{st\}\\right\]\\;=\\;0\(28\)exactly, for everyss\. It’s useful to define the per\-cell*relative*bias

rs​t=1−𝔼s​t​\[A^s​t\]/𝔼s​t​\[As​t\]\.r\_\{st\}=1\-\\mathbb\{E\}\_\{st\}\[\\hat\{A\}\_\{st\}\]/\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\.\(29\)This establishes a connection withPP\. The identity \([28](https://arxiv.org/html/2609.16382#A2.E28)\) then reads∑tPs​t​rs​t=0\\sum\_\{t\}P\_\{st\}\\,r\_\{st\}=0, since𝔼s​\[As​t−A^s​t\]=πs​t​𝔼s​t​\[As​t\]​rs​t=Ps​t​rs​t\\mathbb\{E\}\_\{s\}\[A\_\{st\}\-\\hat\{A\}\_\{st\}\]=\\pi\_\{st\}\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]\\,r\_\{st\}=P\_\{st\}\\,r\_\{st\}\. In other words, the relative biases cancel when each is weighted by the average attentionPs​tP\_\{st\}its pair carries\.

The bias is thus a redistribution of attention mass among key types, not a net distortion: because it sums to zero under the samePP\-weighting that governs the mean\-field prediction, it cannot accumulate in the aggregate quantities the paper reports\. §[B\.3](https://arxiv.org/html/2609.16382#A2.SS3)shows the per\-pair bias is also small empirically, and §[B\.4](https://arxiv.org/html/2609.16382#A2.SS4)removes it entirely with a calibrated estimator\.

### B\.3The Measured Bias ofA^\\hat\{A\}is Small

To assess the empirical bias ofA^s​t\\hat\{A\}\_\{st\}, we proceed as follows\. For each corpus position whose query type is tracked, and each head, we computeNXN\_\{X\}over the inclusive causal prefix and accumulate per \(head, type\) the conditional means𝔼s​t​\[nt\]\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]and𝔼s​t​\[nt/NX\]\\mathbb\{E\}\_\{st\}\[n\_\{t\}/N\_\{X\}\]\.171717Corpus:389389chunks of512512OpenWebText tokens \(∼\\sim200k\)\.The quantity𝔼s​t​\[nt/NX\]/𝔼s​t​\[nt\]=𝔼s​t​\[A^s​t\]/𝔼s​t​\[As​t\]\\mathbb\{E\}\_\{st\}\[n\_\{t\}/N\_\{X\}\]\\big/\\mathbb\{E\}\_\{st\}\[n\_\{t\}\]=\\mathbb\{E\}\_\{st\}\[\\hat\{A\}\_\{st\}\]/\\mathbb\{E\}\_\{st\}\[A\_\{st\}\]by \([25](https://arxiv.org/html/2609.16382#A2.E25)\); a ratio11is unbiased\. The results are in Table[6](https://arxiv.org/html/2609.16382#A2.T6)\. The table shows that the unweighted cell average is0\.9460\.946and0\.9330\.933, a55–7%7\\%under\-prediction for the typical pair, with over95%95\\%of pairs within20%20\\%of unbiased in both models\.𝔼s​\[NX\]≈1\\mathbb\{E\}\_\{s\}\[N\_\{X\}\]\\approx 1is a consistency check onPPandn¯\\bar\{n\}\.

Table 6:Observed bias of \(head,type\) pairs\. “Cells within 20%” counts well\-sampled \(head, type\) pairs whose ratio lies in\[0\.8,1\.2\]\[0\.8,1\.2\]: 129k cells for GPT\-2, 187k for Gemma\.Empirically, in GPT\-2 we observe that correlation betweenntn\_\{t\}and\(NX−1\)/NX\(N\_\{X\}\-1\)/N\_\{X\}is quite similar for different token typestt\. Binning types into quartiles by presence frequency \(the number of scored query positions whose causal prefix contained the type\), with the top quartile split at the9090th percentile, every bin’s mean ratio lies in\[0\.93,0\.95\]\[0\.93,0\.95\], with≈11%\{\\approx\}11\\%of cells above11in each\. The compensation that \([28](https://arxiv.org/html/2609.16382#A2.E28)\) requires therefore sits in the untracked bucket: the position\-00bucket hasn=1n=1by definition \(hence no covariance term by \([27](https://arxiv.org/html/2609.16382#A2.E27)\)\), ratio𝔼⁡\[1/NX\]=1\.36\\mathbb\{E\}\[1/N\_\{X\}\]=1\.36, and≈40%\{\\approx\}40\\%of the attention mass\. Thus,A^\\hat\{A\}has one over\-predicted sink balancing a uniformly under\-predicted field\. The same structure holds in Gemma\-2\-2B: tracked types uniform at0\.930\.93–0\.940\.94across every frequency quartile, and the position\-0 sink over\-predicted at1\.221\.22on40%40\\%of the mass\. A separate run comparingA^\\hat\{A\}against the model’s measured attention reproduces the deficit, confirming that \([25](https://arxiv.org/html/2609.16382#A2.E25)\) is the whole story\.

This small bias has negligible effect on our results\. We demonstrate this as follows\. In the next section we show how to construct a truly unbiased estimator ofA^\\hat\{A\}\. We then show that, using this unbiased estimator, the key results in the paper are unchanged\.

### B\.4Using an Unbiased Estimator ofA^\\hat\{A\}Does Not Change Key Results

The bias in \([25](https://arxiv.org/html/2609.16382#A2.E25)\) exists becauseWWis obtained in closed form, as the ratioP/n¯P/\\bar\{n\}of two*marginal*means, while it is*used*inside a nonlinear per\-context renormalization\.

#### Constructing an unbiased estimator ofA^\\hat\{A\}\.

Instead of usingWWin the definition ofA^\\hat\{A\}, we may instead find a matrixUUthat makes \([22](https://arxiv.org/html/2609.16382#A2.E22)\) true by construction\. Multiplying \([22](https://arxiv.org/html/2609.16382#A2.E22)\) byπs​t\\pi\_\{st\}to remove the conditioning, the requirement is

𝔼s​\[A^s​t​\(U\)\]=𝔼s​\[As​t\]=Ps​tfor every​\(s,t\),\\mathbb\{E\}\_\{s\}\\\!\\left\[\\hat\{A\}\_\{st\}\(U\)\\right\]\\;=\\;\\mathbb\{E\}\_\{s\}\[A\_\{st\}\]\\;=\\;P\_\{st\}\\qquad\\text\{for every \}\(s,t\),\(30\)i\.e\.*the kernel must reproduce the measured attention marginals when run through its own renormalization\.*AnyUUsatisfying \([30](https://arxiv.org/html/2609.16382#A2.E30)\) gives𝔼s​t​\[As​t−A^s​t\]=0\\mathbb\{E\}\_\{st\}\[A\_\{st\}\-\\hat\{A\}\_\{st\}\]=0exactly, for every pair, by construction\.

We note that, for each query typess, achieving \([30](https://arxiv.org/html/2609.16382#A2.E30)\) is a marginal\-matching problem\. To see this, observe that for each contextXXcontainingss, \([21](https://arxiv.org/html/2609.16382#A2.E21)\) makesA^s⋅X​\(U\)\\hat\{A\}\_\{s\\cdot X\}\(U\)a probability vector over the key types: typettis offerednt​\(X\)n\_\{t\}\(X\)times, each occurrence carries weightUs​tU\_\{st\}, and the vector is the resulting share\. Averaging those per\-context vectors over the corpus givesQs⋅=𝔼s\[A^s⋅\(U\)\]Q\_\{s\\cdot\}=\\mathbb\{E\}\_\{s\}\[\\hat\{A\}\_\{s\\cdot\}\(U\)\], again a probability vector over key types — the model’s*predicted type marginal*for query typess\. The measuredPs⋅P\_\{s\\cdot\}is the corresponding*observed type marginal*\. So \([30](https://arxiv.org/html/2609.16382#A2.E30)\) asks for the weightsUUwhose induced marginal, after per\-context normalization and averaging, reproduces the observed one\.

Problems of this marginal\-matching form are typically solved by Sinkhorn\-type iterations\([Sinkhorn and Knopp, 1967](https://arxiv.org/html/2609.16382#bib.bib62);[Idel, 2016](https://arxiv.org/html/2609.16382#bib.bib32)\), i\.e\., iterative proportional fitting \(IPF\)\. IPF is appropriate because per\-context normalization∑tA^s​t​X=1\\sum\_\{t\}\\hat\{A\}\_\{stX\}=1is one constraint family, satisfied automatically by construction; the targetQs⋅=Ps⋅Q\_\{s\\cdot\}=P\_\{s\\cdot\}is the other\. We have exactly one free parameterWs​tW\_\{st\}per constraint, so the system is square\. The update is correspondingly multiplicative: if the current kernel puts too little mass on\(s,t\)\(s,t\), scaleUs​tU\_\{st\}up in proportion to the shortfall and let the renormalization redistribute the rest\. Applying IPF to this problem is shown as Algorithm[2](https://arxiv.org/html/2609.16382#algorithm2)\.

Algorithm 2Unbiased calibration ofWW: the estimatorUU\(run per head; heads are independent\)U\(0\)←W=P/n¯U^\{\(0\)\}\\leftarrow W=P/\\bar\{n\};//the paper’s estimator: a good warm start

1for*k=0,1,2,…k=0,1,2,\\dots*do

2

Qs​t←0Q\_\{st\}\\leftarrow 0for all

\(s,t\)\(s,t\);

ms←0m\_\{s\}\\leftarrow 0for all

ss;//corpus accumulators

3foreach*chunk, each query positionppwith tracked typess*do

4

n←\(nt​\(X\)\)tn\\leftarrow\\big\(n\_\{t\}\(X\)\\big\)\_\{t\};//multiplicities of the causal prefix, incl\. buckets

5

at←nt​Us​t\(k\)a\_\{t\}\\leftarrow n\_\{t\}\\,U^\{\(k\)\}\_\{st\}for all

tt;//numerators of \([21](https://arxiv.org/html/2609.16382#A2.E21)\)

6

N←∑tatN\\leftarrow\\sum\_\{t\}a\_\{t\};//the normalizerNXN\_\{X\}

7

Qs​t←Qs​t\+at/NQ\_\{st\}\\leftarrow Q\_\{st\}\+a\_\{t\}/Nfor all

tt;//accumulateA^s​t​X​\(U\(k\)\)\\hat\{A\}\_\{stX\}\(U^\{\(k\)\}\)into rowssonly

8

ms←ms\+1m\_\{s\}\\leftarrow m\_\{s\}\+1;

9foreach*query typess*do//normalize each row by its own occurrence count

10

Qs​t←Qs​t/msQ\_\{st\}\\leftarrow Q\_\{st\}/m\_\{s\}for all

tt;//nowQs​t=𝔼s​\[A^s​t​\(U\(k\)\)\]Q\_\{st\}=\\mathbb\{E\}\_\{s\}\[\\hat\{A\}\_\{st\}\(U^\{\(k\)\}\)\]

11if*maxs,t⁡\|Qs​t−Ps​t\|<ε\\max\_\{s,t\}\\,\|Q\_\{st\}\-P\_\{st\}\|<\\varepsilon*thenstop;

12foreach*pair\(s,t\)\(s,t\)*do//one multiplicative correction per entry

13

Us​t\(k\+1\)←Us​t\(k\)⋅\(Ps​tmax⁡\(Qs​t,δ\)\)αU^\{\(k\+1\)\}\_\{st\}\\leftarrow U^\{\(k\)\}\_\{st\}\\cdot\\Big\(\\dfrac\{P\_\{st\}\}\{\\max\(Q\_\{st\},\\,\\delta\)\}\\Big\)^\{\\alpha\};

#### Algorithm\.

Applying IPF to this problem is shown as Algorithm[2](https://arxiv.org/html/2609.16382#algorithm2)\. Applying a damping exponentα<1\\alpha<1in Algorithm[2](https://arxiv.org/html/2609.16382#algorithm2)is the standard remedy if the residual were to oscillate rather than decrease\. Empirically it is not needed here — withα=1\\alpha=1the residual converges in three iterations with no oscillation, which is the behavior expected of a contraction\.

#### Using an unbiased estimator ofA^\\hat\{A\}does not change key results\.

At a fixed point of Algorithm[2](https://arxiv.org/html/2609.16382#algorithm2),Q=PQ=Pso \([30](https://arxiv.org/html/2609.16382#A2.E30)\) holds and the bias \([25](https://arxiv.org/html/2609.16382#A2.E25)\) is zero for every\(s,t\)\(s,t\)\. Unbiasedness ofA^​\(U\)\\hat\{A\}\(U\)is then true by construction\. Consequently, we can recompute any body result that uses the context\-conditional kernel with a calibratedUUin place ofW=P/n¯W=P/\\bar\{n\}, holding everything else fixed\. We now do that for four key results: the head ranking of §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3.SSS0.Px1), the training\-dynamics gap of §[5](https://arxiv.org/html/2609.16382#S5), the ICL prompt measurements, and the instance\-level meter\.

We first build a calibratedUUat22M tokens\. GPT\-2 was calibrated at its published weights and Pythia\-160m at training step10001000, each against its ownPPandn¯\\bar\{n\}\. Table[7](https://arxiv.org/html/2609.16382#A2.T7)shows the convergence properties of Algorithm[2](https://arxiv.org/html/2609.16382#algorithm2), noting that good convergence was achieved in 5 iterations for both cases\. We note that the ratioU/W\(0\)U/W^\{\(0\)\}over the67\.667\.6M fitted GPT\-2 cells has a55–95%95\\%range of\[0\.76,1\.29\]\[0\.76,\\,1\.29\]and a median of1\.0051\.005\.

Table 7:Calibration at22M tokens \(39143914chunks of512512\), five iterations,α=1\\alpha=1\. Iteration00is the residual of the present estimatorW=P/n¯W=P/\\bar\{n\}\.Recomputing key results, we find no change to any of our findings\. Specifically:

Training Dynamics of §[5](https://arxiv.org/html/2609.16382#S5):We chose the data point for Pythia\-160m at step10001000as this lies just at the point where mean\-field model performance begins to deviate from true model performance, and applied the calibratedUUfor that model\. At that point,LMFL\_\{\\text\{MF\}\}is unchanged to three significant figures \(invisible on the plot\), andΔ​L\\Delta LandDMFD\_\{\\mathrm\{MF\}\}are also unchanged to three significant figures\.

Experiment 1 of §[6\.1](https://arxiv.org/html/2609.16382#S6.SS1):Under the calibrated kernelUU, every element of the figure survives: the deviation rises across the copy boundary, the single head whose deviation rises most is L5H1, and the in\-context loss reduction increases monotonically with the induced deviation from the first decile to the tenth\.

Experiment 2 of §[6\.1](https://arxiv.org/html/2609.16382#S6.SS1):All values in Table[2](https://arxiv.org/html/2609.16382#S6.T2)are unchanged to two significant figures\.

Head Ranking of §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3.SSS0.Px1):The context\-averaged context\-conditional mean field was recomputed using the calibrated kernelUUfor all144144GPT\-2 heads\. Comparing to Table[4](https://arxiv.org/html/2609.16382#S6.T4), we find the new ranking has Spearman correlation of 0\.9967, the maximum shift in rank over all heads is 4, and 19 out of the top 20 heads as shown in the table are present in the re\-ranked top 20\.

The paper’s other results \(§[4](https://arxiv.org/html/2609.16382#S4)and §[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)\) use the kernelPP, so are not affected by bias inA^\\hat\{A\}\. We thus conclude that not only is the bias ofA^\\hat\{A\}small, as shown here, but that small bias does not affect any of the key results in the paper\.

### B\.5Derivation of the Covariance / Contextualization Split

Next we justify the split of the context\-conditional mean field into its covariance and contextualization terms \([15](https://arxiv.org/html/2609.16382#S6.E15)\)\.

Fix a head\(ℓ,h\)\(\\ell,h\)and a contextXXwith query typess, and writeAs​tA\_\{st\}forAs​t​XA\_\{stX\}as elsewhere in this appendix; all sums run over the present typest∈supp⁡\(X\)t\\in\\supp\(X\)\. The head’s actual contribution to the query is𝐳=∑tAs​t​𝐯¯t\\mathbf\{z\}=\\sum\_\{t\}A\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}, withAs​tA\_\{st\}the attention mass on typettand𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}the in\-context attention\-weighted mean value of the type\-ttkeys\. The context\-conditional mean\-field replaces the attention by the renormalized corpus kernelA^s​t\\hat\{A\}\_\{st\}\([21](https://arxiv.org/html/2609.16382#A2.E21)\) and the in\-context value by the corpus type\-mean𝝁t\\boldsymbol\{\\mu\}\_\{t\}\(which Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px1)equates, to good approximation, with𝔼s​t​\[𝐯¯t\]\\mathbb\{E\}\_\{st\}\[\\overline\{\\mathbf\{v\}\}\_\{t\}\]\), giving𝐳^=∑tA^s​t​𝝁t\\hat\{\\mathbf\{z\}\}=\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}\. The prediction thus differs from the actual output in two factors at once: the attention weights \(As​tA\_\{st\}versusA^s​t\\hat\{A\}\_\{st\}\) and the values \(𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}versus𝝁t\\boldsymbol\{\\mu\}\_\{t\}\)\. To separate the two changes, insert the*mixed*term∑tA^s​t​𝐯¯t\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}— predicted attention on actual values — which changes one factor at a time\. Adding and subtracting this mixed term,

𝐳−𝐳^\\displaystyle\\mathbf\{z\}\-\\hat\{\\mathbf\{z\}\}=∑tAs​t​𝐯¯t−∑tA^s​t​𝝁t\\displaystyle=\\sum\_\{t\}A\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\-\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}=∑tAs​t​𝐯¯t​−∑tA^s​t𝐯¯t\+∑tA^s​t𝐯¯t⏟=0−∑tA^s​t​𝝁t\\displaystyle=\\sum\_\{t\}A\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\;\\underbrace\{\-\\,\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\;\+\\,\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\}\_\{=\\,0\}\\;\-\\,\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}=\(∑tAs​t​𝐯¯t−∑tA^s​t​𝐯¯t\)⏟covariance term\+\(∑tA^s​t​𝐯¯t−∑tA^s​t​𝝁t\)⏟contextualization term\\displaystyle=\\underbrace\{\\Big\(\\sum\_\{t\}A\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\-\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\Big\)\}\_\{\\text\{covariance term\}\}\+\\underbrace\{\\Big\(\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\-\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\boldsymbol\{\\mu\}\_\{t\}\\Big\)\}\_\{\\text\{contextualization term\}\}=∑t\(As​t−A^s​t\)​𝐯¯t\+∑tA^s​t​\(𝐯¯t−𝝁t\)\.\\displaystyle=\\sum\_\{t\}\(A\_\{st\}\-\\hat\{A\}\_\{st\}\)\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\+\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,\\big\(\\overline\{\\mathbf\{v\}\}\_\{t\}\-\\boldsymbol\{\\mu\}\_\{t\}\\big\)\.\(31\)
#### Why we refer to the first term as a contribution to covariance\.

Written for a single contextXXit is∑t\(As​t−A^s​t\)​𝐯¯t\\sum\_\{t\}\(A\_\{st\}\-\\hat\{A\}\_\{st\}\)\\overline\{\\mathbf\{v\}\}\_\{t\}, a realization rather than a covariance\. Its meaning is fixed by the ensemble the kernel is calibrated on: all contexts having query typess\.

As shown above, Equation[22](https://arxiv.org/html/2609.16382#A2.E22)approximately holds for our data, and the empirical deviations from it do not affect our key results\. Hence, taking the expectation of a term of the first group over the contexts with query typessin whichttappears,

𝔼s​t​\[\(As​t−A^s​t\)​𝐯¯t\]=Covs​t⁡\(As​t−A^s​t,𝐯¯t\)\+𝔼s​t​\[As​t−A^s​t\]⏟≈0​𝔼s​t​\[𝐯¯t\]≈Covs​t⁡\(As​t−A^s​t,𝐯¯t\),\\mathbb\{E\}\_\{st\}\\\!\\big\[\(A\_\{st\}\-\\hat\{A\}\_\{st\}\)\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\]=\\operatorname\{Cov\}\_\{st\}\\\!\\big\(A\_\{st\}\-\\hat\{A\}\_\{st\},\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\)\+\\underbrace\{\\mathbb\{E\}\_\{st\}\\big\[A\_\{st\}\-\\hat\{A\}\_\{st\}\\big\]\}\_\{\\approx\\,0\}\\,\\mathbb\{E\}\_\{st\}\\big\[\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\]\\;\\approx\\;\\operatorname\{Cov\}\_\{st\}\\\!\\big\(A\_\{st\}\-\\hat\{A\}\_\{st\},\\,\\overline\{\\mathbf\{v\}\}\_\{t\}\\big\),\(32\)and the corpus constant𝝁t\\boldsymbol\{\\mu\}\_\{t\}may equally be subtracted from𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}inside the covariance, for the same reason\. So the first group is, in expectation over the query\-type ensemble, the per\-type covariance between the head’s*residual*attention — the part ofAs​tA\_\{st\}that presence and corpus propensity, i\.e\. the kernel, do not predict — and the value it thereby reads\. This is the quantity Assumption[3\.3](https://arxiv.org/html/2609.16382#S3.SS3.SSS0.Px2)sets to zero: under proportionalityAs​t=A^s​tA\_\{st\}=\\hat\{A\}\_\{st\}identically, and the covariance vanishes term by term\. More generally it vanishes iff, across contexts sharing the query type, how much*more or less than predicted*the head attends to typettis uncorrelated with*what*it reads there: i\.e\. the head transports content fixed by token type, not selected by context\. A single context cannot exhibit a covariance, but a large first term signals thatXXsits in the tail of this ensemble covariance, which is why we describe its magnitude as the context’s contribution to covariance\.

The second group \(contextualization\) instead varies the*values*at fixed kernel weights: it is nonzero when the in\-context type\-ttvalue𝐯¯t\\overline\{\\mathbf\{v\}\}\_\{t\}departs from its corpus average𝝁t\\boldsymbol\{\\mu\}\_\{t\}, i\.e\. when the tokens the head reads for typettcarry context\-specific content\. Had we instead used the paper’s full mean field𝐳^′=∑tA^s​t​v​\(𝐜t\)\\hat\{\\mathbf\{z\}\}^\{\\prime\}=\\sum\_\{t\}\\hat\{A\}\_\{st\}\\,v\(\\mathbf\{c\}\_\{t\}\)of \([5](https://arxiv.org/html/2609.16382#S3.E5)\), a third value\-map \(Jensen\) group∑tA^s​t​\(𝝁t−v⁡\(𝐜t\)\)\\sum\_\{t\}\\hat\{A\}\_\{st\}\\big\(\\boldsymbol\{\\mu\}\_\{t\}\-v\(\\mathbf\{c\}\_\{t\}\)\\big\)would appear; using the measured𝝁t\\boldsymbol\{\\mu\}\_\{t\}removes it, and it is small in any case \(Appendix[C](https://arxiv.org/html/2609.16382#A3)\)\. The metricsMcov,MctxM\_\{\\mathrm\{cov\}\},M\_\{\\mathrm\{ctx\}\}of \([16](https://arxiv.org/html/2609.16382#S6.E16)\) resolve the two groups of \([31](https://arxiv.org/html/2609.16382#A2.E31)\) along the component of the error orthogonal to𝐳^\\hat\{\\mathbf\{z\}\}— signed shares that sum to one — and scale by the deviationDs,Xℓ​hD^\{\\ell h\}\_\{s,X\}, so per head the two meters sum to the deviation exactly; the network\-level meters average these per\-head quantities over heads and layers\. The weighting of the value drift is byA^s​t\\hat\{A\}\_\{st\}, as shown in \([31](https://arxiv.org/html/2609.16382#A2.E31)\)\.

## Appendix CImpact of the Value\-Map Mean\-Field Approximation

This appendix asks what is lost by evaluating the value map at a type’s centroid rather than averaging it over the type’s occurrences\. We find that little is lost: occurrences of a type sit close enough to their centroid that the two agree\.

The argument has two parts\. The error is exactly the Jensen gap of the attention block’s input normalizer, so it is governed by the within\-type covariance; and that covariance is small enough, at every layer of both models, that𝐯¯t≈v⁡\(𝐜t\)\\bar\{\\mathbf\{v\}\}\_\{t\}\\approx v\(\\mathbf\{c\}\_\{t\}\)\.

The corpus mean field \(Section[3\.2](https://arxiv.org/html/2609.16382#S3.SS2)\) replaces the per\-occurrence valuev⁡\(𝐱t\)v\(\\mathbf\{x\}\_\{t\}\)by the value of the type centroidv⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\), i\.e\. it assumes𝐯¯t=𝔼⁡\[v⁡\(𝐱t\)\]≈v⁡\(𝐜t\)\\bar\{\\mathbf\{v\}\}\_\{t\}=\\mathbb\{E\}\[v\(\\mathbf\{x\}\_\{t\}\)\]\\approx v\(\\mathbf\{c\}\_\{t\}\)\. Because the per\-head value mapv⁡\(𝐱\)=ln1​\(𝐱\)​WV\+𝐛Vv\(\\mathbf\{x\}\)=\\mathrm\{ln\}\_\{1\}\(\\mathbf\{x\}\)\\,W\_\{V\}\+\\mathbf\{b\}\_\{V\}has the input normalizerln1\\mathrm\{ln\}\_\{1\}\(LayerNorm in GPT\-2, RMSNorm in Gemma\-2\) as its only nonlinearity \(andWVW\_\{V\}is linear, so it commutes with the expectation\), the error is exactly the Jensen gap ofln1\\mathrm\{ln\}\_\{1\}over the type\-ttoccurrence cloud,

𝐯¯t−v⁡\(𝐜t\)=\(𝔼⁡\[ln1​\(𝐱t\)\]−ln1​\(𝐜t\)\)​WV\.\\bar\{\\mathbf\{v\}\}\_\{t\}\-v\(\\mathbf\{c\}\_\{t\}\)=\\big\(\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\(\\mathbf\{x\}\_\{t\}\)\]\-\\mathrm\{ln\}\_\{1\}\(\\mathbf\{c\}\_\{t\}\)\\big\)\\,W\_\{V\}\.A second\-order \(delta\-method\) expansion\([Oehlert, 1992](https://arxiv.org/html/2609.16382#bib.bib47);[Gao et al\., 2019b](https://arxiv.org/html/2609.16382#bib.bib22)\)gives𝔼⁡\[ln1​\(𝐱t\)\]−ln1​\(𝐜t\)≈12​tr​\(∇2ln1​\(𝐜t\)​Cov​\(𝐱t\)\)\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\(\\mathbf\{x\}\_\{t\}\)\]\-\\mathrm\{ln\}\_\{1\}\(\\mathbf\{c\}\_\{t\}\)\\approx\\tfrac\{1\}\{2\}\\,\\mathrm\{tr\}\\\!\\big\(\\nabla^\{2\}\\mathrm\{ln\}\_\{1\}\(\\mathbf\{c\}\_\{t\}\)\\,\\mathrm\{Cov\}\(\\mathbf\{x\}\_\{t\}\)\\big\), where the LayerNorm Jacobian and Hessian follow fromln1​\(𝐱\)=d​P​𝐱/‖P​𝐱‖\\mathrm\{ln\}\_\{1\}\(\\mathbf\{x\}\)=\\sqrt\{d\}\\,P\\mathbf\{x\}/\\\|P\\mathbf\{x\}\\\|with the mean\-centering projectionP=I−1d​𝟏𝟏⊤P=I\-\\tfrac\{1\}\{d\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\([Ba et al\., 2016](https://arxiv.org/html/2609.16382#bib.bib3);[Brody et al\., 2023](https://arxiv.org/html/2609.16382#bib.bib9)\)\(RMSNorm omitsPP,ln1​\(𝐱\)=d​𝐱/‖𝐱‖\\mathrm\{ln\}\_\{1\}\(\\mathbf\{x\}\)=\\sqrt\{d\}\\,\\mathbf\{x\}/\\\|\\mathbf\{x\}\\\|, but the argument is unchanged\)\. The gap is therefore governed by the*within\-type covariance*Cov⁡\(𝐱t\)\\mathrm\{Cov\}\(\\mathbf\{x\}\_\{t\}\): if occurrences of a type cluster tightly around their centroid, the value\-map mean field is exact to second order\. We verify this clustering directly\.

Within\-type clustering is the Euclidean counterpart of the contextual self\-similarity of[Ethayarajh \(2019\)](https://arxiv.org/html/2609.16382#bib.bib19), who studied mean pairwise cosine across a word’s occurrences, which they found decreases with depth\.

#### Metrics\.

Write\{𝐱t,i\}i=1nt\\\{\\mathbf\{x\}\_\{t,i\}\\\}\_\{i=1\}^\{n\_\{t\}\}for the residual\-stream representations of thentn\_\{t\}occurrences of typettat a given depth, with type centroid𝐜t=1nt​∑i𝐱t,i\\mathbf\{c\}\_\{t\}=\\tfrac\{1\}\{n\_\{t\}\}\\sum\_\{i\}\\mathbf\{x\}\_\{t,i\}and global centroid𝐜¯=1∑tnt​∑t,i𝐱t,i\\bar\{\\mathbf\{c\}\}=\\tfrac\{1\}\{\\sum\_\{t\}n\_\{t\}\}\\sum\_\{t,i\}\\mathbf\{x\}\_\{t,i\}\. We report three quantities per layer\.*\(i\) Relative RMS deviation:*rt=1nt​∑i‖𝐱t,i−𝐜t‖2/‖𝐜t‖r\_\{t\}=\\sqrt\{\\tfrac\{1\}\{n\_\{t\}\}\\sum\_\{i\}\\\|\\mathbf\{x\}\_\{t,i\}\-\\mathbf\{c\}\_\{t\}\\\|^\{2\}\}\\,/\\,\\\|\\mathbf\{c\}\_\{t\}\\\|, the root\-mean\-square spread of a type’s occurrences about its centroid, relative to the centroid magnitude\.*\(ii\) Variance explained by type,η2\\eta^\{2\}:*a one\-way variance decomposition \(the intraclass correlation\)\. With total varianceVtot=1∑nt​∑t,i‖𝐱t,i−𝐜¯‖2V\_\{\\mathrm\{tot\}\}=\\tfrac\{1\}\{\\sum n\_\{t\}\}\\sum\_\{t,i\}\\\|\\mathbf\{x\}\_\{t,i\}\-\\bar\{\\mathbf\{c\}\}\\\|^\{2\}and pooled within\-type varianceVw=1∑nt​∑t,i‖𝐱t,i−𝐜t‖2V\_\{\\mathrm\{w\}\}=\\tfrac\{1\}\{\\sum n\_\{t\}\}\\sum\_\{t,i\}\\\|\\mathbf\{x\}\_\{t,i\}\-\\mathbf\{c\}\_\{t\}\\\|^\{2\},

η2=1−Vw/Vtot,\\eta^\{2\}=1\-V\_\{\\mathrm\{w\}\}/V\_\{\\mathrm\{tot\}\},the fraction of variance attributable to type identity \(η2→1\\eta^\{2\}\\\!\\to\\\!1means occurrences collapse onto their centroids;η2→0\\eta^\{2\}\\\!\\to\\\!0means type identity carries no information\)\.*\(iii\) Value\-map Jensen gap:*the relative magnitude‖𝔼⁡\[ln1​𝐱t\]−ln1​𝐜t‖/‖ln1​𝐜t‖\\\|\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\\mathbf\{x\}\_\{t\}\]\-\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\\\|/\\\|\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\\\|and the direction1−cos⁡\(𝔼⁡\[ln1​𝐱t\],ln1​𝐜t\)1\-\\cos\(\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\\mathbf\{x\}\_\{t\}\],\\,\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\)— the actual size and orientation of the gap that the second\-order argument above bounds\. All quantities are accumulated in a single pass over 2M OpenWebText tokens, per type and per residual depthE⁡\[ℓ\]E\[\\ell\]\(the input to each block’s attention\)\.

#### Rogue\-dimension control\.

As discussed in more detail in Appendix[I](https://arxiv.org/html/2609.16382#A9), a handful of coordinates — “rogue dimensions” — carry a disproportionate share of the residual variance\.[Timkey and van Schijndel \(2021\)](https://arxiv.org/html/2609.16382#bib.bib66)showed that for GPT\-2’s final layer a single dimension accounts for∼76%\{\\sim\}76\\%of the expected cosine between two random tokens\. A variance\-weighted statistic, including theη2\\eta^\{2\}above \(which sums variance across coordinates\), is therefore dominated by whether those few dimensions separate types, and is nearly blind to the structure in the remaining∼760\{\\sim\}760\. We accordingly also report a per\-dimension\-standardized version: writingσd2\\sigma^\{2\}\_\{d\}andwdw\_\{d\}for the total and pooled\-within variance of coordinatedd, we average the*per\-coordinate*ratio,

ηz2=1d​∑d\(1−wd/σd2\),\\eta^\{2\}\_\{z\}=\\tfrac\{1\}\{d\}\\textstyle\\sum\_\{d\}\\big\(1\-w\_\{d\}/\\sigma^\{2\}\_\{d\}\\big\),which weights every dimension equally — equivalent to z\-scoring each coordinate to unit variance before the decomposition — so that no single rogue dimension dominates\. \(The rawη2=1−∑dwd/∑dσd2\\eta^\{2\}=1\-\\sum\_\{d\}w\_\{d\}/\\sum\_\{d\}\\sigma^\{2\}\_\{d\}is its variance\-weighted counterpart\.\) This is the standardization control recommended by[Timkey and van Schijndel \(2021\)](https://arxiv.org/html/2609.16382#bib.bib66)\.

#### Results\.

Figure[13](https://arxiv.org/html/2609.16382#A3.F13)reports the results for both models\. In GPT\-2 the rogue\-dimension\-robust measure shows type identity explainingηz2=0\.90\\eta^\{2\}\_\{z\}=0\.90of the variance at the model input: occurrences are very tightly clustered around their type centroid relative to the between\-type spread\. The rawη2\\eta^\{2\}*understates*this at0\.400\.40— a few rogue dimensions obscure the within\-type structure, as documented by[Timkey and van Schijndel \(2021\)](https://arxiv.org/html/2609.16382#bib.bib66)\. Both measures decline with depth \(toηz2≈0\.25\\eta^\{2\}\_\{z\}\\approx 0\.25by layer 11\), the geometric form of the increasing context\-specificity of[Ethayarajh \(2019\)](https://arxiv.org/html/2609.16382#bib.bib19)\.

Gemma\-2 behaves similarly:ηz2=0\.85\\eta^\{2\}\_\{z\}=0\.85at the first block input falling to≈0\.15\{\\approx\}0\.15in the deep layers\. There are two aspects specific to Gemma\-2\. First, its input residualE⁡\[0\]E\[0\]is*exactly*degenerate \(η2=1\\eta^\{2\}=1, zero within\-type spread\): RoPE injects position inside the attention computation rather than adding it to the residual, so a type’s input representation is identical across occurrences, whereas GPT\-2’s additive positional embedding already spreads a type’s occurrences at the input\. Second, for Gemma the z\-scoredηz2\\eta^\{2\}\_\{z\}sits slightly*below*the rawη2\\eta^\{2\}rather than above it \(e\.g\.0\.320\.32vs0\.520\.52atE⁡\[4\]E\[4\]\): its high\-variance coordinates are themselves type\-discriminative, so equalizing dimensions removes information rather than noise — the opposite of GPT\-2 — though the clustering stays strong under both measures\.

The impact of this tight clustering for the mean field is the same in both models: the value\-map Jensen gap is negligible at every layer \(right\-hand side of the figure\)\. In terms of cosine distance, in GPT\-2 \(using LayerNorm\),1−cos⁡\(𝔼⁡\[ln1​𝐱\],ln1​𝐜t\)<3×10−51\-\\cos\(\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\\mathbf\{x\}\],\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\)<3\\times 10^\{\-5\}, and in Gemma\-2 \(using RMSNorm\),<3×10−4<3\\times 10^\{\-4\}\. \(These are invisible on the plots\.\) Thus, in both models,𝐯¯t\\bar\{\\mathbf\{v\}\}\_\{t\}andv⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\)are essentially parallel, and have a residual magnitude gap of order1010–28%28\\%\. The clustering is thus tight enough that replacing𝐯¯t\\bar\{\\mathbf\{v\}\}\_\{t\}byv⁡\(𝐜t\)v\(\\mathbf\{c\}\_\{t\}\)is an excellent approximation, particularly in direction — the empirical condition behind the corpus mean field\.

Figure 13:Same\-type token representations and their type centroids \(2M OpenWebText tokens, top\-1000 types\)\.Top row:GPT\-2;bottom row:Gemma\-2\-2B\.\(A\)relative RMS deviation of occurrences around their type centroid,𝔼​‖𝐱−𝐜t‖2/‖𝐜t‖\\sqrt\{\\mathbb\{E\}\\\|\\mathbf\{x\}\-\\mathbf\{c\}\_\{t\}\\\|^\{2\}\}/\\\|\\mathbf\{c\}\_\{t\}\\\|\(median; IQR band over types\)\.\(B\)fraction of variance explained by type identity,η2\\eta^\{2\}: raw \(gray\) and per\-dimension z\-scored \(blue\)\.\(C\)the value\-map Jensen gap‖𝔼⁡\[ln1​𝐱\]−ln1​𝐜t‖/‖ln1​𝐜t‖\\\|\\mathbb\{E\}\[\\mathrm\{ln\}\_\{1\}\\mathbf\{x\}\]\-\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\\\|/\\\|\\mathrm\{ln\}\_\{1\}\\mathbf\{c\}\_\{t\}\\\|\(green\) and its direction1−cos1\-\\cos\(purple\)\.

## Appendix DMean\-Field Kernel Details

The mean\-field rollout tracks theNNmost frequently occurring types,𝒯N⊂𝒯\\mathcal\{T\}\_\{N\}\\subset\\mathcal\{T\}\. So that each query’s attention remains normalized, the kernel for each head carries two key columns beyond theNNtracked types: aBOS/position\-0column and anothercolumn that aggregates every type not in𝒯N\\mathcal\{T\}\_\{N\}\. ThusPℓ​hP^\{\\ell h\}is anN×\(N\+2\)N\\times\(N\{\+\}2\)row\-stochastic matrix: for a query of typess,

∑t∈𝒯NPℓ​hs​t\+Pℓ​hs∙\+Pℓ​hs∘=1,\\sum\_\{t\\in\\mathcal\{T\}\_\{N\}\}P^\{\\ell h\}\_\{st\}\\;\+\\;P^\{\\ell h\}\_\{s\\bullet\}\\;\+\\;P^\{\\ell h\}\_\{s\\circ\}\\;=\\;1,wherePℓ​hs∙P^\{\\ell h\}\_\{s\\bullet\}andPℓ​hs∘P^\{\\ell h\}\_\{s\\circ\}are the corpus\-averaged attention masses a type\-ssquery places on position00and on positions holding non\-tracked types, respectively, defined exactly as the tracked massesPs​tℓ​h=𝔼occ​\[As​t\]P^\{\\ell h\}\_\{st\}=\\mathbb\{E\}\_\{\\mathrm\{occ\}\}\[A\_\{st\}\]\. The per\-head contribution of \([5](https://arxiv.org/html/2609.16382#S3.E5)\) \(withv⁡\(⋅\)v\(\\cdot\)the head’s value map\) is then

𝐳sℓ​h=∑t∈𝒯NPs​tℓ​hv\(𝐜^tℓ\)\+Ps∙ℓ​hv\(𝐛ℓ\)\+Ps∘ℓ​hv\(𝐜¯ℓ\)\.\\mathbf\{z\}^\{\\ell h\}\_\{s\}=\\sum\_\{t\\in\\mathcal\{T\}\_\{N\}\}P^\{\\ell h\}\_\{st\}\\,v\(\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{t\}\)\\;\+\\;P^\{\\ell h\}\_\{s\\bullet\}\\,v\(\\mathbf\{b\}^\{\\ell\}\)\\;\+\\;P^\{\\ell h\}\_\{s\\circ\}\\,v\(\\bar\{\\mathbf\{c\}\}^\{\\ell\}\)\.
#### BOS/position\-0 column\.

Position00of every input holds the BOS token, and under the causal mask it attends only to itself, so its representation has no occurrence\-to\-occurrence variation\.𝐛ℓ\\mathbf\{b\}^\{\\ell\}is that layer\-ℓ\\ellrepresentation: a deterministic function of the model weights, obtained from a single forward pass on the one\-token input\[BOS\]\[\\mathrm\{BOS\}\]\(equivalently, the corpus mean of position\-0 representations\)\. It is therefore not a measured contextual quantity\. Position00is also where the models place a large share of their attention, the attention sink of Appendix[E](https://arxiv.org/html/2609.16382#A5), which is why this column is kept separate rather than folded into “other”\. Every input is prepended with the model’s BOS token — or, for models with no dedicated BOS such as GPT\-2, its surrogate<\|endoftext\|\>— following the TransformerLens convention\. In models with a true BOS the token can be frequent enough to enter𝒯N\\mathcal\{T\}\_\{N\}itself \(Gemma\-2’s<bos\>does\), and it must then be excluded as a*query*type from thePPandWWmatrices: it occurs only at position00, where the causal prefix contains nothing else to attend to, so its kernel row is empty and its renormalized attention undefined\.

#### Other column\.

The non\-tracked types are collapsed into a single column whose value we set to𝐜¯ℓ=1N​∑t∈𝒯N𝐜^tℓ\\bar\{\\mathbf\{c\}\}^\{\\ell\}=\\tfrac\{1\}\{N\}\\sum\_\{t\\in\\mathcal\{T\}\_\{N\}\}\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{t\}, the running mean of the predicted*tracked*centroids at layerℓ\\ell\. This is internal to the rollout and requires no measurement\. Consequently the rollout consults no measured residual at any layer: the tracked centroids𝐜^tℓ\\hat\{\\mathbf\{c\}\}^\{\\ell\}\_\{t\}evolve from the input embeddings, the BOS value𝐛ℓ\\mathbf\{b\}^\{\\ell\}is a weight\-derived constant, and the “other” value𝐜¯ℓ\\bar\{\\mathbf\{c\}\}^\{\\ell\}is the running centroid mean\.

#### Held\-out evaluation\.

All comparisons in Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)are out\-of\-sample\. We split OpenWebText into two disjoint halves by document parity \(even\-indexed documents versus odd\),55M tokens each, and estimate every corpus quantity the model uses — the mean\-field attention kernelPℓ​hP^\{\\ell h\}, the co\-occurrence statistics, and the per\-type value and centroid statistics — on the*train*half, while measuring the true per\-layer centroids𝐜tℓ\\mathbf\{c\}^\{\\ell\}\_\{t\}that the rollout is scored against on the disjoint*test*half\. The rollout therefore never sees the evaluation centroids during estimation\.

#### Estimation ofWW\.

BecauseWs​t=Ps​t/n¯s​tW\_\{st\}=P\_\{st\}/\\bar\{n\}\_\{st\}is a ratio of estimated quantities, its entries are noise\-dominated — and can be arbitrarily large — wherever the pair\(s,t\)\(s,t\)is rarely observed\. We therefore restrictWWto its well\-estimated support: entries whose co\-occurrence countCs​tC\_\{st\}\(the number ofss\-occurrences with at least onettin the causal prefix\) falls below2020are set to zero, andA^s​t​X\\hat\{A\}\_\{stX\}renormalises over the remaining support\. The mask discards53%53\\%of\(s,t\)\(s,t\)pairs but only3%3\\%of attention mass \(Gemma\-2:41%41\\%and5%5\\%\), and all context\-conditional results below are computed under it\.

## Appendix ECoverage of the Tracked Type Set

Every construction in the body of this paper tracks theN=1000N=1000most frequent token types, with everything else collapsed into a position\-00bucket and an “other” bucket \(Appendix[D](https://arxiv.org/html/2609.16382#A4)\)\. How much of the corpus that set covers depends on what is counted: about two thirds of corpus positions carry a tracked type, while the share of attention mass landing on tracked types is far lower,1818–42%42\\%, almost all of the difference attributable to the attention sink at position00\. The choice ofNNitself matters less than either figure suggests: Appendix[M](https://arxiv.org/html/2609.16382#A13)repeats the training\-waypoint substitution atN=2000N=2000andN=4000N=4000and finds the phase structure unchanged\.

#### Position coverage\.

Since each position defines one context \(Section[3\.1](https://arxiv.org/html/2609.16382#S3.SS1)\), corpus positions whose type is tracked equate to contexts for which the mean field has a tracked query type\. The fraction of positions covered is the relevant measure wherever the substitution is applied only at tracked query types\. The remaining positions keep the model’s own computation\. Measured over11M OpenWebText tokens with each model’s own tokenizer and its own top\-1000 set \(Table[8](https://arxiv.org/html/2609.16382#A5.T8)\):

Table 8:Position coverage: the fraction of corpus positions \(11M OpenWebText tokens\) whose type is among the model’s top\-10001000set\.Coverage of positions sits at6464–68%68\\%across a fivefold spread in vocabulary size\.10001000types out of Gemma\-2’s256,000256\{,\}000cover as much text as10001000out of GPT\-2’s50,25750\{,\}257\. Token frequency is Zipfian, so the head of the distribution is the same text however finely the tail is divided\. The two Llamas share a tokenizer, and so share a set and a coverage\.

Figure[14](https://arxiv.org/html/2609.16382#A5.F14)shows why\. The rank–frequency curves of the five tokenizers, measured on the same22M tokens of OpenWebText, lie almost on top of one another over the tracked range, and the rank\-10001000cutoff falls at nearly the same frequency in every model \(0\.960\.96–1\.10×10−41\.10\\times 10^\{\-4\}\)\. They separate only beyond rank∼104\{\\sim\}10^\{4\}, where the larger vocabularies keep resolving types the smaller ones have merged, which is the region the tracked set excludes\. Taking the top10001000types therefore selects the same text in each model, not just the same number of types\.

Figure 14:Rank–frequency curves for the five tokenizers on22M tokens of OpenWebText, log–log, one curve per tokenizer; the two Llamas share a tokenizer and so a curve\. Dashed line: the rank\-10001000cutoff used throughout\. The curves are nearly indistinguishable up to the cutoff and for a decade beyond it, diverging only in the far tail where vocabulary size decides how finely rare text is split\.
#### Attention\-mass coverage\.

The second quantity is how much of the attention a head places lands on tracked types, as opposed to the position\-00bucket or the “other” bucket\. This is the row sum∑t∈𝒯NPs​tℓ​h\\sum\_\{t\\in\\mathcal\{T\}\_\{N\}\}P^\{\\ell h\}\_\{st\}, the share of query typess’s attention that reaches tracked key types, averaged over layers, heads and tracked query types \(Table[9](https://arxiv.org/html/2609.16382#A5.T9)\)\.

Table 9:Attention\-mass coverage: average per\-query attention landing on tracked types, the position\-00sink, and the “other” bucket\. The final column is the tracked share of non\-sink attention, computed from the row entries; it is the quantity comparable to the position coverage of Table[8](https://arxiv.org/html/2609.16382#A5.T8)\.†The Llama\-3\.1\-8B row is measured on the reduced\-vocabulary extraction of Appendix[J](https://arxiv.org/html/2609.16382#A10), which excludes6767structural types that are both frequent and heavily attended, so its tracked share is understated\. Measured on that same reduced basis the Llama\-3\.2\-3B figure falls from0\.2310\.231to0\.1820\.182, against the 8B’s0\.1810\.181, so on a like\-for\-like footing the two models agree, and the 8B’s full\-vocabulary value should be read as≈0\.23\{\\approx\}0\.23\. The final column inherits the same understatement, so the 8B’s0\.5230\.523is not comparable to the full\-vocabulary rows above it\.

The largest single destination in every model is the sink: position 0 takes40%40\\%of all attention in GPT\-2 and Gemma\-2\. It takes49%49\\%in Qwen\-3 and65%65\\%in both Llamas, consistent with the attention\-sink literature\([Xiao et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib72);[Gu et al\., 2025](https://arxiv.org/html/2609.16382#bib.bib28)\)\. The sink also accounts for nearly all of the gap between the two tables\. Raw attention\-mass coverage sits far below position coverage,1818–42%42\\%against6464–68%68\\%, but the raw number charges the tracked set with sink mass that no choice of tracked types could capture\. The comparable quantity is the tracked share of*non\-sink*attention, the final column of Table[9](https://arxiv.org/html/2609.16382#A5.T9), labeled tracked/\(tracked\+other\), which is6767–72%72\\%for every model measured on its full vocabulary\. So when heads attend to token content at all, they place attention on tracked types at slightly above the tracked set’s share of positions, and the shortfall in the raw number belongs to the sink\.

#### What the untracked mass costs\.

Attention that leaves the tracked set still enters the prediction, through the two bucket valuesv⁡\(𝐛ℓ\)v\(\\mathbf\{b\}^\{\\ell\}\)andv⁡\(𝐜¯ℓ\)v\(\\bar\{\\mathbf\{c\}\}^\{\\ell\}\)of Appendix[D](https://arxiv.org/html/2609.16382#A4), and thus the kernel remains row\-stochastic\. However, some detail is lost\. Attention to untracked types is answered by one shared vector rather than by a value per type, so a model with a lower tracked share has more of its heads’ behavior compressed into that single average\.

## Appendix FMean\-Field Predictions Across Corpora

As described in the body, mean field analysis concerns the average behavior of a model over a given corpus\. The results in the body of the paper use OpenWebText as the reference corpus\. Here we describe how mean\-field prediction of centroids varies across four different corpora\. We demonstrate that, across the corpora we study, differences between predictions are very small, and that centroid predictions generalize out of distribution \(i\.e\., cross\-corpora\)\.

### F\.1Corpora Studied

#### Models\.

We evaluate corpus differences using the same models as studied in the paper body: GPT\-2 small and Gemma\-2 2B\. Statistics are re\-extracted from scratch for each model, and each model’s shared vocabulary is re\-derived as*its*1000 most frequent OpenWebText types\.

We choose corpora to reflect a diversity of ordinary English prose:

OpenWebText \(OWT\)\([Gokaslan et al\., 2019](https://arxiv.org/html/2609.16382#bib.bib26)\)\.The reference corpus, used in the paper body\. This corpus consists of web pages linked from Reddit, collected to reproduce GPT\-2’s training data\([Radford et al\., 2019](https://arxiv.org/html/2609.16382#bib.bib58)\)\.

FineWeb\([Penedo et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib54)\)\.This is a recent, heavily filtered crawl of the general web\. We use the 10\-billion\-token sample\. Like OWT, this corpus is general web text, but collected separately with different filtering\.

Wikipedia \(WikiText\-103\)\([Merity et al\., 2017](https://arxiv.org/html/2609.16382#bib.bib41)\)\.This consists of ‘featured’ and ‘good’ Wikipedia articles, in raw form\. This corpus is stylistically different from web text in that it is more formal and more encyclopedic in subject matter\.

Pre\-1919 books \(PG\-19\)\([Rae et al\., 2020](https://arxiv.org/html/2609.16382#bib.bib59)\)\.This set consists entirely of books published before 1919 from Project Gutenberg\. As such, it has a literary style, uses older vocabulary and constructions, and has no web artifacts\.

All corpora are prepared identically: text is tokenized with the model’s tokenizer and packed into 512\-token windows \(windows may span document boundaries\), and statistics are accumulated over 2 million tokens per \(corpus, half\) sample using TransformerLens\([Nanda and Bloom, 2022](https://arxiv.org/html/2609.16382#bib.bib44)\)\. Throughout, the vocabulary is fixed to the 1000 most frequent types of the OWT reference sample, so that all results are over the same set of types\.

### F\.2Mean Field Predictions Are Stable Across Corpora and Generalize Out of Distribution

For each corpusXXwe estimate the mean\-field kernel and roll out predicted centroids onXX’s statistics half\. For each corpusYYwe measure true centroids onYY’s evaluation half\. For each pair\(X,Y\)\(X,Y\)we initialize the rollout fromXX’s layer\-0 centroids, roll it out withXX’s kernel through the model’s weights, and score the predicted centroids at every layer againstYY’s true centroids\. With four corpora this gives a4×44\\times 4matrix of runs, which we ran in full for both models\.

Figure[16](https://arxiv.org/html/2609.16382#A6.F16)shows centroid prediction accuracy using the same open\-loop prediction as Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)\. We summarize those figures in Figure[15](https://arxiv.org/html/2609.16382#A6.F15), where the diagonal cells \(X=YX=Y\) show the average cosine similarity of mean\-field prediction of centroids for that corpus \(average over curve values in Figure[16](https://arxiv.org/html/2609.16382#A6.F16)\)\. The off\-diagonal cells assess generalization: whether statistics measured on one kind of text predict behavior on another\.

![Refer to caption](https://arxiv.org/html/2609.16382v1/corpusdep_heatmaps.png)Figure 15:Average cosine similarities of centroid predictions\. Rows: corpus supplying the mean\-field statistics\. Columns: corpus supplying the evaluation centroids\. Cell values are the cross\-layer mean centered cosine; diagonal \(matched\) cells in bold\.Table[10](https://arxiv.org/html/2609.16382#A6.T10)shows bootstrapped 95% confidence intervals for Figure[15](https://arxiv.org/html/2609.16382#A6.F15), and additionally the context\-free baselines\. The context\-free baseline runs each type through the model alone, as the two\-token input\[BOS,t\]\[\\textrm\{BOS\},t\], and scores the resulting per\-layer residual against the true centroids exactly as a prediction is scored\. Using no corpus statistics beyond the layer\-0 initialization, it is the floor that any corpus\-derived kernel must clear\. The context\-free forward does not depend on a corpus, so a single set of residuals serves every column and only the evaluation centroids change\.

Table 10:Values from Figure[15](https://arxiv.org/html/2609.16382#A6.F15)showing confidence intervals as well as context\-free baselineWe make three observations\.

1. 1\.Mean\-field centroid prediction replicates well on each corpus\. Diagonal cells are all at or near the model’s published OpenWebText level, including for pre\-1919 books\.Thus the conclusions of Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)are not specific to the OWT corpus used\.
2. 2\.OOD generalization is strong\(off\-diagonal cells in Figure[15](https://arxiv.org/html/2609.16382#A6.F15)\)\. Training the mean\-field on one corpus and predicting centroid evolution on a different corpus yields accuracy generally close to the same\-corpus evaluation\. The worst mismatched cell clears the context\-free baseline in every column and in both models\. Statistics from century\-old books still carry usable information about how the model treats modern web text\.
3. 3\.OOD generalization degrades according to expected dissimilarity of corpora, with generalization between OWT and FineWeb best, and pre\-1919 novels least\. All within\-column differences are statistically distinguishable

Figure 16:Per\-layer rollout quality by fractional depth, one column per model\. Top row: replication \(statistics and evaluation on the same corpus\), one curve per corpus\. Bottom row: generalization \(statistics from each corpus, evaluated against OpenWebText centroids\)\.

## Appendix GFull Mean Field Results

The four metrics compare, at each layerℓ\\ell, the mean\-field predictions𝐜^tℓ\\hat\{\\mathbf\{c\}\}^\{\\,\\ell\}\_\{t\}of Algorithm[1](https://arxiv.org/html/2609.16382#algorithm1)with the true centroids𝐜tℓ\\mathbf\{c\}^\{\\,\\ell\}\_\{t\}measured on the held\-out corpus half, over the evaluation set𝒯\\mathcal\{T\}of the10001000most frequent types \(experimental settings in §[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)\)\. Because the representation cloud is strongly anisotropic — it sits far from the origin relative to its own size \(Appendix[H](https://arxiv.org/html/2609.16382#A8)\) — every metric is computed after mean\-centering each cloud by its own mean: writing𝐜¯=1\|𝒯\|​∑t∈𝒯𝐜tℓ\\bar\{\\mathbf\{c\}\}=\\tfrac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\}\\mathbf\{c\}^\{\\,\\ell\}\_\{t\}and𝐜^¯\\bar\{\\hat\{\\mathbf\{c\}\}\}analogously for the predictions, we score the centered quantities𝐜~t=𝐜tℓ−𝐜¯\\tilde\{\\mathbf\{c\}\}\_\{t\}=\\mathbf\{c\}^\{\\,\\ell\}\_\{t\}\-\\bar\{\\mathbf\{c\}\}and𝐜^~t=𝐜^tℓ−𝐜^¯\\tilde\{\\hat\{\\mathbf\{c\}\}\}\_\{t\}=\\hat\{\\mathbf\{c\}\}^\{\\,\\ell\}\_\{t\}\-\\bar\{\\hat\{\\mathbf\{c\}\}\}\. Without centering, the shared offset would dominate every similarity\. The four metrics are then:

- •*Centered cosine similarity*— the mean over types of the per\-type cosine,1\|𝒯\|​∑tcos⁡\(𝐜^~t,𝐜~t\)\\tfrac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{t\}\\cos\\\!\\big\(\\tilde\{\\hat\{\\mathbf\{c\}\}\}\_\{t\},\\ \\tilde\{\\mathbf\{c\}\}\_\{t\}\\big\): per\-type directional agreement, aggregated by a simple mean\.
- •*Representation similarity analysis \(RSA\)*— we compute the pairwise cosine similarities among the centered true centroids\{𝐜~t\}\\\{\\tilde\{\\mathbf\{c\}\}\_\{t\}\\\}, the same among the centered predictions\{𝐜^~t\}\\\{\\tilde\{\\hat\{\\mathbf\{c\}\}\}\_\{t\}\\\}, and report the Pearson correlation between the two lists of off\-diagonal entries\. RSA scores whether the predicted cloud has the right*shape*— which types lie near which — independently of any global rotation of the cloud\.
- •*Centered relative error*— the ratio of means∑t‖𝐜^~t−𝐜~t‖/∑t‖𝐜~t‖\\sum\_\{t\}\\big\\\|\\tilde\{\\hat\{\\mathbf\{c\}\}\}\_\{t\}\-\\tilde\{\\mathbf\{c\}\}\_\{t\}\\big\\\|\\,\\big/\\,\\sum\_\{t\}\\big\\\|\\tilde\{\\mathbf\{c\}\}\_\{t\}\\big\\\|\. A ratio of means rather than a mean of per\-type ratios, so types lying near the cloud center do not arbitrarily inflate the metric; this is the one metric jointly sensitive to direction and magnitude\.
- •*Centered norm ratio*—∑t‖𝐜^~t‖/∑t‖𝐜~t‖\\sum\_\{t\}\\big\\\|\\tilde\{\\hat\{\\mathbf\{c\}\}\}\_\{t\}\\big\\\|\\,\\big/\\,\\sum\_\{t\}\\big\\\|\\tilde\{\\mathbf\{c\}\}\_\{t\}\\big\\\|, the predicted cloud’s mean radius relative to the true cloud’s, isolating magnitude:11is ideal, values above11indicate overshoot\.

§[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)reports only the first of these four metrics\. Figure[3](https://arxiv.org/html/2609.16382#S4.F3)in §[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)shows centered cosine for the kernel\-source substitutions and Figure[4](https://arxiv.org/html/2609.16382#S4.F4)shows it for the attention\-recipe ablations; Figures[17](https://arxiv.org/html/2609.16382#A7.F17)and[18](https://arxiv.org/html/2609.16382#A7.F18)below repeat both comparisons on all four metrics, for Gemma\-2, GPT\-2 and Qwen\-3\-14B\.

Figure 17:Mean\-field predictions are accurate and superior to alternative kernels \(kernel\-source substitutions\)\. Left: Gemma\-2; middle: GPT\-2; right: Qwen\-3\-14B\. Rows \(top to bottom\): centered cosine similarity, representation similarity analysis \(RSA\), centered relative error, and centered norm ratio\. For Qwen theuniformandPavgsubstitutions diverge in magnitude; their relative\-error and norm\-ratio curves leave the plotted range\.Figure 18:Mean\-field predictions are more accurate than the attention\-recipe ablations\. Left: Gemma\-2, Right: GPT\-2\. Rows \(top to bottom\): centered cosine similarity, representation similarity analysis \(RSA\), centered relative error, and centered norm ratio\.
## Appendix HAnisotropy of Representations

As noted in Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2), in both models the cloud of type centroids sits far from the origin relative to its own size\. In particular, the all\-type centroid𝐜¯\\bar\{\\mathbf\{c\}\}grows in magnitude from3535to379379across GPT\-2’s layers and from2525to424424across Gemma\-2’s, while the mean Euclidean distance of a type centroid to𝐜¯\\bar\{\\mathbf\{c\}\}stays a fraction of that magnitude — around0\.8​‖𝐜¯‖0\.8\\,\\\|\\bar\{\\mathbf\{c\}\}\\\|in GPT\-2’s early and middle layers and falling to0\.3​‖𝐜¯‖0\.3\\,\\\|\\bar\{\\mathbf\{c\}\}\\\|at the last layer; in Gemma\-2, it starts above 1\.0 but settles around0\.60\.6–0\.7​‖𝐜¯‖0\.7\\,\\\|\\bar\{\\mathbf\{c\}\}\\\|at deeper layers\. Table[11](https://arxiv.org/html/2609.16382#A8.T11)reports, per layer for both models, the magnitude of the all\-type centroid𝐜¯=1N​∑t𝐜t\\bar\{\\mathbf\{c\}\}=\\tfrac\{1\}\{N\}\\sum\_\{t\}\\mathbf\{c\}\_\{t\}, the mean Euclidean distance‖𝐜t−𝐜¯‖\\\|\\mathbf\{c\}\_\{t\}\-\\bar\{\\mathbf\{c\}\}\\\|of a type centroid from the cloud center, and their ratiomean dist/‖𝐜¯‖\\text\{mean dist\}/\\\|\\bar\{\\mathbf\{c\}\}\\\|\. The table shows that representations occupy a compact region offset from the origin, the motivation for mean\-centering before computing cosine similarities\.

Table 11:Anisotropy of centroid representations, per layer: the magnitude of the all\-type centroid‖𝐜¯‖\\\|\\bar\{\\mathbf\{c\}\}\\\|, the mean Euclidean distance of a type centroid from the cloud center, and their ratio\. Gemma\-2’s layers appear in two columns\.
## Appendix IImpact of Rogue Dimensions on Results

In every model, a handful of theDDdimensions of the representation𝐱i∈ℝD\\mathbf\{x\}\_\{i\}\\in\\mathbb\{R\}^\{D\}carry magnitudes far above the rest, and through the block’s normalization a small error in one perturbs every dimension on the next rollout step\. This appendix examines the effect of those “rogue” dimensions\. Setting them to their corpus value \(which we call ‘pinning’\) is a small, principled correction that modestly improves most models \(§[I\.1](https://arxiv.org/html/2609.16382#A9.SS1)\)\. Note, though, that the results in the body all use the unmodified \(un\-pinned\) representations, and the effects of the correction are examined here in the appendix only\. As an aside, we note that these rogue dimensions also seem to play a role in the instability results for Llama\-3\.2\-3B that we analyze in Appendix[J](https://arxiv.org/html/2609.16382#A10)\.

### I\.1Rogue\-Dimension Pinning: a Modest Correction

#### The phenomenon\.

Every transformer we examine carries a handful of dimensions one to two orders of magnitude above the median: referred to as “rogue dimensions” of GPT\-2 and BERT\([Timkey and van Schijndel, 2021](https://arxiv.org/html/2609.16382#bib.bib66);[Kovaleva et al\., 2021](https://arxiv.org/html/2609.16382#bib.bib37);[Puccetti et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib57)\), “outlier features” that emerge with scale\([Dettmers et al\., 2022](https://arxiv.org/html/2609.16382#bib.bib15)\), and “massive activations” documented across the LLaMA family and other modern models\([Sun et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib63)\)\. The same dimensions underlie the representation anisotropy of[Gao et al\. \(2019a\)](https://arxiv.org/html/2609.16382#bib.bib21)and[Mu et al\. \(2018\)](https://arxiv.org/html/2609.16382#bib.bib43)\. Because each transformed block normalizes its input by a scale computed over allDDdimensions \(root\-mean\-square for the RMSNorm models, standard deviation for GPT\-2’s LayerNorm\), a rogue dimension dominates that scale, so a small relative error in its predicted value perturbs every other dimension on the following step\. Pinning these dimensions to their corpus value removes that coupling\.

#### Why removing them is justified\.

These dimensions are approximately*context\-invariant*, taking nearly the same value whatever the input token, and act as an implicit bias or attention\-sink mechanism rather than as carriers of token values\([Sun et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib63)\)\. These dimensions have a direct link to attention sinks \(the tendency of heads to deposit excess probability mass on a few uninformative positions\)\([Xiao et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib72);[Gu et al\., 2025](https://arxiv.org/html/2609.16382#bib.bib28)\)\.[Bondarenko et al\. \(2023\)](https://arxiv.org/html/2609.16382#bib.bib8)show that this “do\-nothing” routing of attention is what produces the outlier activations\. Holding these dimensions at their corpus value during the rollout thus mainly removes a nuisance term, rather than a signal the mean field is asked to reproduce\.

#### Identifying the set\.

We pin a dimension if at some layer its per\-dimension RMS over the token centroids exceeds15×15\\timesthat layer’s median, a threshold set by the wide magnitude gap separating these dimensions from the bulk; it selects at or below1%1\\%ofDDin each model \(Table[12](https://arxiv.org/html/2609.16382#A9.T12)\)\. The specific indices are in the code release\.

Table 12:Number of dimensions pinned during the rollout by model, and against model widthDD\.
#### Effect of pinning\.

Pinning is a refinement that produces a modest improvement in accuracy of rollout prediction\. Pinning leaves the teacher\-forced prediction and the early rollout layers unchanged, and it improves the final\-layer rollout for the models whose rollout error runs through these dimensions: for GPT\-2 the centered cosine rises→0\.840\.76\\\!\\to\\\!0\.84\(relative error→0\.570\.73\\\!\\to\\\!0\.57\), for Gemma\-2→0\.820\.70\\\!\\to\\\!0\.82\(→0\.560\.83\\\!\\to\\\!0\.56\), and for Qwen\-3\-14B→0\.820\.72\\\!\\to\\\!0\.82\. For the two Llama models it is neutral to slightly negative \(Llama\-3\.1\-8B→0\.710\.72\\\!\\to\\\!0\.71, Llama\-3\.2\-3B→0\.580\.60\\\!\\to\\\!0\.58\), because their rollout error is not dominated by these dimensions, but by the mechanism in Appendix[J](https://arxiv.org/html/2609.16382#A10)\.

### I\.2Rogue Dimensions and the Qwen\-3\-14B Ablations

Qwen\-3\-14B is the one model where pinning changes the*interpretation*of a result rather than just a quantitative improvement\. The mean\-field rollout of Qwen\-3\-14B is barely affected by pinning\. However, three substitute kernels whose rollouts drifted far off course \(Pavg,cooc, androwshuffle\) as shown in Figure[17](https://arxiv.org/html/2609.16382#A7.F17)\(right\-hand column\), are dramatically improved \(and the remaining four improve marginally\) \(Table[13](https://arxiv.org/html/2609.16382#A9.T13)\)\. The rogue dimensions are a conditional amplifier: with the mean\-field kernel the predicted norm never leaves the stable regime, whereas some substitute kernels drive the trajectory off course enough for the mispredicted rogue values, amplified through the shared RMSNorm scale to produce a discrete blow\-up\. In those cases, pinning removes the blowup\. Rogue dimensions therefore do not penalize every kernel uniformly; they engage only once a prediction has already drifted, which is why the mean\-field kernel escapes their effect\.

Table 13:Qwen\-3\-14B rollout under each kernel, unpinned versus pinned: cross\-layer mean centered cosine \(standard error over the 40 layers\) and mean centered relative error\.The sensitivity of Qwen\-3\-14B to rogue dimensions is almost entirely a property of a single layer\. Applying a substitute kernel to a chosen subset of layers and the mean\-field kernel everywhere else localizes the damage \(Table[14](https://arxiv.org/html/2609.16382#A9.T14)\): substituting at layer 0 alone incurs0\.2790\.279of the0\.3270\.327all\-layers total \(an85%85\\%share\), while substituting at all thirty\-nine*other*layers costs only0\.0650\.065\(a20%20\\%share\)\. The two shares sum to105%105\\%, so the decomposition is nearly clean, and layer 0 carries more than four times the damage of the other thirty\-nine combined\. The same measurement on GPT\-2 spreads the damage across depth instead, as one expects of a rollout whose error accumulates layer by layer\. Qwen’s apparent hypersensitivity to kernel substitution is therefore an early\-layer property rather than depth compounding, in the same family as the layer\-1 amplification that destabilizes Llama\-3\.2\-3B’s rollout \(Appendix[J](https://arxiv.org/html/2609.16382#A10)\), though by a different mechanism\.

Table 14:Localizing the damage from a kernel substitution in Qwen\-3\-14B, pinned\. The substitute \(cooc\) is applied only to the listed layers, with the measured kernel used everywhere else; entries are cross\-layer mean centered cosine over the 40 layers and the shortfall against the measured kernel\.Pinning is not uniformly beneficial: the mean field’s own relative error*worsens*slightly under it \(0\.640→0\.7210\.640\\to 0\.721\) even as its cosine improves, anduniformstays broken when pinned, so the amplifier is not the only thing defeating that kernel\.

## Appendix JThe Llama Models: Accurate Per\-Layer Prediction, Failed Rollout

We extended the mean\-field validation of Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)to Llama\-3\.2\-3B and Llama\-3\.1\-8B \(both RMSNorm, RoPE, grouped\-query attention; statistics from a 2M\-token OpenWebText pass over the top\-1000 Llama token types\)\. We find that per\-layer accuracy matches the other models \(Figure[5](https://arxiv.org/html/2609.16382#S4.F5)\)\. However, the open\-loop rollout does not, and exposes a failure mode specific to Llama\. We work out this dependency here, through a sequence of targeted experiments\. While our experiments serve to*eliminate*a number of natural hypotheses, we ultimately lack a crisp explanation\. We report the eliminations along with the final characterization, since several of the eliminated explanations could have been reasonable hypotheses*a priori*\.

#### Summary of results\.

First, we affirm that the mean\-field yields reasonably accurate predictions layer\-by\-layer; it is only when errors accumulate through the open\-loop rollout that predictions diverge\. We show using teacher\-forced predictions that the mean field predicts each layer’s centroid update for both Llamas about as well as for the other models: cross\-layer mean centered cosine0\.8580\.858\(±0\.017\\pm 0\.017\) for Llama\-3\.2\-3B and0\.8370\.837\(±0\.019\\pm 0\.019\) for Llama\-3\.1\-8B, with the familiar mid\-stack dip \(minima0\.690\.69and0\.620\.62\) and recovery\.

Figures[19](https://arxiv.org/html/2609.16382#A10.F19)–[20](https://arxiv.org/html/2609.16382#A10.F20)give the Llama counterparts of Figures[3](https://arxiv.org/html/2609.16382#S4.F3)and[4](https://arxiv.org/html/2609.16382#S4.F4), and Figure[21](https://arxiv.org/html/2609.16382#A10.F21)the counterpart of Figure[6](https://arxiv.org/html/2609.16382#S4.F6)\. Iterated open\-loop, Llama\-3\.1\-8B behaves approximately like GPT\-2, Gemma\-2, and Qwen\-3: cross\-layer mean rollout cosine0\.7190\.719\(±0\.019\\pm 0\.019\) with cosine similarity at the final\-layer of0\.720\.72\. Notably, Llama\-3\.2\-3B instead*diverges*: the predicted centered magnitude inflates to4\.9×4\.9\\timesthe true value, the rollout cosine collapses to0\.300\.30by layer 2 and averages0\.4990\.499\(±0\.028\\pm 0\.028\), and the centered relative error exceeds11from mid\-stack on\.

The teacher\-forced prediction is not severely affected \(Figure[21](https://arxiv.org/html/2609.16382#A10.F21)\)\. These per\-layer measurements remain the important mean\-field fidelity measure for this model\. The rest of this section concerns why the*iteration*fails\.

Figure 19:Kernel\-source substitutions \(as Figure[3](https://arxiv.org/html/2609.16382#S4.F3)\) for the Llama models\. Left: Llama\-3\.2\-3B; right: Llama\-3\.1\-8B\. Centered per\-token cosine of the open\-loop rollout by fractional depth\.Figure 20:Attention\-recipe ablations \(as Figure[4](https://arxiv.org/html/2609.16382#S4.F4)\) for the Llama models\. Left: Llama\-3\.2\-3B; right: Llama\-3\.1\-8B\.Figure 21:Per\-layer \(teacher\-forced\) error decomposition \(as Figure[6](https://arxiv.org/html/2609.16382#S4.F6)\) for the Llama models: \(a\) exact attention with the MLP at the centroid; \(b\) additionally the value map at the centroid; \(c\) the full mean field\. Left: Llama\-3\.2\-3B; right: Llama\-3\.1\-8B\. Note the lower vertical scale than Figure[6](https://arxiv.org/html/2609.16382#S4.F6)\.
#### The divergence is a single\-step amplification in the layer\-1 MLP\.

Tracking the rolled\-out and true trajectories together, the error entering the layer\-1 step is amplified17×17\\timesin that one step and thereafter merely persists \(∼1\.05×\{\\sim\}1\.05\\timesper later step\)\. We note three properties of the error\. \(i\) It is a*curvature*effect and not a large derivative\. Rescaling the incoming error toα​δ\\alpha\\deltaand re\-running the block, the amplification grows nonlinearly with the amount that we scale the error\. So the block’s derivative at the true centroid is benign in the direction the error actually points — at small amplitude the real error and a random one are amplified almost equally \(1\.6×1\.6\\timesversus1\.2×1\.2\\times\)\. What matters is that when the error is large enough,36%36\\%of the centroid norm, it leaves the region over which the MLP is affine\. \(ii\) It is*in the MLP*: decomposing the step, the attention update expands the error by1\.06×1\.06\\times; the subsequent MLP takes it to17×17\\times\. \(iii\) It is*model\-specific*: the same measurement gives∼1×\{\\sim\}1\\timesat every layer for GPT\-2, Gemma\-2, Qwen\-3\-14B, and Llama\-3\.1\-8B\. Further, the rogue\-dimension pinning of Appendix[I](https://arxiv.org/html/2609.16382#A9)does not affect the error \(final rollout cosine0\.602→0\.5840\.602\\to 0\.584for the 3B,0\.720→0\.7080\.720\\to 0\.708for the 8B\): the amplified error is spread broadly across coordinates, with the largest massive\-activation channel contributing only1\.2%1\.2\\%of the error\.

Further investigation identifies that only a few gates \(approximately 25\) in the first MLP are responsible for the great majority of the added error\. Those few gates account for the amplification outright\. Because the MLP output is exactly linear in the post\-activation, the response decomposes exactly over the MLP’s81928192gates; restricting it to the top 25 recovers94\.7%94\.7\\%of the*nonlinear excess*, while an equal number of randomly chosen gates recovers0\.1%0\.1\\%\.

#### The amplified error concentrates on closed\-class tokens\.

The per\-token amplification is heavy\-tailed: the median token’s error grows12×12\\times, the9999th percentile143×143\\times, the maximum323×323\\times, and the ten most\-amplified tokens carry53%53\\%of the total output\-error energy\. Of the fifty most\-amplified, forty\-three are newlines, punctuation, or clause\-boundary tokens such as parentheses or “The\.”\. These are all ‘closed\-class’ tokens: high\-frequency function words and structural markers rather than content words\. Many start at early layers with tiny errors; the extreme case is token‘\)\.\\n\\n’whose centroid error norm is0\.490\.49going into transformer block 1 and error159159coming out of that block\.

#### Exclusion experiment: structural tokens are not the cause\.

The concentration on boundary tokens suggests an obvious hypothesis: the structural tokens themselves drive the failure\. We tested it by rerunning the entire pipeline — token selection, corpus statistics, and rollout, on a fresh 1M\-token extraction — with every newline, whitespace, and pure\-punctuation token excluded from the tracked vocabulary \(6767of the original top\-1000; capitalized words that initialized a clause were retained deliberately, as a probe\)\. The divergence is unchanged: peak magnitude ratio5\.25\.2versus4\.94\.9, mean rollout cosine0\.4260\.426versus0\.4990\.499, teacher\-forced accuracy unchanged \(0\.8520\.852\)\. Token amplification simply*migrates*to remaining closed\-class tokens: conjunctions \(but,and,or\), auxiliaries \(is,are,has,was\), negation stems \(wasn,didn,doesn\), determiners \(the,a\), and sub\-word prefixes \(p,de\) — only one of the top twenty is a capitalized clause\-initial word — with the same concentration \(top ten=54%=54\\%of error energy\)\. The same rerun for Llama\-3\.1\-8B likewise changes nothing material \(mean rollout cosine0\.6150\.615versus0\.7190\.719, peak magnitude ratio1\.451\.45versus1\.701\.70, teacher\-forced unchanged at0\.8270\.827\)\.

#### Attention\-sink accounts, tested\.

The coincidence of boundary tokens, massive\-activation channels, and layer 1 \(where sink structure first forms\) suggests the attention sink\([Xiao et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib72);[Sun et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib63)\)as a common cause\. However, three versions of this idea all give insufficient explanations\.

The first version is that boundary tokens develop large errors because they are local attention sinks\. In fact, boundary tokens genuinely are*local*sink targets — they receive1\.41\.4–1\.7×1\.7\\timesmore attention per occurrence than content tokens\. However, the queries that feed them are not the queries that feed BOS, so boundary tokens are not standing in for the sink\. Further, across tokens, how strong a “magnet” a token is does not predict its error amplification\.

The second version is that boundary tokens develop large errors because they carry the massive activations\. However, we find that boundary tokens’ median magnitude in the massive\-activation dimensions is 0\.71, vs 0\.73 for content tokens\. So they are not unusual in their massive activation dimensions\.

The third version is that boundary tokens are a mixture of tokens with high values of the massive activation dimension and lower values of the massive activation dimension\. This would cause the MLP at the centroid to make strongly incorrect predictions\. However, we find that the massive\-activation dimension values are unimodel across boundary tokens\. What survives is only a graded boundary signal in the massive activation dimension \(1\.21\.2–1\.8×1\.8\\timeshigher for phrase\- and sentence\-initial or \-final occurrences\), too weak and too continuous to account for the amplification\.

#### Capitalization pairs: position is not the driver\.

As a final positional test, we compared mean\-field deviations across all7777capitalized/lowercase pairs in the vocabulary \(The/the,In/in, …\), capitalization being a near\-perfect marker of sentence\-initial position\. There is no systematic effect: the paired median detonation ratio is1\.001\.00\(Wilcoxonp=0\.37p=0\.37\), and teacher\-forced error is actually slightly*lower*for the capitalized forms \(0\.81×0\.81\\times,p<10−4p<10^\{\-4\}\)\. The tails are large in both directions —Thedetonates3\.6×3\.6\\timesits lowercase twin and paragraph\-initialThe7\.1×7\.1\\times, butanddetonates50×50\\timesmore thanAnd— so the well\-knownThecase is atypical rather than representative\.

#### What remains\.

The Llama\-3\.2\-3B divergence is fully characterized: it is confined to a single step in the layer\-1 MLP, specific to this model, and heavy\-tailed over closed\-class tokens, with its output concentrated in the massive\-activation dimensions\. The natural explanations are eliminated: it is not the structural tokens \(exclusion experiment\), not rogue\-dimension coupling \(pinning inert\), not sink\-magnet status, not a sink\-writing gate, and not syntactic position \(capitalization pairs\)\. What is left is a curvature effect in this one model’s layer\-1 MLP, and it is now accounted for: the layer\-0 mean\-field error of high\-frequency, closed\-class tokens is concentrated enough to drive a handful of SwiGLU gates across their nonlinearity, and those gates carry essentially all of the resulting amplification\. This is the MLP\-at\-centroid \(Jensen\) approximation, identified in Section[4\.2](https://arxiv.org/html/2609.16382#S4.SS2)as the leading per\-step error source in every model, meeting an error large enough to leave the affine region\. Why this model, and this layer, place their gates so close to the knee where the others do not, we leave open\.

## Appendix KImpact of the MLP on Mean\-Field Approximation

The last step of Algorithm[1](https://arxiv.org/html/2609.16382#algorithm1)applies the layer’s MLP once, to the mean intermediate representation𝐦s\\mathbf\{m\}\_\{s\}\(MLP⁡\(LN⁡\(𝐦s\)\)\\mathrm\{MLP\}\(\\mathrm\{LN\}\(\\mathbf\{m\}\_\{s\}\)\)\), in place of averaging the MLP over the individual intermediate tokens of typess\(𝔼occ​\[MLP​\(LN​\(𝐦\)\)\]\\mathbb\{E\}\_\{\\mathrm\{occ\}\}\[\\mathrm\{MLP\}\(\\mathrm\{LN\}\(\\mathbf\{m\}\)\)\]\)\. To quantify the resulting error we accumulate both quantities per type and layer over a corpus and compare them\. Table[15](https://arxiv.org/html/2609.16382#A11.T15)reports, per layer, the mean\-centered cosine and relative error \(ratio of means\) between the two quantities, averaged over types\. For both GPT\-2 and Gemma\-2, the behavior follows a similar pattern\. The approximation is near\-exact at the first layers \(cosine≈0\.99\\approx 0\.99for both models\), weakest in the middle of the network \(cosine≈0\.65\\approx 0\.65for GPT\-2 at layers 4–5,≈0\.73\\approx 0\.73for Gemma\-2 at layer 11\), and improved somewhat at later layers\. Overall, the centered cosine averaged over layers is0\.770\.77\(GPT\-2\) and0\.860\.86\(Gemma\-2\)\.

These numbers are what make the MLP step the dominant term in the error decomposition of §[4\.3](https://arxiv.org/html/2609.16382#S4.SS3), yet a cosine of0\.650\.65\(the bark\) sounds far worse than the per\-layer shortfall of0\.020\.02–0\.030\.03\(the bite\) reported there\. Two things account for the difference\. The cosines above measure the MLP contribution against itself, whereas the decomposition measures the predicted centroid, which carries the attention output alongside the MLP contribution, so a given disagreement in the MLP term costs less in the centroid than it does on its own\. And most of the disagreement is a shared offset in*magnitude*rather than an error in*direction*\.

Table 15:MLP mean\-field approximation, per layer: mean\-centered cosine and relative error \(ratio of means\) between the occurrence\-mean MLP contribution for typessand the MLP applied to the mean intermediate token𝐦s\\mathbf\{m\}\_\{s\}, averaged over types\. Gemma\-2’s layers continue in the right\-hand block\.
## Appendix LReplication on OLMo\-2, and Evidence of a Shared Mean\-Field Ceiling

Our waypoint results in §[5](https://arxiv.org/html/2609.16382#S5)were measured on Pythia\. To ask whether our findings were a property of that suite or of language\-model training generally, we repeated the §[5](https://arxiv.org/html/2609.16382#S5)substitution on OLMo\-2\-7B\([OLMo Team, 2024](https://arxiv.org/html/2609.16382#bib.bib48)\)across its 44 stage\-1 checkpoints, and plotted all three models on a common axis of training*tokens*in Figure[22](https://arxiv.org/html/2609.16382#A12.F22)\. A common token axis is necessary because one step is not the same amount of training in the two suites: a Pythia step is2\.0972\.097M tokens and an OLMo step is4\.24\.2M\. OLMo is an independent architecture, applying RMSNorm to the attention*output*rather than its input, so that the residual entering attention is unnormalized, and it is trained on a different corpus\. We measured its statistics exactly as Pythia’s \(a 2M\-token OpenWebText pass over the top10001000types\), giving substitution coverage0\.650\.65against Pythia’s0\.640\.64\.

At the left of the figure all three models fail to improve over their respective mean fields:Δ​L=−0\.001\\Delta L=\-0\.001for Pythia\-1\.4B and−0\.006\-0\.006for Pythia\-12B at0\.130\.13B tokens, and\+0\.003\+0\.003for OLMo at0\.630\.63B\. A model reducing to its mean field early in training is therefore not unique to Pythia; a second architecture trained on different data exhibits the same behavior, and to the same degree: OLMo’s deviation there isDMF=0\.0713D\_\{\\mathrm\{MF\}\}=0\.0713, against0\.07200\.0720for Pythia\-160m at initialization\.

The rest of the figure shows that as training proceeds, the threeLmodelL\_\{\\mathrm\{model\}\}curves separate and keep falling, reaching2\.852\.85,2\.652\.65and2\.412\.41nats by the right\-hand edge\. The threeLMFL\_\{\\mathrm\{MF\}\}curves do not: they flatten into a narrow band and stay there, spanning only4\.934\.93–5\.455\.45nats over the whole decade from1010B to100100B tokens\. The three models differ by a factor of eight in parameters, and OLMo in training data as well, yet the gapΔ​L\\Delta Lat the right\-hand edge is nearly the same for all three:2\.492\.49,2\.812\.81and2\.662\.66nats\. A plausible reading is that the loss a mean field can reach is set by the corpus statistics it is built from rather than by the model whose attention it replaces: the ceiling is fixed, and the widening gap is the model moving away from it\.

The comparison has limits\. OLMo publishes no checkpoint before0\.630\.63B tokens and none at initialization, so its curve begins where Pythia’s has already begun to depart from its mean field\. Thus, the left\-hand decade is Pythia\-only\. In this window there are ten Pythia points against twenty\-nine OLMo points, so the Pythia curves are the coarser of the two\. All three rest on 2M\-token statistics, whose sensitivity we examine next\.

Figure 22:Pythia\-1\.4B, Pythia\-12B and OLMo\-2\-7B on a common axis of training tokens in billions\. Solid lines with filled markers areLmodelL\_\{\\mathrm\{model\}\}; dashed lines with hollow markers areLMFL\_\{\\mathrm\{MF\}\}, the loss when every head’s read is replaced by its per\-context mean\-field read\. Color and marker shape both identify the model\.
## Appendix MSensitivity to the Top\-NNTruncation

To assess the sensitivity to truncation to the top\-NNtypes, we re\-ran the substitution of §[5](https://arxiv.org/html/2609.16382#S5)for Pythia\-160M atN=2000N=2000andN=4000N=4000types, which raised position coverage, the quantity of Appendix[E](https://arxiv.org/html/2609.16382#A5), from65%65\\%to72%72\\%and80%80\\%, respectively\. That substitution resolves the tracked types explicitly and collapses everything else into the position\-00and “other” buckets of Appendix[D](https://arxiv.org/html/2609.16382#A4), each of which carries the measured attention mass but a single shared value vector; queries outside the tracked set keep the model’s unmodified computation\.

#### Controlling estimation quality\.

The kernel entryWs​t=Ps​t/n¯s​tW\_\{st\}=P\_\{st\}/\\bar\{n\}\_\{st\}is estimated from a corpus pass, and its support is the co\-occurrence countCs​tC\_\{st\}\. EnlargingNNenlarges the pair set quadratically while leaving the corpus fixed, so support per pair falls sharply\. MatchingN=1000N=1000’s support at largerNNrequires roughly4×4\\timesthe corpus atN=2000N=2000and1515–20×20\\timesatN=4000N=4000, by three separate criteria \(pair coverage, mass coverage, and the worst\-supported corner\)\. We therefore repeated every extraction at40M tokens, twenty times the original pass and enough to cover the worst case, using the same corpus for all three arms so that corpus size is not itself a variable\.

#### At matched estimation quality, the truncation has little effect\.

Over steps00–6464the three vocabularies giveΔ​L\\Delta Lwithin0\.0010\.001nats of one another \(Figure[23](https://arxiv.org/html/2609.16382#A13.F23)\)\. When each arm’sΔ​L\\Delta Lcurve is normalized to its own final value, the three agree in shape to within0\.010\.01across a fourfold change inNN\. The phase structure detailed in §[5](https://arxiv.org/html/2609.16382#S5)is unchanged\.

Figure 23:The waypoint experiment atN=1000N=1000,20002000and40004000keys, all with 40M\-token statistics\.Left:Δ​L\\Delta L; the arms separate only late in training, by an amount tracking substitution coverage\.Centre:Δ​L\\Delta Lnormalized to each arm’s own final value, whose shape agrees to within0\.010\.01\.Right:DMFD\_\{\\mathrm\{MF\}\}, including the early minimum, which is within0\.0030\.003across the three\. The band marks the induction window as in Figure[7](https://arxiv.org/html/2609.16382#S5.F7)\.
#### The full trajectories\.

Figure[24](https://arxiv.org/html/2609.16382#A13.F24)redraws the three arms in the layout of Figure[7](https://arxiv.org/html/2609.16382#S5.F7), withLmodelL\_\{\\mathrm\{model\}\}andLMFL\_\{\\mathrm\{MF\}\}above,Δ​L\\Delta LandDMFD\_\{\\mathrm\{MF\}\}below, against training step with the induction window shaded, so each column can be read directly against the per\-model panels there\. We note, first, thatLmodelL\_\{\\mathrm\{model\}\}is the same curve in all three columns, agreeing to6×10−76\\times 10^\{\-7\}nats at every checkpoint: the substitution vocabulary is a property of the null model, not of the network, so the model’s own loss is a fixed reference and onlyLMFL\_\{\\mathrm\{MF\}\}moves\. This is a check that the three arms differ in nothing but the intended variable\. Second, every landmark of the trajectory falls at the same checkpoint irrespective ofNN\.LMFL\_\{\\mathrm\{MF\}\}stays within0\.010\.01nats ofLmodelL\_\{\\mathrm\{model\}\}through step6464and detaches between steps128128and256256; it reaches its minimum at step20002000, rises to a late maximum at step3200032000, and falls back by step143000143000\.

Figure 24:The waypoint experiment atN=1000N=1000,20002000and40004000keys \(columns\), 40M\-token statistics throughout, drawn in the layout of Figure[7](https://arxiv.org/html/2609.16382#S5.F7)so the two may be compared panel for panel\.Top:next\-token loss of the model \(LmodelL\_\{\\mathrm\{model\}\}, solid\) and of the context\-conditional mean field \(LMFL\_\{\\mathrm\{MF\}\}, dashed\); the dotted line marks the model’s final loss, and each column is titled with its substitution coverage\.Bottom:Δ​L=LMF−Lmodel\\Delta L=L\_\{\\mathrm\{MF\}\}\-L\_\{\\mathrm\{model\}\}\(left axis\) with the mean per\-head deviationDMFD\_\{\\mathrm\{MF\}\}\(right axis\)\. The band marks the induction window, as in Figure[7](https://arxiv.org/html/2609.16382#S5.F7)\.LmodelL\_\{\\mathrm\{model\}\}is common to the three columns by construction; the shape of the remaining curves is likewise unchanged, and only the magnitude ofΔ​L\\Delta Lgrows with coverage\.

## Appendix NKernel Comparison: Estimation Details

The kernel dynamics of Section[5\.3](https://arxiv.org/html/2609.16382#S5.SS3)compare the per\-key kernelW\(τ\)=P\(τ\)/n¯W^\{\(\\tau\)\}=P^\{\(\\tau\)\}/\\overline\{n\}\(Appendix[A](https://arxiv.org/html/2609.16382#A1)\) at checkpointτ\\taubetween every pair of checkpoints\. The divisorn¯\\overline\{n\}is a pure token count and so is model\-free and identical at every checkpoint; dividing by it removes the frequency weighting carried byPP’s entries, leaving changes in the attention pattern itself\. Similarity is linear CKA, computed per head and averaged over heads, with each kernel column\-centered beforehand and replaced by a rank\-96 SVD approximant \(the CKA is computed exactly from the 96\-dimensional factors, and agrees with the exact value to within 0\.001\)\. All kernel\-level comparisons are restricted to type pairs with context count≥20\\geq 20; without this support mask a small number of near\-zero\-support pairs, whose ratios are unstable, dominate the quadratic CKA\. Every checkpoint’s statistics are extracted from the same corpus sample with the same chunking, so entries are directly comparable across checkpoints\.

#### Numerical values\.

Adjacent\-checkpoint similarity is 0\.99–1\.00 through the frozen stretch \(steps 0–32 at 1\.4B and 12B, 0–64 at 160m\)\. It then falls progressively across the reorganization: at 160m0\.99→0\.86→0\.76→0\.710\.99\\to 0\.86\\to 0\.76\\to 0\.71over steps 32–512, at 1\.4B0\.99→0\.97→0\.79→0\.670\.99\\to 0\.97\\to 0\.79\\to 0\.67, and most steeply at 12B0\.93→0\.65→0\.45→0\.400\.93\\to 0\.65\\to 0\.45\\to 0\.40, with the minimum near steps 256–2000 in every model and deepest at 12B \(0\.400\.40\)\. After the transition the two smaller models recover to∼0\.8\{\\sim\}0\.8by step 4000 and∼0\.9\{\\sim\}0\.9by step 16000; 12B recovers more slowly \(0\.660\.66then0\.750\.75\), consistent with its deeper reorganization\.

#### Row uniformity\.

![Refer to caption](https://arxiv.org/html/2609.16382v1/w_spread_compare.png)Figure 25:How each head divides its attention across token types, all three models, plotted as fold\-variation: a value of one is uniform attention across tokens\.Left:mean and interquartile band across heads\.Right three panels:The same, head by head \(160m, 1\.4B, 12B\)\. Head differentiation begins around step 128; a subset of near\-uniform heads persists throughout, while the mid\-training peak grows with scale \(mean fold\-variation≈4\{\\approx\}4at 12B\)\.The row\-spread measure ismeans⁡stdt⁡\(log⁡W⁡\[s,t\]\)\\operatorname\{mean\}\_\{s\}\\operatorname\{std\}\_\{t\}\(\\log W\[s,t\]\)over the masked key types, computed per head\. At initialization the mean spread is 0\.40 \(160m\) and 0\.38 \(1\.4B\), with all heads within∼15%\{\\sim\}15\\%of one another; it dips slightly at theDMFD\_\{\\mathrm\{MF\}\}\-minimum checkpoints before head differentiation begins\.

Figure[25](https://arxiv.org/html/2609.16382#A14.F25)plots this measure across training\. The left panel shows the mean spread with its interquartile band across heads; the right three panels show the individual spread of each head, grouped by layer, for the 160m, 1\.4B, and 12B models\. Early in training the division is close to even, so a token draws attention in proportion to how often it appears and not otherwise; heads begin to attend selectively when the kernel reorganizes, and the per\-head spread then generally increases through the remainder of training\. The heads also differ sharply from one another: some develop large spread while others barely change from their near\-uniform initialization\.

## Appendix OHow Reliably Each Deviation Metric Separates the Prompt Classes

Table 16:Separation of each prompt class from the random control, as a standardized mean difference \(Hedges’gg\)\. Positive means the class exceeds the control; negative means it runs the other way\. All entries are significant atp<0\.05p<0\.05on a Welch test, most far below it, except the one marked†\\dagger\.Section[6\.1](https://arxiv.org/html/2609.16382#S6.SS1)claims that the mean\-field deviationDMFD\_\{\\mathrm\{MF\}\}, and its covariance and contextualization components, respond to in\-context learning\. We test this against four designed prompt classes, induction, few\-shot, natural text, and a random control, asking whether each class separates from the control\. Table[16](https://arxiv.org/html/2609.16382#A15.T16)reports the separation as Hedges’gg, the gap between a class’s mean and the control’s in units of their pooled within\-class spread; a largeggmeans the two are reliably apart, not that the deviation is large\. The deviationDMFD\_\{\\mathrm\{MF\}\}separates the planted classes from the control in both models \(induction and few\-shotggof\+3\.60,\+2\.06\+3\.60,\+2\.06for GPT\-2 and\+2\.93,\+2\.21\+2\.93,\+2\.21for Gemma\-2\)\. The two components carry the separation differently: in GPT\-2,McovM\_\{\\mathrm\{cov\}\}separates induction \(\+2\.98\+2\.98\) andMctxM\_\{\\mathrm\{ctx\}\}separates few\-shot \(\+6\.84\+6\.84\), while in Gemma\-2,MctxM\_\{\\mathrm\{ctx\}\}separates every class andMcovM\_\{\\mathrm\{cov\}\}is weak or negative\. This matches the model\-dependent split reported in §[6](https://arxiv.org/html/2609.16382#S6)\.

Two points qualify the table\. First, the separation is*cross\-class*:ggmeasures how well the deviation ranks these designed conditions, not individual prompts within a condition\. Within a class the metrics carry almost no information aboutΔ​Lperm\\Delta L\_\{\\mathrm\{perm\}\}\. Prompt class alone explains it atR2=0\.69R^\{2\}=0\.69for GPT\-2 and0\.580\.58for Gemma\-2, and adding any metric raisesR2R^\{2\}by at most0\.0040\.004\. A correlation pooled over prompts would look strong while measuring only the class separation, so we do not report one\.

Second, the separation is uneven across classes\. GPT\-2’sDMFD\_\{\\mathrm\{MF\}\}does not distinguish natural text from the control \(g=\+0\.04g=\+0\.04,p=0\.84p=0\.84\), even though permuting that text raises its loss \(g=\+1\.22g=\+1\.22\)\. In GPT\-2, then, the deviation registers planted ICL but not the background ICL of ordinary text\. Gemma\-2’sDMFD\_\{\\mathrm\{MF\}\}separates natural text as well \(\+1\.06\+1\.06\)\.

## Appendix PProvenance of the Heads’ Roles and Ranking

This appendix supplies the provenance of Table[4](https://arxiv.org/html/2609.16382#S6.T4): how each head’s documented role was established, and what the readings show at both ends of the ranking\.

### P\.1Lenses for Mean Field Analysis

The lenses below all read the per\-occurrence deviationDs​Xℓ​hD^\{\\ell h\}\_\{sX\}, whose type\-averaged form is the ranking scoreS¯ℓ​h\\bar\{S\}^\{\\ell h\}of \([17](https://arxiv.org/html/2609.16382#S6.E17)\); each notes where the per\-occurrence and ranking orderings disagree\. The first lens, the covariance/contextualization plane, is the one the body uses throughout \(§[6\.2](https://arxiv.org/html/2609.16382#S6.SS2)\- §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3)\); the other lenses are instruments for the per\-head readings collected here\.

#### Lens 1: Covariance/contextualization plane\.

Terms \(a\) and \(b\) of \([15](https://arxiv.org/html/2609.16382#S6.E15)\) separate the deviation by which factor varies: where a head attends, versus what values the tokens it attends to carry\. Averaging the per\-context shares of \([16](https://arxiv.org/html/2609.16382#S6.E16)\) over occurrences rather than over heads places every head on two axes,

Mcovℓ​h=𝔼contexts​X​\[σcov​Ds​Xℓ​h\],Mctxℓ​h=𝔼contexts​X​\[σctx​Ds​Xℓ​h\],M\_\{\\mathrm\{cov\}\}^\{\\ell h\}=\\mathbb\{E\}\_\{\\text\{contexts \}X\}\\\!\\big\[\\,\\sigma\_\{\\mathrm\{cov\}\}\\,D^\{\\ell h\}\_\{sX\}\\big\],\\qquad M\_\{\\mathrm\{ctx\}\}^\{\\ell h\}=\\mathbb\{E\}\_\{\\text\{contexts \}X\}\\\!\\big\[\\,\\sigma\_\{\\mathrm\{ctx\}\}\\,D^\{\\ell h\}\_\{sX\}\\big\],\(33\)withσcov\\sigma\_\{\\mathrm\{cov\}\}andσctx\\sigma\_\{\\mathrm\{ctx\}\}the two shares of the error defined at \([16](https://arxiv.org/html/2609.16382#S6.E16)\)\. The shares sum toDs​Xℓ​hD^\{\\ell h\}\_\{sX\}per occurrence, so the axes and the ranking score measure the same deviation, split by mechanism rather than two separately normalized magnitudes\. The meters are signed, so a component that partially cancels the other appears as a negative coordinate, and the value function includes the head’s output projection, so all quantities are in residual\-stream units\.

#### Lens 2: Top\-deviation context mining\.

The per\-occurrence deviationDs​Xℓ​hD^\{\\ell h\}\_\{sX\}can be computed for every contextXXin a corpus stream\. For each head we keep the top\-KKresults and record the surrounding text, the head’s top attended keys, and the actual next token\. We filter out occurrences whose deviation is an artifact of vocabulary truncation rather than of the head’s function; §[P\.2](https://arxiv.org/html/2609.16382#A16.SS2)gives the criteria\.

#### Lens 3: Vocabulary\-space decode\.

To read the deviation vectorΔi=𝐳s,Xiℓ​h−𝐳^s,Xiℓ​h\\Delta\_\{i\}\\;=\\;\\mathbf\{z\}^\{\\ell h\}\_\{s,X\_\{i\}\}\\;\-\\;\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s,X\_\{i\}\}in vocabulary space, we project it using the logit lens[nostalgebraist \(2020\)](https://arxiv.org/html/2609.16382#bib.bib46)\. To improve middle\-layer readouts, we use the additional J\-lens refinement of[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib30), which replacesWUW\_\{U\}withWU​JℓW\_\{U\}J\_\{\\ell\}, whereJℓJ\_\{\\ell\}is the model’s average linear account of how a vector inserted at layerℓ\\ellreaches the output\. §[P\.2](https://arxiv.org/html/2609.16382#A16.SS2)gives the estimation and the per\-occurrence decoding\.

### P\.2Measurement Protocol

#### The repeated\-text condition\.

Several readings below compare a head’s per\-occurrence deviation on natural text with its deviation on*repeated*text\. The construction is that of Experiment 1 \(§[6\.1](https://arxiv.org/html/2609.16382#S6.SS1)\): a real OpenWebText chunkccis presented as\[c;c\]\[c;c\], so that each position in the second copy has an exact earlier copy of its own context\. We sample query occurrences only from the second copy, since the survey reports one deviation per head per condition rather than a matched copy\-1 against copy\-2 difference; sampling from the second copy is also what leaves induction and duplicate detection something to act on\. And the pass covers300,000300\{,\}000tokens at stride 8, with the same position and vocabulary filters as above and against the same mean field\. Rise ranks quoted below are over all144144heads\.

#### Occurrence mining for Lens[P\.1](https://arxiv.org/html/2609.16382#A16.SS1.SSS0.Px2)\.

Lens[P\.1](https://arxiv.org/html/2609.16382#A16.SS1.SSS0.Px2)keeps the top\-KKoccurrences per head by per\-occurrence deviation\. We apply two additional filters: occurrences where the head places more than10%10\\%of its attention on out\-of\-vocabulary tokens are dropped, since the single averaged “other” value is then carrying much of the contribution, and windows containing garbled or rare\-Unicode text are dropped for the same reason\. The vector decoded is𝚫i=𝐳i−𝐳^i\\mathbf\{\\Delta\}\_\{i\}=\\mathbf\{z\}\_\{i\}\-\\hat\{\\mathbf\{z\}\}\_\{i\}, the residual against the context\-conditional mean\-field prediction, the same vector whose angle defines the deviation used for the mining \(note the contrast with §[P\.4](https://arxiv.org/html/2609.16382#A16.SS4), whose𝚫\\mathbf\{\\Delta\}is centered on the corpus type mean instead\)\. We decode each occurrence separately rather than averaging first, since averagingΔi\\Delta\_\{i\}over occurrences mixes distinct entities and topics, which could either create or wash out clusters\.

#### The readout of Lens[P\.1](https://arxiv.org/html/2609.16382#A16.SS1.SSS0.Px3)\.

The logit\-lens readout of a vector𝐮\\mathbf\{u\}inserted after attention at layerℓ\\ellissoftmax⁡\(WU​norm​\(𝐮\)\)\\mathrm\{softmax\}\(W\_\{U\}\\,\\mathrm\{norm\}\(\\mathbf\{u\}\)\), withnorm\\mathrm\{norm\}the final\-layer normalization\. This ignores the mediation between layerℓ\\elland the output, which at middle layers is substantial\. The refinement of[Gurnee et al\. \(2026\)](https://arxiv.org/html/2609.16382#bib.bib30)replacesWUW\_\{U\}withWU​JℓW\_\{U\}J\_\{\\ell\}, where

Jℓ=𝔼⁡\[∂𝐡t′\(L\)∂𝐡t\(ℓ\)\]J\_\{\\ell\}\\;=\\;\\mathbb\{E\}\\\!\\left\[\\frac\{\\partial\\mathbf\{h\}^\{\(L\)\}\_\{t^\{\\prime\}\}\}\{\\partial\\mathbf\{h\}^\{\(\\ell\)\}\_\{t\}\}\\right\]is the average, over positions and prompts of a natural corpus, of the Jacobian of the final residual with respect to the residual at layerℓ\\ell: the model’s own average linear account of how content inserted at layerℓ\\ellarrives at the output\. We estimateJℓJ\_\{\\ell\}once per model from OpenWebText\. At the final layer the estimate is proportional to the identity to nine decimals, a sanity check on the estimation\. The two readouts agree at late layers, where little mediation remains: at layer 9 of GPT\-2 the raw lens already broadly reproduces the J\-lens decode, and at layer 11 they coincide\. We report the J\-lens decode in the tables; where a head sits late enough for the two to agree, the raw lens gives the same reading\.

#### Split\-half stability\.

The ranking is stable across the corpus: recomputed end\-to\-end on each of two disjoint document\-interleaved halves, the two rankings have Spearman correlationρ=0\.999\\rho=0\.999over the144144heads and share1818of their top twenty, the two discrepancies falling near rank twenty\.

### P\.3What the Lenses Show

#### Natural text against repeated text\.

The repeated\-text comparison of §[P\.2](https://arxiv.org/html/2609.16382#A16.SS2)separates the top twenty by how each group responds to repetition, and shows where the corpus ranking mis\-scales\.

The induction heads rise the most\. On natural text their per\-occurrence deviations sit in the bottom third of the network, since natural text rarely contains the repetition they respond to, but under repeated textL5H1,L7H2,L7H10andL6H9show the four largest increases of any head, each between\+0\.48\+0\.48and\+0\.64\+0\.64\. The corpus score understates them, since their trigger is rare in natural text\.

The positional heads move the other way, the only group whose deviation*falls*under repetition:L2H7by the largest amount of any head,0\.3950\.395to0\.1660\.166, and the layer\-1 members by0\.040\.04–0\.060\.06\. When the only structure in the text is repetition, a head that transports absolute position becomes more predictable, not less\.

The remaining groups do not respond as units\. The letter mover L8H11 falls under repetition and patterns with the positional cluster rather than with the movers it is grouped with, andL8H1, which carries a nonzero induction score and the SAE survey’s own induction annotation, rises with the induction heads; the other band members change little\.

#### Head readings rest on occurrences, not type means\.

The readings below characterize heads from their occurrences rather than from a type\-level score, because a type’s mean can be mispredicted by a small consistent offset that produces no occurrence\-level effect\. The month effect ofL6H2shows why\. At type level it replicates under the current kernel: month types sit at median rank9696of999999\(Mann–Whitneyp=1\.4×10−4p=1\.4\\times 10^\{\-4\}\)\. But a full\-network sweep finds the effect is not unique, since L3H1 and L9H10 show stronger month elevation \(p=4\.5×10−5p=4\.5\\times 10^\{\-5\}and1\.0×10−51\.0\\times 10^\{\-5\}; three of144144heads survive a Bonferroni threshold\)\. And mining month*occurrences*directly gives a deviation difference of only\+0\.007\+0\.007\(p=0\.75p=0\.75\), while the same test detectsL9H8’s date sensitivity atp=7×10−8p=7\\times 10^\{\-8\}\.

#### Characterizing heads outside the circuit literature\.

For each head in Table[4](https://arxiv.org/html/2609.16382#S6.T4)with no documented circuit role, we ran the following procedure: mine the head’s eight highest\-deviation occurrences \(600600k OpenWebText tokens, deviation against the context\-conditional mean field\), then examine the context around the max\-deviation token, the logit lens of the moved value, and the J\-lens of the moved value \(the deviation vector𝚫=𝐳​WO−𝐳^\\mathbf\{\\Delta\}=\\mathbf\{z\}W\_\{O\}\-\\hat\{\\mathbf\{z\}\}\)\. Tables[17](https://arxiv.org/html/2609.16382#A16.T17),[18](https://arxiv.org/html/2609.16382#A16.T18),[19](https://arxiv.org/html/2609.16382#A16.T19)and[20](https://arxiv.org/html/2609.16382#A16.T20)give the retained occurrences for the four heads whose readings rest on this evidence\.

Table 17:L9H8: the eight highest\-deviation occurrences from600600k OpenWebText tokens, the top attended key with its share, and the top of the J\-lens decode of each occurrence’s deviation vector\. All 8 decode to the document’s topic\.Table 18:L5H2\. Its attention is on a specific nearby content token \(0\.40\.4–0\.80\.8\), and its deviation decodes to*context\-appropriate degree and function words*: a $1,0001\{,\}000\-a\-head ticket passage decodes to*minimum, whopping, coffers, maximum*; “little, if any” to*whatsoever, incidental*; a transfer\-rules passage \(“only then is B allowed”\) to*nor, authorised*; “one\-on\-one drills” to*thereof, dual, apiece*\. The head appears to move an abstract quantifier/degree completion computed from the local construction\. The reading of L5H2 as movement of degree or limitation in Table[4](https://arxiv.org/html/2609.16382#S6.T4)summarizes these occurrences\.Table 19:L8H3\. Its max\-deviation occurrences are*parallel and enumerated frames*: a repeated “There will beXXdays” construction \(decoding to*ones, those, highs*\), month–day lists, “Jane Doe No\. 1 and Jane Doe” \(decoding to*NAME, PID*, placeholder tokens\), a numbered listicle, a box\-office enumeration\. The attention is on the frame’s anchor \(0\.90\.9ondays,0\.70\.7on1\)\. This suggests structure\-frame tracking\. The reading of L8H3 as parallel and enumerated frames in Table[4](https://arxiv.org/html/2609.16382#S6.T4)rests on these occurrences\.Table 20:L6H8\. This head’s highest\-deviation contexts are*idiomatic comparative frames*: “as close …as you can,” “to say the least,” “about as simple as it could be,” “as quickly as I could,” with attention on the frame’s pivot \(as,sayat0\.50\.5–0\.80\.8\)\. The reading of L6H8 as idiomatic comparative frames in Table[4](https://arxiv.org/html/2609.16382#S6.T4)is based on these occurrences\.
#### Heads at the bottom of the ranking\.

Table[21](https://arxiv.org/html/2609.16382#A16.T21)lists the fifteen*lowest*\-deviation heads of each model under the deviation of \([17](https://arxiv.org/html/2609.16382#S6.E17)\), annotated with the behavioral repetition scores and, for GPT\-2, the head’s*mean\-field impact*rank, the signal complementary to deviation: a head with large impact carries type\-level geometry that the mean field reproduces, while a head with large deviation departs from that geometry in context\.181818To measure impact we ablate the head from the rollout of §[4](https://arxiv.org/html/2609.16382#S4), dropping its kernel termPℓ​hP^\{\\ell h\}and its value contribution at layerℓ\\ellwhile leaving the other heads, the MLPs and the residual structure intact, re\-run the open\-loop prediction from the input centroids, and record the drop in mean\-over\-layers centered cosine\. The two columns are therefore scored against different kernels: the corpus kernelPPfor impact, the context\-averaged context\-conditional kernel forD¯\\bar\{D\}\.The parenthesized IOI labels are that circuit’s own head names, whose three duplicate\-token heads are 0\.1, 0\.10 and 3\.0\([Wang et al\., 2023a](https://arxiv.org/html/2609.16382#bib.bib69)\), and “FV core head” marks a head in the task\-norm top\-10 on at least one of the three tasks of §[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)\. A head is left blank rather than guessed: none of the blank entries is named by the IOI circuit\([Wang et al\., 2023a](https://arxiv.org/html/2609.16382#bib.bib69)\)or the acronym circuit\([García\-Carrasco et al\., 2024](https://arxiv.org/html/2609.16382#bib.bib23)\), and we do not fall back on the unrefereed SAE head survey here\. Membership in the table is not an artifact of which kernel scores it: against the corpus kernelPPinstead, GPT\-2 keeps thirteen of the fifteen, with L1H1 and L4H4 replacingL1H0andL2H3, and Gemma keeps all fifteen in a different order\.

GPT\-2 \(medianD¯=0\.272\\bar\{D\}=0\.272\)

Gemma\-2\-2B \(medianS¯ℓ​h=0\.149\\bar\{S\}^\{\\ell h\}=0\.149\)

Table 21:The fifteen lowest\-deviation heads of each model\.S¯ℓ​h\\bar\{S\}^\{\\ell h\}is the1−cos⁡θ1\-\\cos\\thetadeviation of \([17](https://arxiv.org/html/2609.16382#S6.E17)\) against the context\-averaged context\-conditional kernel, as in the body\. “MF\-impact rank” is the head’s rank of 144 by mean\-field impact, 1 = largest, bold in the top eight\. “duplicate”, “prev\-token” and “BOS\-sink” are attention shares on repeated random sequences, on the query type’s earlier occurrence, the immediately preceding position, and the BOS token respectively, listed when they exceed0\.100\.10,0\.150\.15and0\.350\.35\.We observe three patterns\.\(i\) The bottom of the ranking is where the type\-level geometry lives\.In GPT\-2, four of the six most mean\-field\-predictable heads are simultaneously among the top eight heads by mean\-field impact: these layer\-0 heads do almost exactly what the kernel says, and what the kernel says carries much of the centroid prediction\. The two readings dissociate: measured against the mean field these heads are the most predictable in the network, while measured within it they are among the most important\.\(ii\) The duplicate\-token machinery populates the bottom in both models\.All three GPT\-2 IOI duplicate\-token heads and the strongest behavioral duplicate head \(L0H5, at0\.720\.72\) sit here, as do Gemma’s two lowest\-deviation heads plus three more of its bottom fifteen\. “Attend to your own type” is the diagonal ofWs​tW\_\{st\}, so duplicate detection is expressible in the mean field and therefore invisible to the deviation\.\(iii\) Function\-vector core heads sit at the bottom in Gemma\-2\-2B\.Four of Gemma’s \(L12H6, L25H7, L19H4, L6H0\) are in its bottom fifteen, L6H0 on the capital task\. GPT\-2’s function\-vector heads sit low but not this low: L11H0 ranks1919th of144144and L11H112626th, so we state this pattern for Gemma only\. The heads whose task residuals causally transport tasks in §[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)are thus among the*most*mean\-field\-predictable heads on generic text in Gemma\.

### P\.4Layer\-1 Positional Transport: Re\-encoding the Query’s Position

This appendix details the positional analysis of GPT\-2’s early\-layer deviation referenced in §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3): whether a head’s within\-type output variation is*positional*rather than lexical\. This is visible because GPT\-2 adds a learned positional embedding to the residual stream, so at layer 0 a head’s within\-type variation is exactly its OV map applied toWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]\. On the other hand, Gemma\-2 uses RoPE, which keeps absolute position out of the values\.

All analyses here use GPT\-2 small and Gemma\-2\-2B over OpenWebText; the within\-type write deviation of a head is𝚫=𝐳s​Xℓ​h−𝐳¯sℓ​h\\mathbf\{\\Delta\}=\\mathbf\{z\}^\{\\ell h\}\_\{sX\}\-\\overline\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}withs=T⁡\(q\)s=T\(q\), its per\-occurrence output minus the corpus \(or sample\) mean output𝐳¯sℓ​h\\overline\{\\mathbf\{z\}\}^\{\\ell h\}\_\{s\}for the query’s token type\. Note that𝚫\\mathbf\{\\Delta\}is a different quantity from the deviationDMFD\_\{\\mathrm\{MF\}\}of §[P\.1](https://arxiv.org/html/2609.16382#A16.SS1), which compares𝐳s​Xℓ​h\\mathbf\{z\}^\{\\ell h\}\_\{sX\}to the*per\-context*mean\-field prediction𝐳^s​Xℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}\. We useDMFD\_\{\\mathrm\{MF\}\}to*find*the cluster \(Table[4](https://arxiv.org/html/2609.16382#S6.T4)\), the five positional heads L1H2, L1H3, L1H8, L1H10, and L2H7, and𝚫\\mathbf\{\\Delta\}to*analyze*it, since the appendix’s question is what drives the total within\-type variation of these heads’ writes\. Because𝐳^s​Xℓ​h\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{sX\}is built from type kernels and type means, it carries no positional dependence, so the positional variance measured below is insensitive to which of the two baselines is subtracted\.

#### Positional fraction\.

For each head we fit each coordinate of𝚫\\mathbf\{\\Delta\}, by ordinary least squares, as a smooth function of the normalized query positionp~=p/T\\tilde\{p\}=p/T: the regressors are an intercept together with the basis\{p~,p~2,sin\(2πkp~\),cos\(2πkp~\):k=1,…,4\}\\\{\\tilde\{p\},\\ \\tilde\{p\}^\{2\},\\ \\sin\(2\\pi k\\tilde\{p\}\),\\ \\cos\(2\\pi k\\tilde\{p\}\):k=1,\\dots,4\\\}\(1111in all\)\. We report*posVar*, the fraction of𝚫\\mathbf\{\\Delta\}’s variance*about its mean*that the fitted position function explains; the intercept absorbs the mean and contributes nothing to posVar\. This is a more conservative measure than*decoding*position from𝚫\\mathbf\{\\Delta\}\(a2020\-dimensional positional component is easily decodable while contributing little variance\)\. We probe the high\-deviation heads studied in the text, the highest\-deviation head in each layer, and control heads; Gemma uses the same procedure\.

#### What is moved \(decoding\)\.

Knowing that the cluster’s output varies with position \(posVar\) does not say*what*positional content it writes\. There are three candidates: the*absolute*position of the query; the absolute position of the attended key; or the*relative*offset between them\. Orthogonally, the written content may be merely position\-correlated, or a deterministic function of the model’s own position representation, a linear image of the embedding rowWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]\. Restricting to the layer\-1 cluster \(L1H2/H3/H8/H10\) andL2H7, four measurements separate these\. \(i\) Ridge regression decodes the query’s absolute position from𝚫\\mathbf\{\\Delta\}atR2≈0\.67R^\{2\}\\approx 0\.67\(position\-shuffled baseline≈0\\approx 0\)\. \(ii\) Regressing𝚫\\mathbf\{\\Delta\}directly on the embedding rowsWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]shows that0\.610\.61–0\.670\.67of the write’s variance is a linear image ofWposW\_\{\\mathrm\{pos\}\}: the write is a fixed linear function of the model’s own absolute\-position representation, not some other position\-correlated signal\. \(iii\) Specificity: a subset of layer\-1 heads carriesWposW\_\{\\mathrm\{pos\}\}\(variance0\.560\.56–0\.670\.67\) while others do not \(≤0\.13\\leq 0\.13\), and position per se remains broadly decodable even from non\-carriers \(for example, L2H1 atR2​0\.56R^\{2\}\\,0\.56withWposW\_\{\\mathrm\{pos\}\}variance≤0\.04\\leq 0\.04\), soWposW\_\{\\mathrm\{pos\}\}\-transport, not mere positional decodability, is the discriminating property; the carriers’ position\-readout directions are mutually near\-orthogonal \(\|cos\|≤0\.11\|\\cos\|\\leq 0\.11\), each head writing its tag into its own subspace\. \(iv\) Relative\-code control\. A relative code predicts that the write is a function of the query–key offseto=p−ko=p\-k\. We measured each head’s own attended\-key position on fresh text and partitioned𝚫\\mathbf\{\\Delta\}’s variance between offset features \(one\-hot small offsets plus coarse bins\) andWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]\. The cluster’s attention geometry is heterogeneous, and the relative hypothesis fails differently in each regime:

- •L1H2attends locally\(offset s\.d\.2\.02\.0, median11\), so a relative code would predict a nearly constant write\. Instead,0\.670\.67of𝚫\\mathbf\{\\Delta\}’s variance sweeps through a linear image ofWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]asppranges over88–191191\.
- •L1H10’s offset varies widely\(s\.d\.7070, spanning00–161161\), so a relative code has ample variance to appear in, but offset features explain only0\.260\.26of𝚫\\mathbf\{\\Delta\}’s variance against0\.660\.66forWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\], and add nothing \(\+0\.00\+0\.00\) onceWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]is included: the offset\-correlated part of the write is just the part that co\-varies with absolute position\.
- •L1H3,L1H8, andL2H7attend to BOS, for9090–100%100\\%of query positions\. The question degenerates for them: the offset to a fixed key at position zero is the absolute position, so no regression could separate the two hypotheses, and what identifies the content is finding \(ii\), that0\.610\.61–0\.650\.65of the write is a linear image ofWposW\_\{\\mathrm\{pos\}\}\.

GPT\-2 has no relative\-position mechanism, position entering only through the learned absolute tableWposW\_\{\\mathrm\{pos\}\}, so an absolute tag has a source in the residual stream while a relative one would have to be computed from scratch\. The moved content is therefore an*absolute position tag*, a deterministic image of the key’s absolute position, and not a relative offset or incidental position correlation\.

#### Mechanism: the BOS gate\.

The relative\-code control leaves a puzzle\. ForL1H3,L1H8, andL2H7the top\-attended key is BOS, and the BOS value is the same vector at every query position, carrying no positional variance, so these heads cannot be reading position out of the token they attend to\. Their write nonetheless varies withppthrough the QK side: the query at positionppcarriesWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]into the attention computation, so the amount of attention parked on BOS can depend onppeven though the value retrieved there does not\. Measurement confirms this\. The scalar attention\-on\-BOS,a⁡\(p\)a\(p\), is a nearly deterministic smooth function of the query position: regressingaaonWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]givesR2=0\.996R^\{2\}=0\.996\(L1H3\),0\.9350\.935\(L1H8\), and0\.9110\.911\(L2H7\)\. That one scalar by itself explains0\.360\.36,0\.230\.23, and0\.410\.41of𝚫\\mathbf\{\\Delta\}’s total variance respectively, and the top singular direction of the positional write, the line through the fixed BOS\-value image, carries0\.660\.66–0\.770\.77of it; the remainder comes from the complementary tail of attention over nearby keys, whose distribution also shifts withpp\. The positional write of every head in the cluster is effectively rank33: the top three singular directions of theWposW\_\{\\mathrm\{pos\}\}\-fit carry all of its variance,L1H2and L1H10 included\. Each head thus emits the query’s position as a coefficient pattern over about three fixed write directions\. The mechanism is a*gate*: the head readsppoff its own QK circuit, leans on the BOS value as a fixed reference vector, and generates a position\-dependent output, with nothing positional moved from the attended token\. This is a concrete functional use for the attention\-sink resting behavior of early heads\.

#### Magnification versus re\-encoding\.

The residual stream at positionppalready containsWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\], deposited by the embedding, so a head whose output is a function ofppmight simply be amplifying \(or attenuating\) the code that is already there\. The two possibilities separate in write space: if the cluster magnifies, its positional write should lie inside the subspace spanned by the existing code and align withWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]pointwise; if it re\-encodes, the write should occupy new directions\. The top ten principal components of the centered codeWpos\[8:191\]W\_\{\\mathrm\{pos\}\}\[8\{:\}191\]capture99\.8%99\.8\\%of its variance, yet the heads’ positional writes place little of their own variance in that subspace:5\.0%5\.0\\%\(L1H2\),2\.3%2\.3\\%\(L1H3\),0\.2%0\.2\\%\(L1H8\),7\.1%7\.1\\%\(L1H10\), and18\.8%18\.8\\%\(L2H7\)\. The pointwise cosine between the write andWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]is near zero for every head \(\|cos\|\|\\cos\|between0\.010\.01and0\.180\.18\)\. The write is also as large as or larger than the code itself \(RMS norm ratios0\.970\.97–1\.901\.90\)\. The cluster is therefore writing a*second*position code, at full amplitude, rather than reinforcing the embedding’s\. The ablation supplies the causal evidence: it removes only the cluster heads’ position subspace from their outputs, leaving the embedding’sWpos​\[p\]W\_\{\\mathrm\{pos\}\}\[p\]untouched in the residual stream, and loss still rises by\+0\.0117\+0\.0117nats,38×38\\timesthe controls\. The original code does not supply what the layer\-2 consumers need: downstream reads the re\-encoded dimensions\.

#### The geometry of the new code\.

Do the five heads write position into the*same*new subspace, or does each keep its own? Each head’s positional write is effectively two\-dimensional \(the top two singular directions carry0\.970\.97–0\.980\.98of it\), so we compare the heads pairwise by the meancos2\\cos^\{2\}of the principal angles between their write subspaces \(chance level for random subspaces of this size in768768dimensions:0\.0040\.004\)\. The overlaps form a spectrum:

- •The two non\-BOS heads nearly share a code\.L1H2and L1H10 overlap at0\.610\.61, and in variance\-weighted terms0\.850\.85of L1H10’s positional write lies insideL1H2’s span: the heads that attend to real \(non\-BOS\) keys write into substantially the same channels\.
- •L1H8andL2H7are the most private\.Their mutual overlap is0\.020\.02, essentially disjoint, and each overlaps the remaining heads only modestly \(0\.020\.02–0\.270\.27forL1H8;0\.140\.14–0\.310\.31forL2H7\)\.L2H7is the cluster’s layer\-2 member, writing after the layer\-1 heads\.
- •No pair is orthogonal either\.Every overlap sits well above the0\.0040\.004chance floor\. The code is a partially redundant bus: a few heavily shared channels plus per\-head private ones\.

The union of these partially overlapping channels sits inside the2020\-dimensional cluster subspace, capturing0\.880\.88of the combined output’s variance, used in the ablation below\.

The shape of the code within each channel is also informative\. A BOS\-gated head might encode its distance from BOS in the*magnitude*of what it writes\. The gauge is real: for the BOS\-attending heads the attention on BOS is strongly*monotone*in the query position \(Spearmanρ\\rhoof−0\.85\-0\.85forL1H3,−0\.93\-0\.93forL1H8, and\+0\.94\+0\.94forL2H7; the layer\-1 pair releases attention from BOS asppgrows whileL2H7accumulates it\), so each head maintains a graded reading of how far into the context it is\. But position is*not*encoded in the write’s magnitude: the norm of the positional write correlates only weakly withpp\(\|ρ\|≤0\.31\|\\rho\|\\leq 0\.31for these heads\)\. Instead the*signed coefficient*along the leading axis sweeps monotonically from positive through zero to negative asppgrows \(ρ=−0\.93\\rho=\-0\.93,−0\.96\-0\.96,\+0\.96\+0\.96\), making the norm U\-shaped, smallest mid\-context where the coefficient crosses zero, while the second axis’s coefficient traces an arc\. The code is thus a smooth two\-dimensional curve, roughly \(monotone trend, curvature\), traversed asppadvances:*coordinate*coding along fixed axes, not magnitude coding\. This is the same format as the embedding’s own code: the participation ratio of the centeredWpos\[8:192\]W\_\{\\mathrm\{pos\}\}\[8\{:\}192\]spectrum is2\.02\.0effective dimensions, so the original code is likewise a low\-dimensional curve with position given by location along it\. The re\-encoding therefore preserves the representational format, a smooth two\-dimensional position curve, while relocating it into channels the embedding does not occupy, with per\-head gain and a per\-head warp of the position axis\.

#### Why re\-encode?

We flag this paragraph as interpretation rather than measurement\. The embedding’s code is a single shared signal, frozen in its geometry and entangled with token content in the same residual directions; it cannot be scaled, cleaned, or reshaped for a particular consumer\. Republishing position through attention buys three things\. \(i\)*Clean channels*: each carrier’s write is position and essentially nothing else \(posVar0\.900\.90–0\.990\.99\), in directions of its own, so a downstream QK circuit can read position without also picking up token content\. \(ii\)*Independent gain*: each channel’s amplitude is set by a head that can be modulated separately from the embedding and from the other carriers\. \(iii\)*A reshaped scale*: the gatea⁡\(p\)a\(p\)is a softmax\-derived function of position, so the re\-encoded signal can warp the position axis, compressing late positions or sharpening early ones, in ways the fixed embedding cannot\. On this reading the cluster republishes the query’s absolute position in a standardized, consumable form, and the dereferencing experiment below identifies the subscribers\.

#### Dereferencing \(causal ablation\)\.

We compute the2020\-dimensional subspaceSSinto which the cluster writes position, the top principal directions of theWposW\_\{\\mathrm\{pos\}\}\-fit of the cluster heads’ combined per\-head output, capturing0\.880\.88of its variance, and, mid\-forward, project that component out of the cluster heads’ outputs\. We compare the next\-token\-loss increase to two controls: removing the*same*SSfrom the layer\-1 heads that do not write into it \(specificity\), and removing a random equal\-dimension subspace\. The carrier ablation costs\+0\.0117\+0\.0117nats versus\+0\.0003\+0\.0003\(non\-carrier\) and\+0\.0011\+0\.0011\(random\), a38×38\\timesspecificity ratio\. To find the consumers we record, per downstream head, theL2L\_\{2\}change in its attention pattern under the ablation; the change is concentrated in layer\-2 heads and decays with depth\.

#### Function \(category breakdown\)\.

Under the same ablation we group the per\-token loss increase by the category of the token being predicted: tokens inside a number, word\-initial tokens, within\-word subword continuations, and punctuation\. The increase is largest for numeric tokens \(\+0\.028\+0\.028,2\.6×2\.6\\timesthe average\), moderate for word\-initial tokens \(\+0\.016\+0\.016\), negligible for punctuation \(\+0\.002\+0\.002\), and absent for subword continuations \(−0\.006\-0\.006\), identifying forward sequence progression \(next\-word and numeric/ordinal continuation\), not detokenization, as the supported behavior\.

### P\.5Gemma\-2\-2B Head Survey

We repeat the head survey of §[6\.3](https://arxiv.org/html/2609.16382#S6.SS3)on Gemma\-2\-2B \(26 layers, 8 query heads each, grouped\-query attention, rotary position embeddings\)\. The measurements are those of §[P\.1](https://arxiv.org/html/2609.16382#A16.SS1), computed from a 2M\-token OpenWebText pass over the model’s208208heads, with the same support mask \(C≥20C\\geq 20\) and budget as the GPT\-2 survey in the body\. No circuit catalog exists for Gemma, so the survey tests whether the method reproduces itself but not whether its readings agree with an independent account\. Table[22](https://arxiv.org/html/2609.16382#A16.T22)lists the fifteen heads with the largest deviation\.

Table 22:Gemma\-2\-2B: the fifteen heads with the largest corpus deviationS¯ℓ​h\\bar\{S\}^\{\\ell h\}\([17](https://arxiv.org/html/2609.16382#S6.E17)\), over the999999OpenWebText query types with at least5050occurrences\.The ranking again recovers induction machinery without supervision\. The model’s strongest behavioral induction head, L15H0 \(induction0\.940\.94, the largest of all208208heads\), ranks third, and three more induction heads follow in the top fifteen \(L4H4, L17H3, L22H4\)\. Across all208208heads the deviation–induction Spearman correlation isρ=0\.45\\rho=0\.45\.

The additive split places the induction heads on the*covariance*side: L15H0 draws a covariance share of0\.600\.60against a network mean of0\.520\.52, and L4H4 sits at0\.630\.63\. This matches GPT\-2, where the strongest induction head is also the most covariance\-leaning of the documented heads \(Table[4](https://arxiv.org/html/2609.16382#S6.T4)\): attending to whatever followed the query’s earlier occurrence is a pattern effect, not a value effect\. The covariance lean is a property of these particular heads, not a network trend; across all208208heads induction score and covariance share are mildly*anti*\-correlated \(ρ=−0\.31\\rho=\-0\.31\)\. These heads are also at rest on generic text: on the per\-occurrence survey L15H0 scores1−cos⁡θ=0\.0701\-\\cos\\theta=0\.070, rank203203of208208, the trigger\-dependent signature GPT\-2’s induction heads also show\.

Unlike GPT\-2, no early\-layer positional cluster appears, since RoPE keeps absolute position out of the values \(§[P\.4](https://arxiv.org/html/2609.16382#A16.SS4)\)\. The top of the per\-occurrence survey is instead a dense mid\-stack band: all2020of its top2020heads lie in layers77–1717, median layer1111\. The band leans covariance rather than contextualization, mean splitMcov=0\.43M\_\{\\mathrm\{cov\}\}=0\.43againstMctx=0\.32M\_\{\\mathrm\{ctx\}\}=0\.32\.

## Appendix QFunction\-Vector Extraction: Baselines, Controls, and Results

Here we provide supporting evidence and additional results for §[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)\. We first validate that our average\-indirect\-effect \(AIE\) baseline reproduces the published[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)selection, that the null subtraction is what makes MFA selection work, and that the head\-to\-head comparison does not depend on the injection protocol\. We then present the task\-by\-task results behind §[6\.4](https://arxiv.org/html/2609.16382#S6.SS4)and close with the full analysis of what an injected vector contains at GPT\-2 small’s scale\.

### Q\.1Validations

The results below compare two ways of selecting function\-vector heads: the MFA task\-residual and[Todd et al\.](https://arxiv.org/html/2609.16382#bib.bib67)’s average indirect effect \(AIE\)\. In both cases we take the top\-10 of Gemma\-2\-2B’s208208heads; the MFA selection is stable across the three splits \(sharing77–1010of ten heads\), and the AIE selection is held fixed per task \(2020clean and1010corrupted prompts\)\.

#### The AIE baseline reproduces the published selection\.

We first anchor to GPT\-J\-6B and use the ten universal heads of[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)verbatim \(L15H5, L9H14, L12H10, L8H1, L11H0, L13H13, L8H0, L14H9, L9H2, L24H6\), which fixes head\-for\-head comparability with their results\. Running our own AIE procedure over all448448GPT\-J heads recovers77,88, and77of those ten on the antonym, capital, and past tasks, and reproduces their top five exactly on the antonym task, even though their set is averaged over∼40\{\\sim\}40tasks and ours is computed individually per task\. Thus, we confirm that our AIE arm is a faithful reimplementation\. That fidelity comes at a cost the MFA selection avoids: AIE sweeps every head with a patched forward pass per corrupted prompt \(448448prompts on GPT\-J\), where the task residual is read from a single forward pass\. To apply MFA to GPT\-J we estimate the type\-level null at ‘:’ directly, as the corpus\-mean write of each head at ‘:’ positions \(241241occurrences from an OpenWebText slice\), rather than building full mean\-field statistics; this is the quantity𝐳^ℓ​h‘:’\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{\\text\{\`:'\}\}approximates elsewhere\. GPT\-2 small is treated separately \(§[Q\.3](https://arxiv.org/html/2609.16382#A17.SS3)\)\.

#### The null subtraction is what selects task heads\.

MFA ranks heads by the magnitude of the deviation \([18](https://arxiv.org/html/2609.16382#S6.E18)\), which subtracts the corpus baseline from the head’s task\-average output\. The body assumes throughout that this subtraction is what matters, so we first check that it carries the selection\. To do so, we define a raw\-magnitude control, which ranks by the same average with nothing subtracted\. The two rankings agree on only55–66heads of ten \(on the original 20\-pair pools\), and transfer under the raw\-magnitude selection collapses on capital \(0\.000\.00at every scale\):*without the subtraction, selection is captured by heads whose outputs are large for reasons unrelated to the task, which are useless as function vectors\.*

#### The injection protocol does not bias the comparison\.

We consider two injection protocols\. In the first \(Table[5](https://arxiv.org/html/2609.16382#S6.T5)\), we sum the extracted vectors and add them to the residual; in the second \(reported only in this appendix\) we inject each extracted vector into its corresponding head output\. The two head\-to\-head tables inject the selected heads differently \(§[Q\.2](https://arxiv.org/html/2609.16382#A17.SS2)\), so we check that the choice does not reorder the selections it is used to compare\. Per\-head injection is uniformly stronger than summing the heads into one vector, by0\.1270\.127averaged over the7272paired task×\\timessplit×\\timesselection×\\timesconstruction cells \(0\.5700\.570against0\.4420\.442, Wilcoxonp<10−4p<10^\{\-4\}\)\. That penalty falls almost equally on the two selections \(−0\.125\-0\.125for AIE against−0\.130\-0\.130for MFA, Mann–Whitneyp=0\.83p=0\.83\), even though the AIE heads sit at L12–L15, much further from the source layer than the MFA heads at L6–L9\. The protocol rescales the comparison without reordering it, so the selection verdicts hold under either table\.

### Q\.2Detailed Results

Table 23:Gemma\-2\-2B: zero\-shot transfer for each combination of head selection and vector construction, with the overlap of the two selections\. Bold: best per task, ties within0\.020\.02\. These cells use per\-head injection, each selected head’s vector added at that head’s own layer at scale1010, and are therefore not the same measurement as Table[5](https://arxiv.org/html/2609.16382#S6.T5), which uses the summed single\-vector injection of[Todd et al\. \(2024\)](https://arxiv.org/html/2609.16382#bib.bib67)\.These tables separate two choices in the pipeline: which heads are selected, and how their vectors are constructed\.

#### Gemma\-2\.

We first consider head selection\. In Gemma\-2, the two head selections overlap by only22–33heads of ten, but neither dominates\. Under the per\-head injection \(Table[23](https://arxiv.org/html/2609.16382#A17.T23)\) AIE selection is superior on two tasks \(capital,0\.900\.90vs\.0\.200\.20, and currency,0\.760\.76vs\.0\.000\.00\), MFA head selection is superior on three tasks \(past, language, and translation, with confidence intervals disjoint on the latter two\), and the two are statistically indistinguishable on antonym\. Under the summed injection of Table[5](https://arxiv.org/html/2609.16382#S6.T5)the same picture holds with language moving to a tie, which is the one per\-task verdict the protocol changes\. AIE also has less headroom on some tasks than others\. It scores heads by recovery over a label\-corrupted baseline, so where the answer is guessable from the query that baseline is already high: mean corrupted\-prompt answer probability is0\.350\.35\(antonym\) and0\.280\.28\(currency\), against0\.170\.17\(capital\),0\.160\.16\(language\),0\.110\.11\(past\), and0\.040\.04\(translation\)\. Its ranking is correspondingly noisier on the guessable tasks\. The two selections differ because they look in different places: on every task the AIE top\-10 concentrates on the same compact mid\-stack group \(L14H0 is rank 1 on all six tasks, and L12H1, L14H1, L13H4, L14H3, L13H5, L15H3 recur\), while the MFA norms select earlier\-layer writers \(an L6–L9 band\) whose selection varies more by task\. On the three added tasks the recurring group is recovered in its entirety\.

Turning to vector construction, the MFA deviation dominates: under per\-head injection, across the twelve selection×\\timestask cells the deviation vector matches or beats the raw average in ten, the two exceptions both on capital and both within the0\.020\.02tie margin, with decisive margins exactly where the raw average is weakest \(\+0\.23\+0\.23on currency,\+0\.21\+0\.21on translation, both with AIE\-selected heads; at most0\.070\.07on MFA\-selected heads, where the largest shortfall is0\.010\.01, on capital\)\. Under the summed injection of Table[5](https://arxiv.org/html/2609.16382#S6.T5)this advantage grows on AIE\-selected heads \(\+0\.200\+0\.200against\+0\.106\+0\.106\) and reverses on MFA\-selected heads, from\+0\.018\+0\.018\(n\.s\.\) to−0\.036\-0\.036\(p=0\.018p=0\.018\)\.

#### GPT\-J\.

With[Todd et al\.](https://arxiv.org/html/2609.16382#bib.bib67)’s published set as reference, MFA head selection recovers7/107/10\(antonym\),5/105/10\(capital\), and4/104/10\(past\) of their heads, and its transfer ties theirs on all three tasks; the apparent antonym gap seen in smaller pools does not survive atn=120n=120\(Table[24](https://arxiv.org/html/2609.16382#A17.T24); scale 2 is used throughout on GPT\-J\)\. Unlike Gemma\-2, the raw\-norm control and MFA\-norm selections agree on GPT\-J: the function\-vector carriers there are not masked by generic\-content heads\.

Table 24:GPT\-J\-6B, the analog of Table[23](https://arxiv.org/html/2609.16382#A17.T23), with Todd et al\.’s published causal set in place of the AIE selection\. Overlap is measured on the original 20\-pair pools\. No cell is bolded: within each task the columns lie within overlapping confidence intervals\.
#### Overall\.

No single pipeline wins outright, but their union covers all six tasks: every task has a FV transported at0\.600\.60–0\.970\.97by at least one selection\-vector combination\. On GPT\-J \(Table[24](https://arxiv.org/html/2609.16382#A17.T24)\) the two selections are statistically indistinguishable on every task despite choosing largely different heads, with every within\-task difference inside the95%95\\%confidence intervals, which is why no cell is marked; random ten\-head controls transfer at0\.000\.00–0\.010\.01\. In other words, where the task signal is strong the pipelines are interchangeable, and where it is weak they fail in different places\.

### Q\.3Decoding Category Vectors

GPT\-2 is competent at two of the three original tasks \(few\-shot0\.540\.54capital and0\.900\.90past tense on the6060\-pair pools\)\. On capital, injecting the deviation vector makes the model’s top reply a capital city on all120120held\-out queries, but \(aside from one lucky guess\) the city it names is incorrect\. Past tense behaves differently: the deviation vector reaches0\.220\.22accuracy, on a task whose answer can be obtained from the query itself by a superficial change, but it is also inconsistent on providing categorical answers\. To see whether this reflects what the vector contains or only how the model reads it out, we decode the task residuals and the injected vectors directly, for GPT\-2 and, as a contrast, Gemma\-2\.

#### Task residuals are decodable via J\-Lens\.

The lenses of §[P\.1](https://arxiv.org/html/2609.16382#A16.SS1)may be used as well with task\-specific deviations\. We demonstrate for GPT\-2, by decoding each head’s*per\-prompt*residual𝐳ℓ​h‘:’,X−𝐳^ℓ​h‘:’\\mathbf\{z\}^\{\\ell h\}\_\{\\text\{\`:'\},X\}\-\\hat\{\\mathbf\{z\}\}^\{\\ell h\}\_\{\\text\{\`:'\}\}throughJℓJ\_\{\\ell\}\([Gurnee et al\., 2026](https://arxiv.org/html/2609.16382#bib.bib30)\)for its layer and recording the rank of that prompt’s correct answer \(in its leading\-space token form, the form the corrected prompt format elicits\)\. The MFA residual for top task\-residual heads names the answer or its ingredients via the J\-lens\. On country→\\tocapital, L8H11’s andL9H8’s decodes have median answer rank55and66of50,25750\{,\}257, L10H3’s2121, and the name mover L10H0’s1010, with individual prompts exact: for*Sweden :*, L10H0’s residual decodesStockholmat rank 0 \(top\-3:*Stockholm, Swedish, Sweden*\), and L10H3’s top\-3 is a capital\-city family \(*Helsinki, Tokyo, London*\)\. The division of labor is visible in the decodes:L9H8\(the aggregator of Table[17](https://arxiv.org/html/2609.16382#A16.T17)\) writes the query’s*category*\(*Swedish, Swed, Sweden*; rank 6 because the answer is nearby in that topical family\), while L10H0 and L10H3 write the*answer*family\. On present→\\topast the best decoding heads are L11H5 \(median rank6767; for*show :*its top\-3 is*show, showed, show*\) and L9H1 \(377377\)\. Even L8H6’s poorly\-decoding residual \(median380380\) is category\-formatted: its top tokens are past forms of the*wrong*verbs \(*prayed, laughed, drank*\), the category\-without\-binding behavior established below\. On antonym, the task at GPT\-2’s capability floor, no head’s residual decodes the answer well \(best median rank341341\)\. As in the catalog, per\-prompt decoding, not averaging, is what makes the content legible\.

#### Memorization versus binding\.

Decoding the aggregated injected vector itself \(each top head’s component through the J\-lens at its layer\) allows us to distinguish FV content from model behavior, and sharpens the answers\-versus\-category contrast between GPT\-2 and Gemma\-2\. We rank four token populations in each decode: the ten extraction\-pool answers \(seen in prompts\), the ten held\-out evaluation answers, fifteen same\-category words appearing nowhere in the task, and fifteen unrelated words\. On GPT\-2 the best\-decoding capital head \(L10H3\) ranks all three answer populations at the top of the vocabulary: seen answers at median rank99, held\-out at3737, out\-of\-task capitals at3434, against∼21,000\{\\sim\}21\{,\}000for unrelated words\. The GPT\-2 function vector’s*content*is thus category\-wide, and the behavioral parroting reflects a small but consistent edge for seen answers at the very top ranks, which argmax amplifies into a deterministic preference\. On the past task the gradient is steeper \(seen4444, held\-out265265, out\-of\-task1,4461\{,\}446\), closer to answer storage\.

In contrast, Gemma\-2’s function vectors show*no seen\-answer preference at all*: its best capital head \(L17H0\) ranks seen/held\-out/out\-of\-task at23/52/8923/52/89of a256256k vocabulary, and on the past task the held\-out answers outrank the seen ones \(1,5501\{,\}550vs3,0283\{,\}028\)\.

These results suggest that GPT\-2’s vector remembers the answers it saw, with the category elevated beneath them; Gemma\-2’s vector carries the task with no memory of which exemplars taught it\. GPT\-2’s deficit is thus better described as a failure of argument*binding*at readout than as an absence of category content\.