句法回路的涌现与稀疏化

# The Emergence and Sparsification of a Syntactic Circuit
Source: [https://arxiv.org/html/2610.33299](https://arxiv.org/html/2610.33299)
Sasha BoguraevAffiliation:The University of Texas at AustinEmail:[sasha\.boguraev@utexas\.edu](mailto:)Tiago PimentelAffiliation:ETH ZurichEmail:[kyle@utexas\.edu](mailto:)Kyle MahowaldAffiliation:The University of Texas at AustinEmail:[tiago\.pimentel@inf\.ethz\.ch](mailto:)

###### Abstract

To understand the emergence of grammatical behavior in language models, we track circuits for subject–verb number agreement inpythia\-1band in ten seeds ofpythia\-410macross training\. We find that subject–verb agreement is learned early, and is initially handled by a diffuse collection of neurons\. However, after agreement*behavior*has stabilized, the*circuit*responsible for it continues to reorganize\. The attention heads which mediate agreement often shift during training: in the case ofpythia\-1b, behavior is controlled by two heads early in training but by a third \(different\) head later\. Furthermore, the circuit shrinks over training, withpythia\-1b’s final circuit being strikingly sparse\. We take these results to be informative about how linguistic knowledge emerges and is represented in language models\.

## 1Introduction

Our goal is to tell the story, in as much detail as possible, of how subject–verb number agreement emerges across training in language models \(LMs\)\. Number agreement has long served as a probe for syntactic competence in language models\([Linzen et al\., 2016](https://arxiv.org/html/2610.33299#bib.bib33);[Gulordava et al\., 2018](https://arxiv.org/html/2610.33299#bib.bib21);[Marvin and Linzen, 2018](https://arxiv.org/html/2610.33299#bib.bib35);[Futrell et al\., 2019](https://arxiv.org/html/2610.33299#bib.bib17);[Mueller et al\., 2020](https://arxiv.org/html/2610.33299#bib.bib38);[Warstadt et al\., 2020](https://arxiv.org/html/2610.33299#bib.bib53);[Wei et al\., 2021](https://arxiv.org/html/2610.33299#bib.bib54)\)\. It has also been used to probe how these models encode grammatical information internally\([Finlayson et al\., 2021](https://arxiv.org/html/2610.33299#bib.bib15);[Lasri et al\., 2022](https://arxiv.org/html/2610.33299#bib.bib32);[Arora et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib3)\)\. Notably, this task’s historical interest results from its symbolic specification, which, despite being simple, resists classic statistical methods\.111E\.g\., it requires tracking structural information — grammatical subjecthood — over potentially unbounded distances, and can thus not be solved by simple Markov models\.A minimal circuit for handling it has to \(1\) recognize and store the grammatical number of the subject, \(2\) transport this information to the prediction position, and \(3\) license the corresponding output form\.

Figure 1:The subject–verb agreement circuit forpythia\-1bat step 2k \(left\) and step 143k \(right\)\. While agreement is acquired early \(within2%2\\%of training\) and the model remains proficient thereafter, the underlying circuit continues to reorganize and sparsify\. Circles represent the smallest subset of top\-ranked neurons whose sufficiency and necessity exceed 50% \(see[Section4\.2](https://arxiv.org/html/2610.33299#S4.SS2)\)\.In this work, we first \(Experiment 1\) trace the emergence of such a circuit over the course of language model training, studying ten seeds ofpythia\-410mand one seed ofpythia\-1b\. We find that agreement is learned quickly and is initially controlled by a diffuse circuit\. Over the course of training, however, we find that while the model’s performance remains mostly unchanged, the underlying circuit gradually evolves: it both sparsifies \(concentrating in a smaller set of neurons\) and is rerouted within the model \(being implemented by different attention heads\)\. By the end of training, a surprisingly sparse circuit captures most of the agreement signal inpythia\-1b\([Figure1](https://arxiv.org/html/2610.33299#S1.F1)\)\.

Such a gradual transition is consistent with recent reports of emergent syntactic behavior\([Chen et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib8)\), and with work in developmental LM interpretability more broadly\([Wang et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib51);[Chang et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib7);[Fehlauer et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib13);[Kangaslahti et al\., 2026](https://arxiv.org/html/2610.33299#bib.bib28)\), including recent work tracking the emergence of circuits longitudinally in open\-checkpoint models\([Tigges et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib48);[Ou et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib42);[Hakimi et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib22);[Aljaafari et al\., 2026](https://arxiv.org/html/2610.33299#bib.bib2)\)\. This circuit’s trajectory also parallels accounts of grokking\([Power et al\., 2022](https://arxiv.org/html/2610.33299#bib.bib44);[Nanda et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib40);[Varma et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib50);[Merrill et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib37)\), where a task is first learned by memorization, but that solution is replaced by a sparser and more general circuit after training accuracy has saturated\.

Later in our paper \(Experiment 2\), we zoom in onpythia\-1b\. We first track how the circuit evolves in this model, tracing how a subset of neurons remains relevant for agreement throughout training, but changes “allegiance” at some point, shifting the attention head they write to\. Second, we further sparsify this model’s final circuit while preserving its performance\. We find that just 18 neurons and a single attention head can capture the primary channel for agreement in this model\. We show this sparse circuit generalizes across different forms of subject–verb number agreement and to in\-the\-wild examples from WikiText\([Merity et al\., 2017](https://arxiv.org/html/2610.33299#bib.bib36)\)\. Finally, we demonstrate that this circuit’s information flows through a one\-dimensional channel of the relevant transfer head\. Overall, we show that mechanistic reorganization and sparsification continue beneath a stable agreement behavior\. In the case ofpythia\-1b, this culminates in a sparse and generalizable circuit for number agreement\.222Throughout this paper, we focus on subject–verb number agreement, but often term it more compactly as ‘subject–verb agreement’ or simply ‘agreement’\.

## 2Finding agreement in LMs

How can we understand and characterize the mechanisms responsible for a behavior in LMs? The field of \(mechanistic\) interpretability focuses on answering this question\. Historically, researchers relied on diagnostic probes to find signals of the task within LMs’ representations\([Alain and Bengio, 2016](https://arxiv.org/html/2610.33299#bib.bib1);[Hewitt and Manning, 2019](https://arxiv.org/html/2610.33299#bib.bib26);[Hewitt and Liang, 2019](https://arxiv.org/html/2610.33299#bib.bib25);[Pimentel et al\., 2020](https://arxiv.org/html/2610.33299#bib.bib43);[Kulmizev et al\., 2020](https://arxiv.org/html/2610.33299#bib.bib30),i\.a\.\)\. In the years since, though, the field has grown into a substantial literature, with causality becoming one of its central aspects\([Olah et al\., 2020](https://arxiv.org/html/2610.33299#bib.bib41);[Elhage et al\., 2021](https://arxiv.org/html/2610.33299#bib.bib12);[Elazar et al\., 2021](https://arxiv.org/html/2610.33299#bib.bib11)\), and circuit\-discovery a major thread within it\([Wang et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib52);[Hanna et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib23);[Conmy et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib10);[Stolfo et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib45);[Hanna et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib24),i\.a\.\)\.

Notably, some of the earliest work on this topic tried to identify the mechanisms behind number agreement in LMs\. Studying LSTMs,[Giulianelli et al\. \(2018\)](https://arxiv.org/html/2610.33299#bib.bib19)and[Lakretz et al\. \(2019\)](https://arxiv.org/html/2610.33299#bib.bib31)identified components or circuits in these models which implemented number agreement\. Studying transformer models,[Clark et al\. \(2019\)](https://arxiv.org/html/2610.33299#bib.bib9)identified syntax\-aware attention heads in BERT, and[Ferrando and Costa\-jussà \(2024\)](https://arxiv.org/html/2610.33299#bib.bib14)identified a bilingual number\-transport attention head for English and Spanish ingemma\-2b\. More recently, there has been a shift towards trying to identify more fine\-grained circuits within LMs\.[Marks et al\. \(2025\)](https://arxiv.org/html/2610.33299#bib.bib34), for instance, used sparse autoencoders \(SAEs\) to identify sparse circuits \(with fewer than 250 nodes\) responsible for subject–verb agreement in bothpythia\-70mandgemma\-2\-2b\. Building on such work,[Arora et al\. \(2026\)](https://arxiv.org/html/2610.33299#bib.bib4)showed that circuits over MLP activations can be as sparse as SAE\-based ones\.

Most relevantly,[Tigges et al\. \(2024\)](https://arxiv.org/html/2610.33299#bib.bib48)trace Pythia circuits across four tasks, one of which is subject–verb agreement\. They find that the underlying algorithm implemented by the circuit remains stable across training, but that the circuit’s components and size can change\. While they focus on a variety of tasks and localize their circuit at a coarse level of MLP blocks and attention heads, we zoom in on a single task, studying its behavior across many random seeds and focusing our analyses at the neuron level\. We track how many and which neurons are needed to causally control agreement across training, we characterize how they reroute between number\-transport heads, and we describe a complete subject\-to\-prediction circuit in the model\.

Overall, our work provides three primary contributions\. First, building on prior longitudinal circuit analyses across Pythia checkpoints and scales\([Tigges et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib48), e\.g\.,\), we study subject–verb agreement at finer granularity, combining head\- and neuron\-level interventions to track the agreement circuit across checkpoints and random seeds\. Second, we characterize not only which components change in this circuit, but how agreement\-relevant neurons sparsify and reroute as the dominant transport head changes\. Finally, we show that most of the agreement behavior inpythia\-1bcan be captured by a sparse circuit\.

## 3Methods and materials

### 3\.1Models

We study models of thepythiafamily\([Biderman et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib5)\)\. We focus our analysis onpythia\-1b— a 16\-layer autoregressive transformer with88attention heads per layer, a hidden size of 2048 neurons, and MLPs with intermediate dimension81928192— and on the ten seeds ofpythia\-410m, nine of which are from PolyPythias\([van der Wal et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib49)\), each with 24 layers, 16 attention heads per layer, hidden size 1024, and MLP dimension40964096\. By performing our analysis across model sizes and seeds, we study the behavior and internal representations of the models more generally\.

### 3\.2Subject–verb agreement task

We study subject–verb agreement, focusing on sentences with an intervening prepositional phrase, a task we refer to asPPfor convenience\. An exemplar minimal pair can be seen in \(1\)\.

We denote token roles with subscripts:pos1\\text\{pos\}\_\{1\}for the subject position,pos2\\text\{pos\}\_\{2\}for the last token position of the preposition, andpos4\\text\{pos\}\_\{4\}for the final token/prediction position\. In order to generate a large number of unique minimal pairs, we adapt the templates from[Arora et al\. \(2024\)](https://arxiv.org/html/2610.33299#bib.bib3)\. Specifically, we restrict the lexical inventory to single\-token nouns and verbs \(in both singular and plural forms\) and only generate minimal pairs differing in subject number\.

In[Section5](https://arxiv.org/html/2610.33299#S5), we evaluate how the circuits we discover generalize to other agreement examples\. Beyond PP sentences, in that section, we also investigate sentences with a subject relative clause distractor \(SubjectRC\), an object relative clause distractor \(ObjectRC\), or no distractor \(Simple\)\.

For thePPtask, we create a training set of 400 minimal pairs, balanced in the number of subjects and distractors\. We use this set to identify candidate circuit components\. We also create disjoint evaluation sets forPPand the other templates, each consisting of 192 balanced minimal pairs \(with the exception ofSimple, which has 32 minimal pairs\)\. For both behavioral validation \([Section3\.3](https://arxiv.org/html/2610.33299#S3.SS3)\) and circuit evaluation \([Section3\.4](https://arxiv.org/html/2610.33299#S3.SS4)\), we use the evaluation sets\. For more details on all templates, along with associated exemplars, see App\.[C](https://arxiv.org/html/2610.33299#A3)\.

### 3\.3Behavioral validation

To measure the behavioral performance of the studied models on subject–verb agreement, we calculate their accuracy on the evaluation set using the following paradigm\. A model is deemed to correctly process a minimal pair if for each item, the probability for the correct continuation is higher than the alternative\. That is, given the minimal\-pair sentences in \(1\), a model is accurate if it correctly predicts for*both*sentences that

p⁡\(is\|The key …\)\>p⁡\(are\|The key …\)​and\\displaystyle p\(\\text\{is\}\|\\text\{The key \.\.\.\}\)\>p\(\\text\{are\}\|\\text\{The key \.\.\.\}\)\\,\\texttt\{ and \}\(1a\)p⁡\(is\|The keys …\)<p⁡\(are\|The keys …\)\.\\displaystyle p\(\\text\{is\}\|\\text\{The keys \.\.\.\}\)<p\(\\text\{are\}\|\\text\{The keys \.\.\.\}\)\.\(1b\)

### 3\.4Circuit evaluation

Let𝒞\\mathcal\{C\}be a model component, or a collection of components, proposed to be a part of a circuit of interest and thus responsible for a certain task\. To evaluate the causal effect of𝒞\\mathcal\{C\}on that task, we intervene on its activations to calculate itsnecessityandsufficiencyfor downstream behavioral control\. This section details these metrics\.

Both of these metrics rely on a metric we refer to as theagreement gap,GG\. Let𝐛\\mathbf\{b\}and𝐬\\mathbf\{s\}be base and source minimal\-pair sentences, withℓ𝐛\\ell\_\{\\mathbf\{b\}\}andℓ𝐬\\ell\_\{\\mathbf\{s\}\}their respective continuations\. In ex\. \(1\), for instance,ℓ𝐛=is\\ell\_\{\\mathbf\{b\}\}=\\textit\{is\}andℓ𝐬=are\\ell\_\{\\mathbf\{s\}\}=\\textit\{are\}\. Given this pair,GGis the expected difference in log probability between the base and source continuations:

G𝐛\\displaystyle G^\{\\mathbf\{b\}\}=𝔼\(𝐛,𝐬\)∈D​log⁡\(p𝜽​\(ℓ𝐛\|𝐛\)p𝜽​\(ℓ𝐬\|𝐛\)\),\\displaystyle=\\underset\{\(\\mathbf\{b\},\\mathbf\{s\}\)\\in D\}\{\\mathbb\{E\}\}\\log\\left\(\\frac\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{b\}\}\|\\mathbf\{b\}\)\}\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{s\}\}\|\\mathbf\{b\}\)\}\\right\),\(2\)G𝐬\\displaystyle G^\{\\mathbf\{s\}\}=𝔼\(𝐛,𝐬\)∈D​log⁡\(p𝜽​\(ℓ𝐛\|𝐬\)p𝜽​\(ℓ𝐬\|𝐬\)\)\\displaystyle=\\underset\{\(\\mathbf\{b\},\\mathbf\{s\}\)\\in D\}\{\\mathbb\{E\}\}\\log\\left\(\\frac\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{b\}\}\|\\mathbf\{s\}\)\}\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{s\}\}\|\\mathbf\{s\}\)\}\\right\)wherep𝜽p\_\{\\boldsymbol\{\\theta\}\}stands for the evaluated model run under a specific configuration\.G𝐛G^\{\\mathbf\{b\}\}andG𝐬G^\{\\mathbf\{s\}\}therefore represent agreement gaps of the model when processing the base sentence or source sentence, respectively\. A higher preference for the base continuation corresponds to a more positive gap score\. For a behaviorally correct model,G𝐛G^\{\\mathbf\{b\}\}should be positive, whileG𝐬G^\{\\mathbf\{s\}\}should be negative\. Our notions of sufficiency and necessity then measure how much this gap changes under manipulations of activations\.

A model component𝒞\\mathcal\{C\}isnecessaryif performing a mean ablation\([Wang et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib52)\)causes the model’s agreement gap to drop to near\-zero:

Nec⁡\(𝒞\)=G𝐛−G𝒞←avgG𝐛\\operatorname\{Nec\}\(\\mathcal\{C\}\)=\\frac\{G^\{\\mathbf\{b\}\}\-G^\{\\mathcal\{C\}\\leftarrow\\texttt\{avg\}\}\}\{G^\{\\mathbf\{b\}\}\}\(3\)whereG𝒞←avgG^\{\\mathcal\{C\}\\leftarrow\\texttt\{avg\}\}is the agreement gap on input𝐛\\mathbf\{b\}when we replace𝒞\\mathcal\{C\}’s activations with their average value over a training set\.333Mean activations are computed on training sets for thePPtemplate and on the evaluation set for other templates\.

A model component𝒞\\mathcal\{C\}issufficientif exchanging its activations across minimal pairs𝐛←𝐬\\mathbf\{b\}\\leftarrow\\mathbf\{s\}shifts the verb predictions for𝐛\\mathbf\{b\}towards those of𝐬\\mathbf\{s\}\. This is measured by the following metric:

Suff⁡\(𝒞\)=G𝐛−G𝒞←𝐬G𝐛−G𝐬,\\operatorname\{Suff\}\(\\mathcal\{C\}\)=\\frac\{G^\{\\mathbf\{b\}\}\-G^\{\\mathcal\{C\}\\leftarrow\\mathbf\{s\}\}\}\{G^\{\\mathbf\{b\}\}\-G^\{\\mathbf\{s\}\}\},\(4\)whereG𝒞←𝐬G^\{\\mathcal\{C\}\\leftarrow\\mathbf\{s\}\}is the agreement gap when the model is run with input𝐛\\mathbf\{b\}with𝒞\\mathcal\{C\}’s activations replaced by their values under input𝐬\\mathbf\{s\}\. Intuitively, a component is sufficient if intervening on𝒞\\mathcal\{C\}’s activations has roughly the same causal effect as directly swapping the subject token for its singular/plural counterpart\.

#### Relation to other metrics\.

Existing circuit work evaluatescompletenessandfaithfulness\([Wang et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib52);[Marks et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib34);[Arora et al\., 2026](https://arxiv.org/html/2610.33299#bib.bib4)\)\. Within a predefined scope \(global set of relevant components\), a circuit iscompleteif ablating the circuit degrades performance as much as ablating the entire scope, andfaithfulif ablating its complement preserves performance\. Our notion of necessity is conceptually similar to completeness, as ablating the entire scope should typically bring the model close to chance performance — and thus near zero agreement gap\. We choose to evaluate sufficiency rather than faithfulness as defining a circuit’s complement \(to ablate\) is difficult, especially when evaluating single components\.

### 3\.5Circuit discovery methods

This section details the methods we use to localize our circuit\. Throughout our experiments, we discover and evaluate our circuits on thePPtemplate; we later show that the discovered circuit generalizes to the other forms of number agreement\.

#### Head identification\.

We identify attention heads responsible for transporting a subject’s number by measuring each head’s necessity and sufficiency at the prediction position,pos4\\text\{pos\}\_\{4\}, across the training set\. We intervene on the full head to compute these metrics, not only on its subject\-\-prediction values\.444After a head is selected as a circuit component, it is then evaluated for necessity and sufficiency on the evaluation sets\.

#### Neuron identification\.

We follow[Arora et al\. \(2026\)](https://arxiv.org/html/2610.33299#bib.bib4)in using Relevance Patching\([Jafari et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib27), RelP;\), a gradient\-attribution method, to localize neurons responsible for LM behavior\. We utilize the unpaired variant of RelP which scores components by the product of their activation on the base input and the gradient of a task metric with respect to that component\. It differs from attribution patching methods\([Nanda, 2023](https://arxiv.org/html/2610.33299#bib.bib39);[Kramár et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib29);[Syed et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib47)\)by computing the gradient on a locally linear replacement model — where in the backward pass each nonlinearity is frozen at its forward value\. This results in a less noisy and more faithful gradient\([Jafari et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib27)\)\. Details of our RelP implementation, including the linearization rules, an integrated\-gradient robustness check, and task\-specific modifications, are in App\.[D\.1](https://arxiv.org/html/2610.33299#A4.SS1)\.

We score each MLP neuron,𝒞n\\mathcal\{C\}\_\{n\}, as in \([5](https://arxiv.org/html/2610.33299#S3.E5)\), wherevn​\(𝐛\)v\_\{n\}\(\\mathbf\{b\}\)is𝒞n\\mathcal\{C\}\_\{n\}’s activation on the base input,mmis the per\-example base agreement gap, and the gradient is taken relative to the replacement model:

RelP⁡\(𝒞n\)=𝔼\(𝐛,𝐬\)∈D​\[vn​\(𝐛\)​∂m⁡\(𝐛,𝐬\)∂vn​\(𝐛\)\],\\displaystyle\\mathrm\{RelP\}\(\\mathcal\{C\}\_\{n\}\)=\\underset\{\(\\mathbf\{b\},\\mathbf\{s\}\)\\in D\}\{\\mathbb\{E\}\}\\\!\\left\[\\,v\_\{n\}\(\\mathbf\{b\}\)\\,\\frac\{\\partial m\(\\mathbf\{b\},\\mathbf\{s\}\)\}\{\\partial v\_\{n\}\(\\mathbf\{b\}\)\}\\,\\right\],\(5\)where​m​\(𝐛,𝐬\)=log⁡p𝜽​\(ℓ𝐛∣𝐛\)p𝜽​\(ℓ𝐬∣𝐛\)\.\\displaystyle\\qquad\\qquad\\texttt\{where \}m\(\\mathbf\{b\},\\mathbf\{s\}\)=\\log\\frac\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{b\}\}\\mid\\mathbf\{b\}\)\}\{p\_\{\\boldsymbol\{\\theta\}\}\(\\ell\_\{\\mathbf\{s\}\}\\mid\\mathbf\{b\}\)\}\.A positive RelP score indicates that𝒞n\\mathcal\{C\}\_\{n\}pushes the model toward verbs licensed by their subject\. We apply this method to score every post\-GELU MLP neuron in each layer at the relevant token positions for a given analysis\. As RelP is an attribution method, we treat it as a hypothesis generator for causally\-relevant neurons\. To ensure their importance, we then causally validate such candidate neurons through interventions detailed in[Section3\.4](https://arxiv.org/html/2610.33299#S3.SS4)\.

#### Neuron and head routing\.

Having separately identified neurons and heads relevant to agreement, we want to establish routing relations between them\. For instance, given two heads that could both transport number information, we want to determine which route a candidate neuron takes\. To this end, we attribute each neuron to heads \(possibly to both or neither\) using a path\-patching\-like method\([Wang et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib52);[Goldowsky\-Dill et al\., 2023](https://arxiv.org/html/2610.33299#bib.bib20)\)\. For neurons before the prediction position, we \(1\) intervene on the neuron and measure the downstream causal effect and \(2\) intervene on the same neuron while holding a head𝒞h\\mathcal\{C\}\_\{h\}’s corresponding value edge fixed at the base activation\. Any effect that drops out under the second intervention relative to the first is attributed to the route via𝒞h\\mathcal\{C\}\_\{h\}\. Prediction\-side neurons are likewise classified by which head interventions they read out from\. We describe our methodology in detail in App\.[D\.2](https://arxiv.org/html/2610.33299#A4.SS2)\.

## 4Experiment 1: Behavior–mechanism decoupling in training

In our first experiment, we track the internal organization ofpythia\-1band the ten seeds ofpythia\-410mas they develop the capacity to perform subject–verb agreement\. At each checkpoint, we measure their agreement capabilities, identify the neurons associated with agreement using RelP, and determine which heads are sufficient and necessary for agreement\. Our goal here is not yet to produce complete agreement circuits, but to provide a description of the models’ behavior and identify agreement\-relevant components through training\.

### 4\.1Agreement behavior is learned within 2% of total training steps

First, we measure the behavioral performance of our studied models across training\. The y\-axis in[Figure2](https://arxiv.org/html/2610.33299#S4.F2)shows our results\. At steps 256 and 512, the models have not learned agreement yet, always preferring singular continuations over plural ones, regardless of subject number\. Since agreement accuracy requires predicting both singular and plural verbs correctly, through step 512, no model is even2%2\\%accurate on agreement\. By step 2k, though, agreement is high for all tenpythia\-410mseeds, with a median accuracy of96\.9%96\.9\\%\(range88\.588\.5–100%100\\%\), and a median agreement gap of3\.353\.35\. At the end of training, the models’ median accuracy and gap are91\.7%91\.7\\%and3\.393\.39, respectively\. This shows agreement capabilities are learned early in training and remain relatively stable throughout\. The largerpythia\-1bshows the same pattern — its accuracy and gap change from99\.0%99\.0\\%and3\.213\.21at step 2k to94\.8%94\.8\\%and3\.483\.48at the final checkpoint\.555The better performance at step 2k compared to the final checkpoint is somewhat mysterious\. The final checkpoint errors are cases with opposite\-number distractors, suggesting that models are perhaps falling for agreement attraction effects\. Broadly, though, this result is consistent with the “unruly” BLiMP curves shown in[Bunzeck and Zarrieß \(2024\)](https://arxiv.org/html/2610.33299#bib.bib6)\.

Overall, our results are consistent with[van der Wal et al\. \(2025\)](https://arxiv.org/html/2610.33299#bib.bib49)’s finding that linguistic knowledge is largely acquired by103−10410^\{3\}\-10^\{4\}steps\. This raises a question, though: once agreement behavior is saturated, is the mechanism behind this syntactic competence effectively learned and frozen in place, or does it continue to evolve in ways that behavioral measurements do not reveal? To answer this question, we first track the neurons that process subject number at the subject, and then the attention heads that transport it to the prediction position\.

Takeaway\. The models learn agreement after only around 2k training steps, with behavior remaining generally consistent thereafter\.

### 4\.2Circuit neurons sparsify through training

We now consider the neurons implicated in the agreement circuits at the subject position, and compare how they change across training checkpoints\. In particular, if the agreement circuit is effectively “finalized” after proficiency is achieved in this task, the number of circuit neurons should remain stable\.

For each model and checkpoint, we identify the neurons most responsible for agreement by \(1\) computing a RelP score for all MLP neurons across all layers at the subject position on the training set, \(2\) ranking them in descending order, since a larger positive RelP score indicates stronger support for agreement, and \(3\) evaluating their causal effect on the model\. For \(3\), we definek50k\_\{50\}as the smallestkksuch that a circuit consisting of the top\-kkranked neurons,𝒞k\\mathcal\{C\}\_\{k\}, is at least50%50\\%necessary and sufficient under intervention:666Finding the precise value ofkkis prohibitively expensive\. In practice, we compute the score in \([6](https://arxiv.org/html/2610.33299#S4.E6)\) at intervals of powers of two, and interpolate linearly inlog2\\log\_\{2\}space for an estimate\.

k50=min⁡\{k∣min⁡\(Nec⁡\(𝒞k\),Suff⁡\(𝒞k\)\)≥50%\}\.\\displaystyle k\_\{50\}=\\min\\left\\\{k\\mid\\min\(\\operatorname\{Nec\}\(\\mathcal\{C\}\_\{k\}\),\\operatorname\{Suff\}\(\\mathcal\{C\}\_\{k\}\)\)\\geq 50\\%\\right\\\}\.

\(6\)
Figure 2:Agreement accuracy vs\. \# of subject\-neurons in the circuit through training across our models\. Each point is a checkpoint\. Typically, steps 1k–2k form the agreement acquisition phase\. In the remaining steps, the circuit sparsifies before reaching some equilibrium\.Our results are in[Figure2](https://arxiv.org/html/2610.33299#S4.F2)\. We find that the agreement circuit tends to sparsify throughout early training — even as performance remains constant — leveling off around halfway through training\. Further, the same neurons are generally deemed important across checkpoints\. For example, inpythia\-1b, we find that all 21 neurons in step 18k’sk50k\_\{50\}are present in the larger 42\-neuronk50k\_\{50\}at step 8k, with 20 also appearing in the 41\-neuronk50k\_\{50\}at step 50k, and 18 remaining in the final checkpoint’sk50k\_\{50\}\. The same pattern holds across most runs ofpythia\-410m: taking the intersection of thek50k\_\{50\}neuron sets of all post\-8k checkpoints, we find a median of85%85\\%overlap with each seed’s smallestk50k\_\{50\}neuron set\. For more details on neuron overlap across checkpoints and on whether results hold across thresholds ofkk, see App\.[E](https://arxiv.org/html/2610.33299#A5)\.

Notably, we observe a substantial increase ink50k\_\{50\}during training for twopythia\-410mseeds \(3 and 4\)\. Interestingly, these seeds are exactly the training outliers identified by[van der Wal et al\. \(2025\)](https://arxiv.org/html/2610.33299#bib.bib49), which underperformed on benchmarks after suffering loss spikes during training\. These loss spikes occur precisely where we see sharp rebounds in individualk50k\_\{50\}counts \(at steps 83k and 123k for seeds 3 and 4, respectively; more details in App\.[E\.3](https://arxiv.org/html/2610.33299#A5.SS3)\)\.

This analysis tells us that agreement circuits become sparser at the subject position during training, involving fewer neurons\. But the complete circuit also involves transporting that signal to the prediction position\. Next, we ask whether the number\-transporting heads remain stable after agreement is learned, or if they also undergo reorganization\.

Takeaway\. After models learn agreement, their circuits still sparsify, with progressively fewer neurons being required to capture the same level of causal control over agreement across training\.

### 4\.3Number is carried by a small set of heads

We now identify the attention heads in the agreement circuits at the prediction position by similarly measuring necessity and sufficiency across training\.[Figure3\(a\)](https://arxiv.org/html/2610.33299#S4.F3.sf1)demonstrates that inpythia\-1b, early agreement circuits are almost entirely mediated by two heads \(L15 H7andL15 H2\)\. However, by step 18k, headL13 H7becomes the most important, and remains so through the final checkpoint \(althoughL15 H2persists as a weaker secondary channel\)\. In comparison, all other heads contribute negligibly\.

[Figure3\(b\)](https://arxiv.org/html/2610.33299#S4.F3.sf2)tells a similar story forpythia\-410m\. Across all models, a small set of heads is responsible for carrying agreement information, often with a clear leading head individually achieving over50%50\\%in necessity and sufficiency\. Despite consistently small numbers of relevant heads, none of the examined models have the same top\-3 heads at steps 2k and 143k\. \(Detailed results in App\.[E\.4](https://arxiv.org/html/2610.33299#A5.SS4)\.\)

\(a\)Head scores across training forpythia\-1b\.\(b\)Top\-five heads across model runs\.
Figure 3:Attention\-head reorganization\.Top:Inpythia\-1b, attention heads reorganize across training checkpoints, with the heads important early,L15 H7andL15 H2, ceding importance toL13 H7\. All other heads are negligible\.Bottom:The identities of the leading heads change across training\. Circle area is proportional to each head’s individualmin⁡\{Nec,Suff\}\\min\\\{\\operatorname\{Nec\},\\operatorname\{Suff\}\\\}score, with each individual head a different color\.Takeaway\. We find that agreement is generally mediated through a small set of late\-layer attention heads, and the leading transport heads often shift even after behavior stabilizes\.

Figure 4:Top\-32 ranked RelP neurons across positions and attention headsL13 H7andL15 H7across checkpoints as identified onPPtraining set \(N=400N=400\)\. Neurons are colored by their correspondence to the respective heads, on a gradient between blue \(L15 H7\) and red \(L13 H7\), determined by path patching\. Gray neurons are not attributed to either route\. Line width indicates each head’s sufficiency\. We see that throughout training, there is a gradual transition from anL15 H7\-routed mechanism to anL13 H7\-routed one\.

## 5Experiment 2: The sparsification of thepythia\-1bcircuit

Having shown the general developmental trajectory of agreement circuits across seeds and models, we now zoom in on the final circuit for one model:pythia\-1b\. First, we show that its neuron\- and head\-level reorganizations happen in tandem\. Next, we prune the circuit down to 18 neurons and one attention head\. We then demonstrate this hand\-constructed sparse circuit generalizes to novel templates and in\-the\-wild text\. Finally, we show the head’s transport can be further localized to just a one\-dimensional subspace of its output channel\.

### 5\.1Joint reorganization

In[Section4](https://arxiv.org/html/2610.33299#S4), we showed that while agreement is learned early, its circuit reorganizes during training\. Two changes occur together, namely that the number of subject\-neurons in the circuit decreases, and the heads transporting the number information shift from a L15 route to aL13 H7route\. In this section, we link these observations and show that circuit neurons change the pathway they feed into or read out from as the dominant head changes\.

To this end, we first collect the top\-32 ranked neurons according to RelP at each checkpoint across three token positions: subjectpos1\\text\{pos\}\_\{1\}, prepositionpos2\\text\{pos\}\_\{2\}, and predictionpos4\\text\{pos\}\_\{4\}\. We then apply the routing attribution method from[Section3\.5](https://arxiv.org/html/2610.33299#S3.SS5)to measure the extent to which a neuron reads from or writes toL15 H7\(the head primarily responsible for information transport early in training\) orL13 H7\(the head responsible late in training\)\.

[Figure4](https://arxiv.org/html/2610.33299#S4.F4)shows our results across checkpoints\. We see that as the relative head importance changes, the neurons follow\. That is, early in training, we see neurons route throughL15 H7, the early primary transport channel\. However, as training progresses andL13 H7’s channel emerges, pre\-L13 neurons begin to show mixed routing\. Then, asL13 H7becomes increasingly prominent, most neurons reorganize to start writing to or reading from that channel\. By the final checkpoint, both upstream and downstream neurons are primarily aligned with theL13 H7channel\.

Takeaway\. The agreement circuit inpythia\-1bshifts from a route mediated predominantly byL15 H7to one mediated byL13 H7throughout training\. As the heads shift, the neurons follow\.

### 5\.2A final sparse circuit

Having shown the trajectory of the joint neuron and head circuit through training, we now zoom in on the final checkpoint ofpythia\-1bto produce a sparse circuit for agreement\.

#### Circuit construction\.

Our goal here is to construct a sparse circuit for subject–verb agreement\. To do so, we made a series of choices that we believe would represent the model’s internal agreement mechanism while minimizing the circuit size\.

We started with two qualitative observations based on the results of the earlier analyses and further manual inspection of the results\. First, number agreement at the final checkpoint is largely carried by one head,L13 H7\. Second, MLPs propagate agreement information across layers, while self\-attention is not strongly involved in this process\.777We show in App\.[F\.2](https://arxiv.org/html/2610.33299#A6.SS2)that ablating self\-attention has negligible effect on agreement\.

Our goal was then to construct a circuit consisting of the one relevant head and a set of relevant neurons that captures the role of the MLPs in processing agreement\. To this end, we again used RelP to identify a set of potentially important neurons for the circuit\. We further augmented this set with a few extra hand\-picked neurons, identified as important via other methodologies\.888We performed an initial analysis, prior to applying RelP, using Distributed Alignment Search\([Geiger et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib18), DAS;\), a supervised interpretability method which localizes LM behavior to causally sufficient subspaces in the higher\-dimensional activation space\. Consistent with previous works\([Arora et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib3), e\.g\.\), we found that the number feature could be localized to a single dimension of the activation space, but this subspace appeared to change across layers due to MLPs writing additional number information into the residual stream\. This motivated us to identify important neurons which write to these DAS subspaces\. For each neuron, we computed its number\-dependent contribution to the next\-layer as the product of its paired activation difference between singular and plural inputs and the neuron’s output vector\. The cosine similarity of the resulting contribution with the next\-layer DAS subspace was then used to rank neurons\. DAS settings and the neuron selection and pruning procedures are in App\.[F\.1](https://arxiv.org/html/2610.33299#A6.SS1)\.This resulted in a small set of 21 candidate neurons\. We then greedily pruned neurons whose removal had negligible effects on circuit necessity and sufficiency\. This procedure removed three redundant neurons, leaving us with our final circuit\.

Notably, this circuit’s construction process involved several judgment calls\. Our goal was not to provide a systematic solution for its creation, but to try to qualitatively understand and improve the circuit iteratively\. There are no guarantees that this circuit is maximally performant or sparse\. At the end of the procedure, we had a sparse circuit,𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}, which contains 18 neurons and one attention head\. Ten neurons are at the subject positionpos1\\text\{pos\}\_\{1\}and upstream of the headL13 H7\(in L1–L12 MLPs\); eight neurons are at the prediction positionpos4\\text\{pos\}\_\{4\}and downstream of the head \(in L14–L15 MLPs\)\.

Table 1:Necessity and sufficiency of the full circuit,𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}, relative to the scope,𝒞scope\\mathcal\{C\}\_\{\\text\{scope\}\}, tested on an evaluation set of 192 PP items\. The circuit is nearly as necessary and sufficient as the full scope, despite containing around 1/6000 as many MLP neurons\.Importantly, we do not see the circuit as the*complete*agreement mechanism forpythia\-1b—[Figure3\(a\)](https://arxiv.org/html/2610.33299#S4.F3.sf1)and[Figure4](https://arxiv.org/html/2610.33299#S4.F4)show thatL15 H2and preposition neurons also carry agreement signal — nor is it constructed to maximize any metric\. We interpret the circuit to be a sparse representation of how the model transports the number feature from the subject to the prediction position via a specificL13 H7route\. The scope of the circuit𝒞scope\\mathcal\{C\}\_\{\\text\{scope\}\}thus consists of L1–L12 MLPs atpos1\\text\{pos\}\_\{1\},L13 H7, and L14–15 MLPs atpos4\\text\{pos\}\_\{4\}\.999We omit the subject L0 MLP from the scope as MLP ablation there is globally destructive\. We take this as evidence that the L0 MLP may be responsible for*constructing*the number feature with our circuit responsible for*transporting*the feature to readout\. Full details regarding scope selection and L0 MLP ablation results are in App\.[F\.2](https://arxiv.org/html/2610.33299#A6.SS2)\.

#### Circuit components\.

The circuit seems to carry out the minimal mechanism laid out in[Section1](https://arxiv.org/html/2610.33299#S1)\. \(1\)*Amplifier*neurons𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}at the subject process and propagate the number feature, \(2\) an attention head𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}transports this information to the prediction position, and \(3\) the head outputs and readout neurons𝒞𝙿\\mathcal\{C\}\_\{\\mathtt\{P\}\}and𝒞𝙽\\mathcal\{C\}\_\{\\mathtt\{N\}\}license the corresponding output form\.

Our characterization of the neurons is as follows: neurons in𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}have high activations on plural subjects, while their activations on singular inputs are comparable to those on non\-nouns — we thus think of𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}as*plural detectors*\. Readout neurons come in two flavors:*pro\-agreement*neurons𝒞𝙿\\mathcal\{C\}\_\{\\mathtt\{P\}\}promote the correct verb continuations, while*anti\-agreement*neurons𝒞𝙽\\mathcal\{C\}\_\{\\mathtt\{N\}\}seem to oppose agreement\. For the complete set of circuit components and more details on this characterization, see App\.[F\.3](https://arxiv.org/html/2610.33299#A6.SS3)\.

#### Evaluating the circuit\.

We now measure the necessity and sufficiency of our final circuit, providing the same for the entire scope as a baseline in[Table1](https://arxiv.org/html/2610.33299#S5.T1)\. We find our circuit is78%78\\%necessary, with its mean ablation dropping the agreement gap nearly as much as mean ablating the full scope\. We also find our circuit is83%83\\%sufficient, which is comparable to the88%88\\%sufficiency of the entire scope\. Further, the circuit’s complement has little effect on agreement \(App\.[F\.4](https://arxiv.org/html/2610.33299#A6.SS4)\)\.

Takeaway\. A sparse circuit of 18 neurons and a single attention head captures nearly all of the agreement information flow within the scope\.

### 5\.3Characterizing the final circuit

Having produced and validated a sparse circuit onPPtemplates, we now show that it mediates agreement more generally\. Then, we demonstrate that we can compress the sparse circuit even further, localizing the information flow within𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}to a one\-dimensional subspace\.

#### Generalization\.

First, we show our circuit is necessary and sufficient for agreement in further settings\. In particular, we measure our circuit on three synthetic templates with novel structure — agreement across a subject relative\-clause \(SubjectRC\), an object relative\-clause \(ObjectRC\), or no distractors \(Simple\) — as well as on mined natural examples from WikiText\([Merity et al\., 2017](https://arxiv.org/html/2610.33299#bib.bib36)\)for which we manually construct the minimal pair\. Templates and exemplars for each evaluation are in App\.[C](https://arxiv.org/html/2610.33299#A3)\.

Table 2:Circuit necessity and sufficiency across templates \(N=32N=32forSimple,N=192N=192for other templates, andN=80N=80for WikiText\)\. Mean activations are computed over the larger training set forPP, and directly on the evaluation set otherwise\.In[Table2](https://arxiv.org/html/2610.33299#S5.T2), we broadly see that our circuit’s necessity and sufficiency on these templates are comparable to those forPP\. The circuit performs slightly worse on theSimpletemplate, which likely reflects the structural difference of the stimuli\. In sum, these results suggest that our circuit is general, and not overfit to just thePPtemplates from which it was discovered\.

We also perform additional generalization tests, showing that circuit activations drawn from one template can control agreement in another, and that the circuit holds under most perturbations to the templates and on novel subjects and verbs\. These results, along with failure cases \(the circuit failing on templates with an appended adverb, andSimplebeing a weak donor for cross\-template intervention\) are found in App\.[F\.5](https://arxiv.org/html/2610.33299#A6.SS5)\.

#### 𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}can be localized to a 1D channel\.

We can further localize the head transport to a subspace ofL13 H7’s output\-value circuits\([Elhage et al\., 2021](https://arxiv.org/html/2610.33299#bib.bib12)\)using DAS\. To this end, we train three seeds of DAS on thePPtraining set to successfully localize behavior to a 1D subspace of the head’s value output\. \(See App\.[F\.6](https://arxiv.org/html/2610.33299#A6.SS6)for DAS training details\.\)

[Table3](https://arxiv.org/html/2610.33299#S5.T3)shows the results\. We find a 1D subspace which essentially captures the head’s total causal effect, and both intervening on and mean ablating the subspace complement barely affect agreement\. These results suggest that the head’s contribution to the agreement task can be surgically localized to a one\-dimensional channel\.

Takeaway\. Our circuit is general despite being discovered on thePPtemplate, mediating agreement robustly across synthetic and naturalistic sentences\. Further, the information is routed through a single dimension within the attention head\.

Table 3:The 1D channel identified by DAS has a comparable effect to the entire 256D𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}\. We report the mean necessity and sufficiency±\\pmstandard error across three independent seeds of DAS\.

## 6Conclusion

We believe that the circuit discovered makes good on the mechanistic account of number agreement that we started with\. Within the scoped L13 H7\-mediated pathway inpythia\-1b, ten subject\-position neurons propagate number information, L13 H7 carries it to the prediction position through a one\-dimensional causal channel, and eight downstream neurons control the verb readout\. Agreement circuits inpythia\-1band the tenpythia\-410mseeds were diffuse early in training but generally became sparser over time\.

We leave it to future work to analyze how and why this transition occurs, although we hypothesize the following: the early circuit emerges stochastically out of a set of features that proved useful for doing the subject–verb agreement task\. Then, this diffuse circuit serves as a scaffold for a sparser and more specialized circuit to emerge\. Our hope is that, by granularly studying a single syntactic phenomenon, we can complement wider investigations into LMs’ syntactic abilities and thereby better characterize linguistic knowledge in LMs\([Futrell and Mahowald, 2026](https://arxiv.org/html/2610.33299#bib.bib16), as called for by\)\.

## Limitations

The goal of this paper was to focus on just one linguistic task of subject–verb agreement, but this is also, of course, a limitation\. We also focus only on English\. Future work could generalize these methods to other constructions, models, and languages\. For present purposes, we felt that a wider investigation would have undermined the particular motivation of this project\.

Further, we find neuron sparsification and head rerouting to be common across model training runs, but have not provided an explanation for it\. Another limitation is that the method for identifying the circuit in Experiment 2 was iterative and involved manual analyses\. We were able to proceed in this way because of the highly focused nature of the work\. But more work would be needed to develop a general procedure for applying it to other tasks and settings\.

## Acknowledgments

We acknowledge funding from NSF CAREER grant 2339729 to Kyle Mahowald and funding from the Harrington Fellowship to Qing Yao\. The work of Tiago Pimentel was funded by the Data Analytics Lab at ETH Zürich\. We thank Jed Guzelkabaagac, Felix Sarnthein, Rodrigo Casado, Souvik Banerjee, Jan Tempus, and Philip Whittington for comments on this manuscript\.

## References

- Alain and Bengio \(2016\)Guillaume Alain and Yoshua Bengio\. 2016\.[Understanding intermediate layers using linear classifier probes](https://arxiv.org/abs/1610.01644)\.*Preprint*, arXiv:1610\.01644\.
- Aljaafari et al\. \(2026\)Nura Aljaafari, Danilo Carvalho, and Andre Freitas\. 2026\.[Emergence and localisation of semantic role circuits in LLMs](https://doi.org/10.18653/v1/2026.findings-acl.1964)\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 39402–39433, San Diego, California, United States\. Association for Computational Linguistics\.
- Arora et al\. \(2024\)Aryaman Arora, Dan Jurafsky, and Christopher Potts\. 2024\.[CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://doi.org/10.18653/v1/2024.acl-long.785)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 14638–14663, Bangkok, Thailand\. Association for Computational Linguistics\.
- Arora et al\. \(2026\)Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, and Sarah Schwettmann\. 2026\.[Language model circuits are sparse in the neuron basis](https://openreview.net/forum?id=OrviwFWcN4)\.In*Forty\-third International Conference on Machine Learning*\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\. 2023\.[Pythia: A suite for analyzing large language models across training and scaling](https://proceedings.mlr.press/v202/biderman23a.html)\.In*International Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA*, volume 202 of*Proceedings of Machine Learning Research*, pages 2397–2430\. PMLR\.
- Bunzeck and Zarrieß \(2024\)Bastian Bunzeck and Sina Zarrieß\. 2024\.[Fifty shapes of BLiMP: syntactic learning curves in language models are not uniform, but sometimes unruly](https://aclanthology.org/2024.clasp-1.7/)\.In*Proceedings of the 2024 CLASP Conference on Multimodality and Interaction in Language Learning*, pages 39–55, Gothenburg, Sweden\. Association for Computational Linguistics\.
- Chang et al\. \(2024\)Tyler A\. Chang, Zhuowen Tu, and Benjamin K\. Bergen\. 2024\.[Characterizing learning curves during language model pre\-training: Learning, forgetting, and stability](https://doi.org/10.1162/tacl_a_00708)\.*Transactions of the Association for Computational Linguistics*, 12:1346–1362\.
- Chen et al\. \(2024\)Angelica Chen, Ravid Shwartz\-Ziv, Kyunghyun Cho, Matthew L\. Leavitt, and Naomi Saphra\. 2024\.[Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs](https://openreview.net/forum?id=MO5PiKHELW)\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\.
- Clark et al\. \(2019\)Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D\. Manning\. 2019\.[What Does BERT Look at? An Analysis of BERT’s Attention](https://doi.org/10.18653/v1/W19-4828)\.In*Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, pages 276–286, Florence, Italy\. Association for Computational Linguistics\.
- Conmy et al\. \(2023\)Arthur Conmy, Augustine Mavor\-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga\-Alonso\. 2023\.[Towards automated circuit discovery for mechanistic interpretability](https://openreview.net/forum?id=89ia77nZ8u)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 16318–16352\.
- Elazar et al\. \(2021\)Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg\. 2021\.[Amnesic probing: Behavioral explanation with amnesic counterfactuals](https://doi.org/10.1162/tacl_a_00359)\.*Transactions of the Association for Computational Linguistics*, 9:160–175\.
- Elhage et al\. \(2021\)Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield\-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others\. 2021\.[A mathematical framework for transformer circuits](https://transformer-circuits.pub/2021/framework/index.html)\.*Transformer Circuits Thread*\.
- Fehlauer et al\. \(2025\)Finlay Fehlauer, Kyle Mahowald, and Tiago Pimentel\. 2025\.[Convergence and divergence of language models under different random seeds](https://doi.org/10.18653/v1/2025.emnlp-main.1675)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 32982–32991, Suzhou, China\. Association for Computational Linguistics\.
- Ferrando and Costa\-jussà \(2024\)Javier Ferrando and Marta R\. Costa\-jussà\. 2024\.[On the similarity of circuits across languages: a case study on the subject\-verb agreement task](https://doi.org/10.18653/v1/2024.findings-emnlp.591)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 10115–10125, Miami, Florida, USA\. Association for Computational Linguistics\.
- Finlayson et al\. \(2021\)Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov\. 2021\.[Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://doi.org/10.18653/v1/2021.acl-long.144)\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 1828–1843, Online\. Association for Computational Linguistics\.
- Futrell and Mahowald \(2026\)Richard Futrell and Kyle Mahowald\. 2026\.[How linguistics learned to stop worrying and love the language models](https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/how-linguistics-learned-to-stop-worrying-and-love-the-language-models/2395C24EB472B60B2514F7D5F93EB9A8)\.*Behavioral and Brain Sciences*, 49:e198\.
- Futrell et al\. \(2019\)Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy\. 2019\.[Neural language models as psycholinguistic subjects: Representations of syntactic state](https://doi.org/10.18653/v1/N19-1004)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 32–42, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Geiger et al\. \(2024\)Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman\. 2024\.[Finding alignments between interpretable causal variables and distributed neural representations](https://proceedings.mlr.press/v236/geiger24a.html)\.In*Proceedings of the Third Conference on Causal Learning and Reasoning*, volume 236 of*Proceedings of Machine Learning Research*, pages 160–187\. PMLR\.
- Giulianelli et al\. \(2018\)Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema\. 2018\.[Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information](https://doi.org/10.18653/v1/W18-5426)\.In*Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, pages 240–248, Brussels, Belgium\. Association for Computational Linguistics\.
- Goldowsky\-Dill et al\. \(2023\)Nicholas Goldowsky\-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora\. 2023\.[Localizing model behavior with path patching](https://arxiv.org/abs/2304.05969)\.*Preprint*, arXiv:2304\.05969\.
- Gulordava et al\. \(2018\)Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni\. 2018\.[Colorless green recurrent networks dream hierarchically](https://doi.org/10.18653/v1/N18-1108)\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\)*, pages 1195–1205, New Orleans, Louisiana\. Association for Computational Linguistics\.
- Hakimi et al\. \(2025\)Ahmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, and Hinrich Schuetze\. 2025\.[Time course MechInterp: Analyzing the evolution of components and knowledge in large language models](https://doi.org/10.18653/v1/2025.findings-acl.654)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 12633–12653, Vienna, Austria\. Association for Computational Linguistics\.
- Hanna et al\. \(2023\)Michael Hanna, Ollie Liu, and Alexandre Variengien\. 2023\.[How does GPT\-2 compute greater\-than?: Interpreting mathematical abilities in a pre\-trained language model](https://openreview.net/forum?id=p4PckNQR8k)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 76033–76060\.
- Hanna et al\. \(2024\)Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov\. 2024\.[Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms](https://openreview.net/forum?id=TZ0CCGDcuT)\.In*First Conference on Language Modeling*\.
- Hewitt and Liang \(2019\)John Hewitt and Percy Liang\. 2019\.[Designing and interpreting probes with control tasks](https://doi.org/10.18653/v1/D19-1275)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 2733–2743, Hong Kong, China\. Association for Computational Linguistics\.
- Hewitt and Manning \(2019\)John Hewitt and Christopher D\. Manning\. 2019\.[A structural probe for finding syntax in word representations](https://doi.org/10.18653/v1/N19-1419)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 4129–4138, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Jafari et al\. \(2025\)Farnoush Rezaei Jafari, Oliver Eberle, Ashkan Khakzar, and Neel Nanda\. 2025\.[RelP: Faithful and efficient circuit discovery in language models via relevance patching](https://arxiv.org/abs/2508.21258)\.*Preprint*, arXiv:2508\.21258\.
- Kangaslahti et al\. \(2026\)Sara Kangaslahti, Elan Rosenfeld, and Naomi Saphra\. 2026\.[Hidden breakthroughs in language model training](https://openreview.net/forum?id=eiEIKGuqaf)\.In*The Fourteenth International Conference on Learning Representations*\.
- Kramár et al\. \(2024\)János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda\. 2024\.[AtP\*: An efficient and scalable method for localizing LLM behaviour to components](https://arxiv.org/abs/2403.00745)\.*Preprint*, arXiv:2403\.00745\.
- Kulmizev et al\. \(2020\)Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre\. 2020\.[Do neural language models show preferences for syntactic formalisms?](https://doi.org/10.18653/v1/2020.acl-main.375)In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4077–4091, Online\. Association for Computational Linguistics\.
- Lakretz et al\. \(2019\)Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni\. 2019\.[The emergence of number and syntax units in LSTM language models](https://doi.org/10.18653/v1/N19-1002)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 11–20, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Lasri et al\. \(2022\)Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell\. 2022\.[Probing for the usage of grammatical number](https://doi.org/10.18653/v1/2022.acl-long.603)\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 8818–8831, Dublin, Ireland\. Association for Computational Linguistics\.
- Linzen et al\. \(2016\)Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg\. 2016\.[Assessing the ability of LSTMs to learn syntax\-sensitive dependencies](https://doi.org/10.1162/tacl_a_00115)\.*Transactions of the Association for Computational Linguistics*, 4:521–535\.
- Marks et al\. \(2025\)Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller\. 2025\.[Sparse feature circuits: Discovering and editing interpretable causal graphs in language models](https://openreview.net/forum?id=I4e82CIDxv)\.In*The Thirteenth International Conference on Learning Representations*\.
- Marvin and Linzen \(2018\)Rebecca Marvin and Tal Linzen\. 2018\.[Targeted syntactic evaluation of language models](https://doi.org/10.18653/v1/D18-1151)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 1192–1202, Brussels, Belgium\. Association for Computational Linguistics\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\. 2017\.[Pointer sentinel mixture models](https://openreview.net/forum?id=Byj72udxe)\.In*International Conference on Learning Representations*\.
- Merrill et al\. \(2023\)William Merrill, Nikolaos Tsilivis, and Aman Shukla\. 2023\.[A tale of two circuits: Grokking as competition of sparse and dense subnetworks](https://openreview.net/forum?id=8GZxtu46Kx)\.In*ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models*\.
- Mueller et al\. \(2020\)Aaron Mueller, Garrett Nicolai, Panayiota Petrou\-Zeniou, Natalia Talmina, and Tal Linzen\. 2020\.[Cross\-linguistic syntactic evaluation of word prediction models](https://doi.org/10.18653/v1/2020.acl-main.490)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 5523–5539, Online\. Association for Computational Linguistics\.
- Nanda \(2023\)Neel Nanda\. 2023\.[Attribution patching: Activation patching at industrial scale](https://www.neelnanda.io/mechanistic-interpretability/attribution-patching)\.
- Nanda et al\. \(2023\)Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt\. 2023\.[Progress measures for grokking via mechanistic interpretability](https://openreview.net/forum?id=9XFSbDPmdW)\.In*The Eleventh International Conference on Learning Representations*\.
- Olah et al\. \(2020\)Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter\. 2020\.[Zoom in: An introduction to circuits](https://doi.org/10.23915/distill.00024.001)\.*Distill*\.
- Ou et al\. \(2025\)Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, and Huajun Chen\. 2025\.[How do LLMs acquire new knowledge? A knowledge circuits perspective on continual pre\-training](https://doi.org/10.18653/v1/2025.findings-acl.1021)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 19889–19913, Vienna, Austria\. Association for Computational Linguistics\.
- Pimentel et al\. \(2020\)Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell\. 2020\.[Information\-theoretic probing for linguistic structure](https://doi.org/10.18653/v1/2020.acl-main.420)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4609–4622, Online\. Association for Computational Linguistics\.
- Power et al\. \(2022\)Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra\. 2022\.[Grokking: Generalization beyond overfitting on small algorithmic datasets](https://arxiv.org/abs/2201.02177)\.*Preprint*, arXiv:2201\.02177\.
- Stolfo et al\. \(2023\)Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan\. 2023\.[A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis](https://doi.org/10.18653/v1/2023.emnlp-main.435)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7035–7052, Singapore\. Association for Computational Linguistics\.
- Sundararajan et al\. \(2017\)Mukund Sundararajan, Ankur Taly, and Qiqi Yan\. 2017\.[Axiomatic attribution for deep networks](https://proceedings.mlr.press/v70/sundararajan17a.html)\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pages 3319–3328\. PMLR\.
- Syed et al\. \(2024\)Aaquib Syed, Can Rager, and Arthur Conmy\. 2024\.[Attribution patching outperforms automated circuit discovery](https://doi.org/10.18653/v1/2024.blackboxnlp-1.25)\.In*Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP*, pages 407–416, Miami, Florida, US\. Association for Computational Linguistics\.
- Tigges et al\. \(2024\)Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman\. 2024\.[LLM circuit analyses are consistent across training and scale](https://doi.org/10.52202/079017-1287)\.In*Advances in Neural Information Processing Systems*, volume 37, pages 40699–40731\.
- van der Wal et al\. \(2025\)Oskar van der Wal, Pietro Lesci, Max Müller\-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem H\. Zuidema, and Stella Biderman\. 2025\.[PolyPythias: Stability and outliers across fifty language model pre\-training runs](https://openreview.net/forum?id=bmrYu2Ekdz)\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\.
- Varma et al\. \(2023\)Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar\. 2023\.[Explaining grokking through circuit efficiency](https://arxiv.org/abs/2309.02390)\.*Preprint*, arXiv:2309\.02390\.
- Wang et al\. \(2025\)George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet\. 2025\.[Differentiation and specialization of attention heads via the refined local learning coefficient](https://openreview.net/forum?id=SUc1UOWndp)\.In*The Thirteenth International Conference on Learning Representations*\.
- Wang et al\. \(2023\)Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt\. 2023\.[Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small](https://openreview.net/forum?id=NpsVSN6o4ul)\.In*The Eleventh International Conference on Learning Representations*\.
- Warstadt et al\. \(2020\)Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng\-Fu Wang, and Samuel R\. Bowman\. 2020\.[BLiMP: The benchmark of linguistic minimal pairs for English](https://doi.org/10.1162/tacl_a_00321)\.*Transactions of the Association for Computational Linguistics*, 8:377–392\.
- Wei et al\. \(2021\)Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick\. 2021\.[Frequency effects on syntactic rule learning in transformers](https://doi.org/10.18653/v1/2021.emnlp-main.72)\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 932–948, Online and Punta Cana, Dominican Republic\. Association for Computational Linguistics\.

## Appendix AComputation cost

We use one A40 GPU to run all experiments\. Forpythia\-1b, a RelP attribution pass over all MLP neurons at one token position, using 400 examples with batch size 4, takes around 40 seconds of GPU time after model loading\. Integrated\-gradient variants are slower, about 2 minutes per checkpoint\-position\.

DAS training uses 400 training examples, batch size 4, 5 epochs, and 1\-dimensional interventions\. In our runs, this takes around 2 minutes per layer\. Forward\-pass based validation is comparatively cheap: a single necessity or sufficiency evaluation over an evaluation set takes about 1–2 seconds\.

## Appendix BUse of AI assistants

We used AI assistants to aid in developing code for experiments and analyses\. We also used AI for summarizing results and suggesting edits\.

## Appendix CTask templates and data

We first introduce our synthetic templates:PP,SubjectRC,ObjectRC, andSimple\. Then we discuss extensions of these synthetic templates, both in structure and in lexical items, that are used to further stress\-test circuit generalization\. Lastly, we show the list of all WikiText\([Merity et al\., 2017](https://arxiv.org/html/2610.33299#bib.bib36)\)agreement sentences\.

All templates share 16 subject pairs \(singular/plural forms of the same lemma\)\. The singular forms are the following:guard, doctor, farmer, author, officer, athlete, manager, senator, consultant, teacher, customer, pilot, executive, minister, actor, architect\. All subjects, objects, and RC verbs are single tokens under thepythia\-1btokenizer\. Base and source sentences always use the singular/plural pair of the same lemma, so they differ only in subject number\. We measure model predictions on the verb tokens \{is,was,has\} for singular and \{are,were,have\} for plural; each is a single token\.

### C\.1Synthetic templates

Below, we detail all synthetic templates used to find and test the circuit\. Exact templates and exemplars can be seen in[Table4](https://arxiv.org/html/2610.33299#A3.T4)\.

#### PP\.

This is the primary template used to find and evaluate our circuit\. We draw distractor noun\-phrases \(NPs\) from 16 human\-role lemmas, using both singular and plural forms of these distractors\. We select from 6 prepositions \(near,behind,in front of,next to,across from,to the side of\)\.

#### ObjectRC\.

We use this template to measure generalization\. We again draw distractor noun\-phrases \(NPs\) from 16 human\-role lemmas, but use only the singular form of these distractors\. Within the distractors, we utilize 8 past\-tense verbs \(hated,injured,liked, etc\.\)\.

#### SubjectRC\.

We use this template to measure generalization\. LikeObjectRC, distractor noun\-phrases \(NPs\) are drawn from 16 human\-role lemmas with only the singular form, and utilize the same set of 8 past\-tense verbs\.

#### Simple\.

We use this template to measure generalization\. We do not sample any distractor NPs for this template as it possesses no distractor, with the position after the subject position being the prediction position\.

#### Adjective insertion\.

Adjective\-on\-subjectinserts a single adjective between the determiner and the subject noun, e\.g\.,The tall guard near the doctors\. Theadjective\-on\-objectperturbation instead inserts a single adjective inside the distractor NP, e\.g\.,The guard near the tall doctors\. There are 16 adjectives total, all applicable to human nouns\. They are the following:tall, old, quiet, busy, new, kind, rich, brave, calm, smart, happy, nice, shy, bold, wise, young\.

#### Adverb insertion\.

Theadverbperturbation places a frequency adverb after the distractor object and before the predicted verb, e\.g\.,The guard near the doctors often\. Unlike the adjective perturbations, this changes the lexical category of the token immediately before the prediction position\. It is therefore a stronger test of whether the downstream readout module is tied to the final distractor NP token used in the original PP template\. There are 6 adverbs:always, often, usually, sometimes, rarely, never\.

#### Prefix insertion\.

The prefix perturbations add material before the subject\. Thesingle\-prefixcondition uses one of the six neutral discourse markers:Apparently,Clearly,Surprisingly,Obviously,In fact,For context; thelocative\-prefixcondition uses one of six neutral locative phrases:In the park,At the office,At the hospital,At the theater,In the garden,On the street\. We avoid tensed prefixes such asYesterday,because they can independently affect the distribution over verb forms\. These perturbations shift the absolute position of the subject and all downstream tokens while leaving the dependency structure unchanged\.

#### Longer distractor\.

This perturbation inserts two hand\-designed adjectives in the object NP, e\.g\.,The guard near the respected senior doctors\. This lengthens the intervening attractor phrase and increases the distance between subject and prediction\. There are 6 such adjective pairs:experienced local,respected senior,careful young,skilled retired,friendly new,talented junior\.

Table 4:Agreement templates used in cross\-template and perturbation generalization experiments\. The object/distractor pool contains both singular and plural forms forPP\-based templates, and base and source examples differ only in the subject number\. We evaluateN=192N=192pairs \(N=32N=32forSimple\)\.

### C\.2Templates for lexical generalizations

The lexical\-generalization experiments keep the PP dependency structure fixed and vary the lexical items used either in the subject slot or in the next\-token verb label\. In all cases, the base/source pair is still a minimal subject\-number pair\. Concrete examples are in[Table5](https://arxiv.org/html/2610.33299#A3.T5)\.

#### Held\-out\-subject\.

This condition replaces the original subject lexicon with 13 single\-token human\-role lemmas:lawyer, engineer, nurse, scientist, driver, artist, musician, dancer, priest, cook, singer, worker, painter\. Both singular and plural forms are single tokens under thepythia\-1btokenizer\. The rest of the template is untouched\.

#### Held\-out\-verb\.

This condition uses the original PP subject, preposition, and distractor pools, but changes the next\-token prediction labels\. We evaluate 36 single\-token third\-person singular/plural verb pairs:runs/run, walks/walk, eats/eat, goes/go, sees/see, knows/know, says/say, gets/get, takes/take, makes/make, comes/come, gives/give, finds/find, tells/tell, thinks/think, feels/feel, writes/write, plays/play, works/work, buys/buy, sells/sell, hears/hear, looks/look, seems/seem, calls/call, means/mean, wants/want, needs/need, loves/love, hates/hate, lives/live, opens/open, closes/close, starts/start, stops/stop, waits/wait\.

Table 5:Lexical\-generalization templates\. We test a wider range of subject and verb tokens in thePPtemplate to ensure circuit robustness\.

### C\.3WikiText agreement cases

We mined 80 agreement sentences from WikiText, balanced across 40 plural\-subject and 40 singular\-subject cases\. For each sentence we manually created a counterfactual that differs in subject number\. The subject, counterfactual subject, and verb continuations tokenize as single tokens\. When the attested WikiText verb is past tense, we manually convert the verb to its present tense and score the singular/plural pair\.

For each item below,b\(base\) is the attested WikiText prefix plus its attested verb \(converted to present\-tense, if necessary\), ands\(source\) is the manually constructed opposite\-number subject version plus the opposite\-number present\-tense verb\.

#### Plural\-subject cases\.

1. 1\.b: The origins of Imagism are s: The origin of Imagism is
2. 2\.b: The characteristics of the sculpture have s: The characteristic of the sculpture has
3. 3\.b: The eyes of the monument are s: The eye of the monument is
4. 4\.b: The affairs of the CWGC are s: The affair of the CWGC is
5. 5\.b: The walls of the nave are s: The wall of the nave is
6. 6\.b: The leaders of the Coalition are s: The leader of the Coalition is
7. 7\.b: The details of the murder are s: The detail of the murder is
8. 8\.b: The reasons behind it are s: The reason behind it is
9. 9\.b: The origins of Olaf are s: The origin of Olaf is
10. 10\.b: The dimensions of the Tetrarch are s: The dimension of the Tetrarch is
11. 11\.b: The objectives of the force are s: The objective of the force is
12. 12\.b: The stones of the dolmen are s: The stone of the dolmen is
13. 13\.b: The symptoms of AML are s: The symptom of AML is
14. 14\.b: The actions of the governor are s: The action of the governor is
15. 15\.b: The highlights of the show are s: The highlight of the show is
16. 16\.b: The commands of the starter are s: The command of the starter is
17. 17\.b: The roles of each deity are s: The role of each deity is
18. 18\.b: The effects of antimony are s: The effect of antimony is
19. 19\.b: The plots of such works are s: The plot of such works is
20. 20\.b: The roles of these area commands are s: The role of these area commands is
21. 21\.b: The reasons for going to war are s: The reason for going to war is
22. 22\.b: The chances of rapid intensification are s: The chance of rapid intensification is
23. 23\.b: The origins of the Knights Hospitaller are s: The origin of the Knights Hospitaller is
24. 24\.b: The arrangements for conscientious objectors are s: The arrangement for conscientious objectors is
25. 25\.b: The regions of north County Galway and east County Mayo have s: The region of north County Galway and east County Mayo has
26. 26\.b: The origins of Hurricane Lorenzo are s: The origin of Hurricane Lorenzo is
27. 27\.b: The shows in Japan are s: The show in Japan is
28. 28\.b: The heads in Lisbon are s: The head in Lisbon is
29. 29\.b: The characteristics of its body, head, fins, and teeth are s: The characteristic of its body, head, fins, and teeth is
30. 30\.b: The levels of ionizing radiation on Io, Europa, and Ganymede are s: The level of ionizing radiation on Io, Europa, and Ganymede is
31. 31\.b: The events of the battle are s: The event of the battle is
32. 32\.b: The districts of Somerset are s: The district of Somerset is
33. 33\.b: The corners of the base have s: The corner of the base has
34. 34\.b: The identities of the perpetrator or the instigator are s: The identity of the perpetrator or the instigator is
35. 35\.b: The statues in the west tower are s: The statue in the west tower is
36. 36\.b: The levels in Perfect Dark have s: The level in Perfect Dark has
37. 37\.b: The songs on the album are s: The song on the album is
38. 38\.b: The objectives of the German, Austrian and Ottoman advance are s: The objective of the German, Austrian and Ottoman advance is
39. 39\.b: The tactics employed by the Anzac Mounted Division are s: The tactic employed by the Anzac Mounted Division is
40. 40\.b: The lines about the corners of the eyes and mouth are s: The line about the corners of the eyes and mouth is

#### Singular\-subject cases\.

1. 1\.b: The game of checkers is s: The games of checkers are
2. 2\.b: The influence of the churches is s: The influences of the churches are
3. 3\.b: The strength of the rearguards is s: The strengths of the rearguards are
4. 4\.b: The tail behind the stings has s: The tails behind the stings have
5. 5\.b: The family of four is s: The families of four are
6. 6\.b: The head of a given divine image is s: The heads of a given divine image are
7. 7\.b: The number of pilots licensed by the CAA to fly powered aircraft in 2005 is s: The numbers of pilots licensed by the CAA to fly powered aircraft in 2005 are
8. 8\.b: The objective of regulation is s: The objectives of regulation are
9. 9\.b: The goal of the blockade is s: The goals of the blockade are
10. 10\.b: The attack on Ancona is s: The attacks on Ancona are
11. 11\.b: The objective of the bombardment of Ancona is s: The objectives of the bombardment of Ancona are
12. 12\.b: The application of the Johnson\-Corey\-Chaykovsky reaction in organic synthesis is s: The applications of the Johnson\-Corey\-Chaykovsky reaction in organic synthesis are
13. 13\.b: The tone of these chapters is s: The tones of these chapters are
14. 14\.b: The structure of Dominican society is s: The structures of Dominican society are
15. 15\.b: The panel to the east of the north portico is s: The panels to the east of the north portico are
16. 16\.b: The door of the central shrine has s: The doors of the central shrine have
17. 17\.b: The state of Polish primary schools is s: The states of Polish primary schools are
18. 18\.b: The inclusion of occupational debris over the bones is s: The inclusions of occupational debris over the bones are
19. 19\.b: The use of Sb as the standard chemical symbol for antimony is s: The uses of Sb as the standard chemical symbol for antimony are
20. 20\.b: The origin of this is s: The origins of this are
21. 21\.b: The formation of halogenated antimony compounds is s: The formations of halogenated antimony compounds are
22. 22\.b: The budget of the film is s: The budgets of the film are
23. 23\.b: The cost of construction is s: The costs of construction are
24. 24\.b: The peak of the roof is s: The peaks of the roof are
25. 25\.b: The method and timing of mowing is s: The methods and timing of mowing are
26. 26\.b: The influence of weather on chick survival is s: The influences of weather on chick survival are
27. 27\.b: The goal of induction therapy is s: The goals of induction therapy are
28. 28\.b: The island of Hispaniola, shared by Haiti and the Dominican Republic, is s: The islands of Hispaniola, shared by Haiti and the Dominican Republic, are
29. 29\.b: The consequence of failing a puzzle of this sort is s: The consequences of failing a puzzle of this sort are
30. 30\.b: The relationship between the tolling points and interchanges along SR 878 and SR 874 is s: The relationships between the tolling points and interchanges along SR 878 and SR 874 are
31. 31\.b: The purpose of this plan is s: The purposes of this plan are
32. 32\.b: The amount of water in a toilet or sink is s: The amounts of water in a toilet or sink are
33. 33\.b: The form of the poem is s: The forms of the poem are
34. 34\.b: The rate of protein synthesis is s: The rates of protein synthesis are
35. 35\.b: The study of proteins in vivo is s: The studies of proteins in vivo are
36. 36\.b: The field of bioinformatics is s: The fields of bioinformatics are
37. 37\.b: The task allocated to the British Commonwealth force is s: The tasks allocated to the British Commonwealth force are
38. 38\.b: The definition of leg before wicket is s: The definitions of leg before wicket are
39. 39\.b: The object on the right seated on legs alongside a box is s: The objects on the right seated on legs alongside a box are
40. 40\.b: The song, “a cheery little tune about death, destruction and the end of the world” is s: The songs, “a cheery little tune about death, destruction and the end of the world” are

## Appendix DAdditional methodological details

### D\.1RelP neuron attribution

We apply RelP\([Jafari et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib27);[Arora et al\., 2026](https://arxiv.org/html/2610.33299#bib.bib4)\)to form a broad candidate set of MLP neurons before running causal validation\. The attribution score is computed over the PP training set \(N=400N=400\)\.

#### Replacement\-gradient attribution\.

For each layerLLand neuronnn, we capture the post\-nonlinearity MLP activationvL,nv\_\{L,n\}at the relevant token position and score

scoreL,n=𝔼\(𝐛,𝐬\)​\[vL,n​\(𝐛\)​∂m⁡\(𝐛,𝐬\)∂vL,n​\(𝐛\)\],\\mathrm\{score\}\_\{L,n\}=\\mathbb\{E\}\_\{\(\\mathbf\{b\},\\mathbf\{s\}\)\}\\left\[v\_\{L,n\}\(\\mathbf\{b\}\)\\frac\{\\partial m\(\\mathbf\{b\},\\mathbf\{s\}\)\}\{\\partial v\_\{L,n\}\(\\mathbf\{b\}\)\}\\right\],wherem⁡\(𝐛,𝐬\)=log⁡\(p⁡\(ℓ𝐛∣𝐛\)\)−log⁡\(p⁡\(ℓ𝐬∣𝐛\)\)m\(\\mathbf\{b\},\\mathbf\{s\}\)=\\log\(p\(\\ell\_\{\\mathbf\{b\}\}\\mid\\mathbf\{b\}\)\)\-\\log\(p\(\\ell\_\{\\mathbf\{s\}\}\\mid\\mathbf\{b\}\)\)is the per\-example agreement gap\.

Linearizations are architecture specific\. For Pythia, we use the following locally linear replacement model: LayerNorm centering and scaling factors are frozen, GELUs are replaced by their local linear gates, and attention softmax probabilities are detached before the value\-weighted sum\.

There are two implementation choices which deviate from[Arora et al\. \(2026\)](https://arxiv.org/html/2610.33299#bib.bib4)— both of which we show do not qualitatively affect our results\. First, the score in[Equation5](https://arxiv.org/html/2610.33299#S3.E5)uses the zero\-baseline formvn⋅∇v\_\{n\}\\cdot\\nabla, which[Arora et al\. \(2026\)](https://arxiv.org/html/2610.33299#bib.bib4)adopt in the unpaired setting, rather than the activation\-difference formΔvn⋅∇\\Delta v\_\{n\}\\cdot\\nablathey use with paired data\. The former estimates a neuron’s contribution on the base input, while the latter estimates the effect of activation replacement from the counterfactual\. Zero\-baseline is a computationally simpler method since it does not require a source\-activation pass and storage\. This choice has little impact on the candidates: comparing both the zero\-baseline formvn⋅∇v\_\{n\}\\cdot\\nablaand the activation\-difference formΔvn⋅∇\\Delta v\_\{n\}\\cdot\\nabla, we find that they rank the top5050neurons identically and overlap on944944of the top10241024neurons pooled across positions\. This suggests our findings are largely unaffected by the choice of zero\-baseline\.

Second, we also detach the outputs of all other MLPs when scoring a neuron within a particular layerLL, so the gradient reachingLLflows only through the residual stream and attention\.[Arora et al\. \(2026\)](https://arxiv.org/html/2610.33299#bib.bib4)introduce this for edge attribution, but we also apply it to node scoring here\. There is no clear unbiased choice here: our approach will underweight neurons whose contributions depend on downstream MLP processing, while the alternative can give credit to both upstream and downstream neurons that mediate the same effect\.

We compared the two approaches and found that the broader ranking changes, but the conclusion does not\. We recomputed the top neuron candidates forpythia\-1bwith the stop\-gradient removed \(i\.e\., total\-effect scoring, where the gradient can flow through downstream MLPs\)\. The two methods overlap in 42 of the top 50 ranking neurons, though this overlap falls to 356 at top\-1024\. However, the final circuit’s 18 neurons are all within the top\-35 in both rankings, and the sparsification claim of[Section4\.2](https://arxiv.org/html/2610.33299#S4.SS2)does not change\.

[Table6](https://arxiv.org/html/2610.33299#A4.T6)shows that the portion of necessity or sufficiency recovered from adding the top\-16 neurons to the head\-baseline remains low early in training and high later in training, regardless of direct\- or total\-effect scoring\.

Table 6:Sparsification is robust to the stop gradient implementation\. All values are percentages on thePPevaluation set\. “Full pool” is the necessity/sufficiency of the 1024\-neuron circuit with respective attention heads, and “Top\-16” is the fraction of the gain from the head\-only baseline to the full pool captured by the 16 highest\-ranked neurons\.
#### Integrated\-gradient check\.

As a robustness check during candidate generation, we also compute integrated gradients on MLP activations\([Sundararajan et al\., 2017](https://arxiv.org/html/2610.33299#bib.bib46);[Marks et al\., 2025](https://arxiv.org/html/2610.33299#bib.bib34)\)\. This method interpolates the captured MLP activation from the counterfactual source activation to the base activation, keeps all other activations on the base run, and averages gradients along akk\-step path\. The resulting score is

\(vL,nbase−vL,nsrc\)⋅1k∑i=1k∂m∂vL,n\(i\)\.\(v^\{\\mathrm\{base\}\}\_\{L,n\}\-v^\{\\mathrm\{src\}\}\_\{L,n\}\)\\cdot\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\frac\{\\partial m\}\{\\partial v^\{\(i\)\}\_\{L,n\}\}\.The integrated\-gradient scores are used to check that high\-ranked candidates are not biased by the local linearizations\.

Usingk=8k=8, we find that at the final checkpoint ofpythia\-1b, all1818neurons in the final circuit fall within the top3535neurons under both pooled RelP rankings and integrated gradients, and the two agree on4444of the top5050neurons\.

### D\.2Neuron–head routing attribution

At each checkpoint, we consider the top\-32 positive RelP neurons at the subject, preposition, and prediction positions\. In[Section5\.1](https://arxiv.org/html/2610.33299#S5.SS1), we ask whether these neurons route through the earlyL15 H7pathway, the laterL13 H7pathway, a mixture of the two, or neither\.

Our detailed methodology is as follows\. For an upstream neuronnn, we first compute its sufficiency in isolation\. Next, we repeat the intervention, but this time hold one downstream headhhat its base value \(for the edge between the neuron’s position and the prediction position\)\. This manipulation blocksnn’s effect from propagating throughhh\. We denote the neuron’s*routed sufficiency*viahhas:

RSuffn→h=Suff⁡\(n\)−Suff⁡\(n;block⁡\(h\)\)\.\\operatorname\{RSuff\}\_\{n\\rightarrow h\}=\\operatorname\{Suff\}\(n\)\-\\operatorname\{Suff\}\(n;\\operatorname\{block\}\(h\)\)\.The routed sufficiency value denotes the portion of the neuron’s effect that disappears oncehhis blocked\.

Similarly, for a downstream neuron,nn, and an upstream head,hh, we first intervene on the head, then repeat the intervention whilst restoring the neuronnn’s activation\.

RSuffh→n=Suff⁡\(h\)−Suff⁡\(h;restore⁡\(n\)\)\.\\operatorname\{RSuff\}\_\{h\\rightarrow n\}=\\operatorname\{Suff\}\(h\)\-\\operatorname\{Suff\}\(h;\\operatorname\{restore\}\(n\)\)\.This measures how much the head intervention effect is mediated throughnn\.

#### Visualization\.

We now explain how we colored neurons in[Figure4](https://arxiv.org/html/2610.33299#S4.F4)\. These computations and thresholds are purely for visualization purposes\.

We first identify the neurons on either head’s route\. This takes different forms for upstream and downstream neurons\. For an upstream neuron to qualify, we jointly block both heads, and require the neuron to lose at least5%5\\%of the standalone sufficiency\. For a downstream neuron, we require restoring the activation to remove at least5%5\\%of the joint sufficiency from an intervention on both heads\. If a neuron does not meet these thresholds, it is considered to not be on either head’s route\.

For the neurons on either head’s routes, we attribute their colors as follows\. We first define

r13​\(n\)=\{RSuffn→13,n​is upstream,RSuff13→n,n​is downstream,r\_\{13\}\(n\)=\\begin\{cases\}\\operatorname\{RSuff\}\_\{n\\rightarrow 13\},&n\\text\{ is upstream\},\\\\ \\operatorname\{RSuff\}\_\{13\\rightarrow n\},&n\\text\{ is downstream\},\\end\{cases\}and definer15​\(n\)r\_\{15\}\(n\)likewise\. This allows us to define positive routing shares

s13​\(n\)\\displaystyle s\_\{13\}\(n\)=\[r13​\(n\)\]\+\[r13​\(n\)\]\+\+\[r15​\(n\)\]\+,\\displaystyle=\\frac\{\[r\_\{13\}\(n\)\]\_\{\+\}\}\{\[r\_\{13\}\(n\)\]\_\{\+\}\+\[r\_\{15\}\(n\)\]\_\{\+\}\},s15​\(n\)\\displaystyle s\_\{15\}\(n\)=1−s13​\(n\),\\displaystyle=1\-s\_\{13\}\(n\),where\[x\]\+\[x\]\_\{\+\}denotesmax⁡\(x,0\)\\max\(x,0\)\.

This metric gives us a gradient between how much a neuron isL15 H7\-routed orL13 H7\-routed\. Neurons are duly colored by this gradient: blue whens15=1s\_\{15\}=1, purple when the two shares are equal, and red whens13=1s\_\{13\}=1\.

## Appendix EExperiment 1: Additional results

### E\.1Different levels ofkk

We also produce[Figure2](https://arxiv.org/html/2610.33299#S4.F2)at different threshold levels\. Definek30k\_\{30\}andk70k\_\{70\}similarly to eq\.[6](https://arxiv.org/html/2610.33299#S4.E6), we reproduce[Figure2](https://arxiv.org/html/2610.33299#S4.F2)in[Figure6](https://arxiv.org/html/2610.33299#A5.F6)and[Figure6](https://arxiv.org/html/2610.33299#A5.F6), respectively\.

We see that broadly speaking, the number of neurons in these sets decrease over training\. Fork70k\_\{70\}inpythia\-1b, we see a rebound at step 143k\. This is likely due to the final circuit ofpythia\-1brelying on downstream prediction\-position readout neurons, which we find in Experiment 2\.

We take these results to be consistent with the trend that causal effects of subject neurons becomes more concentrated over training\. This notion of sparsification may not always be reflected in our figures, and building a complete circuit requires examining model\- and checkpoint\-specific mechanisms beyond the subject position\.

Figure 5:Agreement accuracy vs\.k30k\_\{30\}over training checkpoints\.Figure 6:Agreement accuracy vs\.k70k\_\{70\}over training checkpoints\.

### E\.2Overlap of neuron sets across training

For each checkpoint, we report each model’sk50k\_\{50\}set size, and report the overlap between consecutive evaluated checkpoints’k50k\_\{50\}sets\. Since this set does not monotonically decrease, we “normalize” by also showing the smaller size of two consecutive sets\.

Table 7:Size and overlap of thek50k\_\{50\}neuron sets across checkpoints\. The first number is thek50k\_\{50\}size, the numerator of the fraction is the overlap between the checkpoint’s and the previous checkpoint’sk50k\_\{50\}set, and the denominator of the fraction is the size of the smallerk50k\_\{50\}set of the two\.Our results are in[Table7](https://arxiv.org/html/2610.33299#A5.T7)\. We see that generally, latter checkpoint’s neuron sets are largely subsets of the previous checkpoint’s neuron sets\. In the rare case thatk50k\_\{50\}counts expand \(possibly due to instabilities in how they are estimated\), the neurons belonging to the smaller set are largely retained\. Outliers show a qualitatively different pattern entirely, with abrupt changes in their neuron set memberships\.

### E\.3Late\-training outlier trajectories

The pattern of models first learning behavior before the internal mechanism sparsifies seems to hold across non\-outlier seeds\. However, the outlier seeds are interestingly different —[Figure7](https://arxiv.org/html/2610.33299#A5.F7)shows that at exactly the training setbacks reported by[van der Wal et al\. \(2025\)](https://arxiv.org/html/2610.33299#bib.bib49), the outlier models’k50k\_\{50\}scores rebound\. For seed 3, the setback is not on agreement behavior but only on the mechanism, so subsequent training simply re\-sparsifies its neuron mechanism\. However, seed 4 loses a significant portion of its agreement abilities\. As such, in the remaining training steps, it relearns agreement, but runs out of time for re\-sparsification\.

Taken together, these results suggest that there is a general pressure to sparsify after agreement is learned, even for outlier runs\.

Figure 7:k50k\_\{50\}rebound co\-occurs with training regressions\. Seed 3 rebound happens early and the model recovers a sparse neuron mechanism by end of training\. Seed 4 suffered setbacks late and itsk50k\_\{50\}ended high\.
### E\.4Attention\-head localization across checkpoints and seeds

To identify candidate transport heads, we run a scan on all heads at the prediction position at each checkpoint, testing for both necessity and sufficiency\. We rank heads bymin⁡\(Nec,Suff\)\\min\(\\operatorname\{Nec\},\\operatorname\{Suff\}\)\.

Forpythia\-1b, we show the top\-5 heads at each checkpoint in[Table8](https://arxiv.org/html/2610.33299#A5.T8), bolding the three heads in the emergence analysis \(L13 H7,L15 H7, andL15 H2\)\. We omit early checkpoints because normalizing against a near\-zero agreement gap for necessity/sufficiency is unstable\. Across the ten seeds, the dominant channel at the end of training is between layers 16–23 \(the last layer has index 23\)\. On average, the head settles at a depth of 83% relative to the total number of layers, comparable to13/16≈0\.8113/16\\approx 0\.81inpythia\-1b\. This suggests that the late\-layer placement of the transport head may not be incidental\.

Table 8:Top five attention heads at each checkpoint scored bymin⁡\(Nec,Suff\)\\min\(\\mathrm\{Nec\},\\mathrm\{Suff\}\)\. The three main heads are bolded\.Table 9:Leading head across ten seeds ofpythia\-410mat step 2k and step 143k\. Score denotesmin⁡\(Nec,Suff\)\\min\(\\operatorname\{Nec\},\\operatorname\{Suff\}\)\. There is no predictable pattern in how the models reorganize the transport channels\.We similarly track the top heads across seeds ofpythia\-410mbymin⁡\(Nec,Suff\)\\min\(\\operatorname\{Nec\},\\operatorname\{Suff\}\), with the tracked heads and results in[Table9](https://arxiv.org/html/2610.33299#A5.T9)\. The composition of the top\-three head set changes in every run, suggesting that the primary transport channel is generally unstable across training\. The single most dominant head changes in bothpythia\-1band six seeds ofpythia\-410m\. Among such runs, in three seeds the transport head moves from a later layer toward a middle layer, echoing thepythia\-1bpattern\. Two seeds, 4 and 6, shift toward a later layer\.

We leave for future studies to understand the learning pressures that drive these dynamics\.

## Appendix FExperiment 2: Additional results

### F\.1Initial circuit discovery details

The neuron candidates for our final circuit were those which write toward the next\-layer number representation and those with high RelP scores\. Here we provide some details on how we identified number representations in the model, and how we pruned to obtain the final circuit\.

#### Distributed alignment search for number\.

We use Distributed Alignment Search\([Geiger et al\., 2024](https://arxiv.org/html/2610.33299#bib.bib18), DAS;\)to test whether the agreement information carried byL13 H7is concentrated in a low\-dimensional subspace\. To this end, we train DAS, with the default orthogonal alignment map set at one dimension, on thePPtraining set with learning rate 5e\-3 and batch size 4, for 5 epochs\. We train at every layer in the subject and prediction positions with five initialization seeds\. To score a neuron’s output alignment with one representative number\-feature subspace, we use the average of the five subspaces after normalization\.

#### Pruning\.

Neuron candidates surfaced by output\-alignment or RelP are not necessarily causally important, and it is possible that two candidates can exhibit interaction effects that lower their joint effects\. For these reasons, after constructing an initial circuit with promising neurons, we apply leave\-one\-out validation to prune the circuit\.

On the 21\-neuron circuit, we remove each remaining neuron and re\-evaluate the circuit on thePPtraining set\. We remove the neuron whose deletion had the least effect on circuit necessity and sufficiency\. This applied to both neurons with positive and negative agreement effects, as we want to keep strong anti\-agreement neurons to better represent the model’s mechanism\. We repeated this process three times to remove three neurons\.

### F\.2Circuit scope

Our final circuit is presented as a sparse representation of howpythia\-1bcarries the number feature from the subject position \(from L1 onwards\) to the prediction position viaL13 H7\. To this end, we ignorepos0\\text\{pos\}\_\{0\}andpos3\\text\{pos\}\_\{3\}, exclude the L0 MLP from the scope, and ignore the effects of self\-attention\. We provide justification for these decisions below\.

#### Position selection\.

We first show that we can safely ignore neurons atpos0\\text\{pos\}\_\{0\}andpos3\\text\{pos\}\_\{3\}\(which are both the token “the”\) by confirming that no relevant heads transfer information from those positions, and that their RelP attributions are low\.

To this end, we first compute each head’s sufficiency on thePPtraining set at each checkpoint\. Then, we compute the sufficiency from intervening on the head, but restricted only to the attention between a source token position and the prediction position\. The fraction of the edge\-restricted sufficiency relative to the full\-head sufficiency serves as a measure for the degree of importance for this position\.

We show this metric across heads and checkpoints in[Table10](https://arxiv.org/html/2610.33299#A6.T10)\. Position\-specific edge patching shows that the early L15 transport heads are almost entirely subject\-to\-prediction channels\. By contrast,L13 H7starts as a weak subject edge but becomes the dominant late channel, reading both from the subject edge and from the preposition edge at 143k\. The determiner positions are not accessed by any of these heads\.

[Table11](https://arxiv.org/html/2610.33299#A6.T11)further shows the maximum absolute RelP score over all MLP neurons in all layers, demonstrating that the estimated effect of neurons at determiner positionspos0\\text\{pos\}\_\{0\}andpos3\\text\{pos\}\_\{3\}are negligible\. This result provides further confirmation that excluding these positions has negligible impact on the final circuit\.

Table 10:Position\-specific edge interventions into the prediction position\. Entries are fractions of the full head sufficiency recovered by the single\-edge intervention\.L15 H7andL15 H2are concentrated on subject\-to\-prediction edges, whileL13 H7is a strong subject edge that increasingly uses the preposition\-to\-prediction edge late in training\.Table 11:Maximum absolute RelP score byPPtemplate token position across checkpoints\. Largest RelP scores occur at the subject and prediction\-side positions after agreement emerges, while the sentence\-initial determiner and the distractor determiner remain small throughout\.
#### The L0 MLP\.

We find ablating the complement of the circuit is unusually destructive at L0\. That is, we compare full MLP mean ablation at the subject position for L0 and L5, using thePPsubject\-position mean as the replacement and measure cross\-entropy of the agreement target at the prediction position\. Our results can be seen in[Table12](https://arxiv.org/html/2610.33299#A6.T12)\. On both PP templates and hand\-curated WikiText agreement prefixes, L0 mean ablation raises CE by about0\.820\.82nats, while L5 mean ablation leaves CE essentially unchanged\. For this reason, we only present our circuit relative to a subject\-to\-prediction path that excludes subject L0\. Further, these results suggest that L0 may be responsible for the*construction*of the number feature, with our circuit merely responsible for its*transport*to prediction\.

#### Subject\-side self\-attention makes little contribution to agreement\.

We find that mean ablating of each layer and head demonstrates that the self\-attention contribution at the subject position,αh​\[pos1,pos1\]​WOh​Vh​\(xpos1\)\\alpha\_\{h\}\[\\text\{pos\}\_\{1\},\\text\{pos\}\_\{1\}\]W\_\{O\}^\{h\}V\_\{h\}\(x\_\{\\text\{pos\}\_\{1\}\}\)is negligible\. In particular, we replace such values with the corresponding dataset mean while leaving non\-self attention edges untouched\. Across thePPdataset this procedure changes the agreement gap from\+3\.48\+3\.48to\+3\.43\+3\.43, a drop of only1\.5%1\.5\\%\. This supports ignoring attention heads at the subject position\. At the prediction position, no single self\-attention head under ablation causes a drop of more than2\.5%2\.5\\%of the baseline \(L9 H7\), and heads L14 or later have near\-zero mean ablation effects\.

Table 12:Cross\-entropy impact of full\-MLP mean ablation at the subject position\. CE is measured on the correct agreement target at the prediction position\.PPevaluation set\(N=192\)\(N=192\), WikiText\(N=80\)\(N=80\)\.

### F\.3Circuit components and functional roles

Table 13:Components of the subject–verb agreement circuit𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}inpythia\-1b\.𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}transports the number feature, and𝒞𝙿\\mathcal\{C\}\_\{\\mathtt\{P\}\}and𝒞𝙽\\mathcal\{C\}\_\{\\mathtt\{N\}\}neurons promote or suppress the labeled verb forms\.A full list of circuit components is in[Table13](https://arxiv.org/html/2610.33299#A6.T13), consisting of four modules:𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\},𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\},𝒞𝙿\\mathcal\{C\}\_\{\\mathtt\{P\}\}, and𝒞𝙽\\mathcal\{C\}\_\{\\mathtt\{N\}\}\. Here we describe our interpretation of each module’s contribution to the final circuit, as well as the relevant metrics supporting such interpretations\.

#### Amplifiers\.

The upstream neurons in𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}behave primarily as plural detectors\. To demonstrate this, we first hand\-wrote 95 singular/plural noun pairs, and then used AI assistance to extract words ending in\-sin the first 5000 rows of WikiText, giving us a total of447447singular/plural noun pairs\. We also manually generate 144 non\-nouns which are single\-token verbs, adjectives, adverbs, and function words\. We measure all MLP neurons’ activations at the subject position for these words\. To mirror the template, activations are extracted in the context<\|endoftext\|\>The \{word\}\.

The𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}neurons activate similarly, near 0 on singular nouns and non\-nouns, but activate strongly on plural nouns\. Most of the selected𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}circuit neurons are among the top one or two plural\-selective neurons \(i\.e\., largest mean\-activation difference between plural and singular inputs\) in their layer:L1 n7988,L5 n3793,L8 n6308,L9 n5060,L11 n6624, andL12 n1535are ranked first by plural\-selectivity in their respective layers;L9 n3101,L10 n6267,L10 n747, andL11 n239are also highly ranked\. See[Table14](https://arxiv.org/html/2610.33299#A6.T14)\.

Table 14:Activations for𝒞𝙰\\mathcal\{C\}\_\{\\mathtt\{A\}\}at the subject position\. Means are post\-activation MLP neuron values\. The selected neurons distinguish plural nouns from singular nouns, while singular nouns remain close to non\-nouns\.
#### Transport\.

Results from Experiment 1 at the final checkpoint tell us thatL13 H7is the most important head for agreement\. Further, App\.[F\.6](https://arxiv.org/html/2610.33299#A6.SS6)shows that we can localize the number channel to a 1D subspace within the head\. Taken together, these results characterize𝒞𝙷\\mathcal\{C\}\_\{\\mathtt\{H\}\}as the transport channel for the circuit, moving relevant information from the subject position to the prediction position\.

#### Readout: pro\- and anti\-agreement neurons\.

The prediction\-side neurons are readout components\. They are gated by subject number and write directly toward or away from the correct verb\-number direction\. The two strongest pro\-readouts,L14 n1160andL14 n4146, have opposite output alignments:L14 n1160promotes plural verbs such asare,were, andhave, while suppressing singulars such asis,was, andhas;L14 n4146has the opposite alignment\.

Their activations also appear to be gated by the upstream subject number:L14 n1160is active on plural\-subject examples and nearly inactive on singular\-subject examples, whileL14 n4146is active on singular\-subject examples and nearly inactive on plural\-subject examples\.L14 n6888is an additional singular\-side readout with the same gating pattern\. See[Table15](https://arxiv.org/html/2610.33299#A6.T15)\.

Table 15:Readout neuron activations at the prediction position\. The pro\- and anti\-readouts are gated by the number state reaching the prediction position: plural\-side neurons activate on plural\-subject examples, while singular\-side neurons activate on singular\-subject examples\. Each mean is computed over9696singular\-subject and9696plural\-subject base sentences in thePPevaluation set \(N=192N=192\)\.The anti\-readouts are active components of the same computation but write in the opposite direction from the pro\-readouts\. For a baseline agreement gap of3\.483\.48, individually mean ablating anti\-agreement neurons increases the gap by2\.7%2\.7\\%–7\.0%7\.0\\%\. Mean ablating all five increases the gap24\.7%24\.7\\%relative to baseline \(N=192N=192\)\.

### F\.4Off\-circuit interventions

As an additional robustness test, we intervene on non\-circuit MLP activations within the declared scope\. Here, we intervene on all MLP blocks within the scope \(L1–L12 MLPs atpos1\\text\{pos\}\_\{1\}and L14–L15 MLPs atpos4\\text\{pos\}\_\{4\}\) except the circuit neurons\.

For each base sentence𝐛\\mathbf\{b\}in thePPevaluation set, we intervene on𝐛\\mathbf\{b\}with2020randomly sampled donor PP sentences and measure the agreement gap\. If the complement does not participate in agreement, the post\-intervention agreement gap should also show little to no sensitivity to the donor’s number\. Indeed,[Table16](https://arxiv.org/html/2610.33299#A6.T16)shows that the circuit\-complement has little effect on agreement under intervention\.

Table 16:Agreement gap under off\-circuit intervention \(N=192N=192,Ndonors=20N\_\{\\text\{donors\}\}=20each; 1924 same\-class and 1916 cross\-class trials\)\. With the circuit untouched, replacing relevant non\-circuit activation with a donor’s activation leaves agreement intact regardless of donor class\.
### F\.5Generalization datasets and additional results\.

In the main text, we have shown that the circuit generalizes to other synthetic templates and naturalistic sentences\. We further show that the final circuit is a robust agreement mechanism by showing that the circuit \(1\) can be used to control agreement across templates, \(2\) applies to most perturbations to thePPtemplate, with the exception of adding an adverb before predicting the next verb, and \(3\) applies to subjects and verbs in general\.

#### Circuit activations are transferable\.

The circuit controls agreement across templates, so we expect that activations on the modules are transferable across templates\. We test this by intervening on the circuit𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}across templates, and measuring sufficiency\.

For each pair of templates, we createN=192N=192pairs \(N=32N=32forSimple\) differing in subject number drawn from their evaluation sets\. We intervene on circuit components, and measure sufficiency\.[Table17](https://arxiv.org/html/2610.33299#A6.T17)contains our results\.

Table 17:Sufficiency from cross\-template intervention\. Rows are the base template; columns are the donor template \(each pair drawn at random, with the donor’s subject opposite\-number from base’s\)\. Donors are drawn to have opposite number to each base sentence\.N=32N=32base examples forSimpleandN=192N=192otherwise, balanced across sg/pl\-base\.We see that interventions across templates achieve comparable sufficiency to interventions within templates\. TheSimpletemplate is the odd one out: circuit activations on other templates transfer into theSimpletemplate effectively, but not vice versa\.

#### Generalization to perturbations\.

We test whether the circuit is sensitive to absolute positions of the subject and prediction tokens by considering 6 perturbations of the PP template described in App\.[C\.1](https://arxiv.org/html/2610.33299#A3.SS1): adjective on subject, adjective on object, adverb insertion, clause\-initial prefix, locative prefix, and longer distractor\. For each perturbation, we apply interventions on𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}across pairs differing in subject number and report sufficiency\. Results can be seen in[Table19](https://arxiv.org/html/2610.33299#A6.T19)\. We find that the circuit is sufficient for most perturbations, staying within81%−88%81\\%\-88\\%range, compared to the PP template’s83%83\\%\. Adding an adverb after the object is the exception, which suggests that the circuit readout components may be specific to predictions after NPs\. Follow\-up work could test this by holding constant the circuit components before the readout position and searching for relevant readout neurons\.

Table 18:On\-circuit intervention on the perturbedPPtemplate \(N=192N=192\)\. The circuit applies to all settings apart from adding an adverb after the distractor object\.Table 19:The circuit generalizes on thePPtemplate with held\-out subjects or with held\-out verbs\.N=192N=192pairs per condition\. Held\-out subjects experiment samples from 13 novel subjects\. Held\-out verb experiment averages continuation probabilities across 36 novel verb singular/plural pairs\.
#### Generalization to novel words\.

So far, we have evaluated our circuit on a limited set of 16 subjects and objects, and measured probabilities on only six verb forms\. Here, we show that the circuit is not only general across templates and perturbations, but robust to the choice of lexical items in the templates\.

For this test, we create 13 single\-token singular/plural subject pairs not in the original 16\-pair lexicon, and 36 single\-token singular/plural verb pairs\. Examples of these are shown in App\.[C\.2](https://arxiv.org/html/2610.33299#A3.SS2)\.

We then intervene on𝒞𝚏𝚞𝚕𝚕\\mathcal\{C\}\_\{\\mathtt\{full\}\}, and find that it indeed applies to novel subjects and verbs \([Table19](https://arxiv.org/html/2610.33299#A6.T19)\)\. The lower baseline gap in held\-out verbs reflects the model’s weaker preference for less common verb completions, but the intervention still brings the gap85%85\\%of the way to the natural target\.

### F\.6DAS analysis of the transport channel

#### Training details\.

We use a rotation alignment map with 1D intervention\. The DAS intervention is applied toL13 H7’s value\-output channel at the prediction position\. Inpythia\-1b, this is the 256\-dimensional head slice inside the attention value output forL13 H7\. Training uses thePPtraining set with batch size44,55epochs, Adam optimizer with learning rate5⋅10−35\\cdot 10^\{\-3\}, and three random seeds\.

#### Ablation evaluation\.

In[Section5](https://arxiv.org/html/2610.33299#S5), we demonstrate the sufficiency of the 1D channel for controlling agreement behavior\. In this section, we further evaluate mean ablation variants on thePPevaluation set\. We first compute the meanL13 H7pre\-output\-projection activation at the prediction position over the base examples\. Letx∈ℝ256x\\in\\mathbb\{R\}^\{256\}be the naturalL13 H7slice for an example, letx¯\\bar\{x\}be the mean slice, and letvvbe the normalized 1D DAS direction\.

For necessity, we compare three ablations:

full:x′\\displaystyle\\text\{full:\}\\quad x^\{\\prime\}=x¯,\\displaystyle=\\bar\{x\},\(7\)1D channel:x′\\displaystyle\\text\{1D channel:\}\\quad x^\{\\prime\}=x\+\(⟨x¯,v⟩−⟨x,v⟩\)​v,\\displaystyle=x\+\(\\langle\\bar\{x\},v\\rangle\-\\langle x,v\\rangle\)v,\(8\)complement:x′\\displaystyle\\text\{complement:\}\\quad x^\{\\prime\}=x¯\+\(⟨x,v⟩−⟨x¯,v⟩\)​v\.\\displaystyle=\\bar\{x\}\+\(\\langle x,v\\rangle\-\\langle\\bar\{x\},v\\rangle\)v\.\(9\)The full ablation replaces the entire 256\-dimensional head slice with its mean\. The 1D\-channel ablation replaces only the DAS direction component with its mean value and preserves the orthogonal complement\. The complement ablation does the reverse, mean ablating the orthogonal complement while preserving the natural DAS\-direction component\.