Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution

# A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution
Source: [https://arxiv.org/html/2604.16889](https://arxiv.org/html/2604.16889)
###### Abstract

Existing feature\-interpretation pipelines typically operate on uniformly sampled units, but only a small fraction of cross\-layer transcoder \(CLT\) features matter for a target behavior, with the rest resulting in expensive feature explaining and evaluating costs\. We introduce the first CLT\-native end\-to\-end framework,PIE, connecting Pruning, automatic Interpretation, and interpretation Evaluation, enabling systematic measurement of behavioral fidelity and downstream interpretability under pruning\. To achieve this, we propose Feature Attribution Patching \(FAP\), a patch\-grounded attribution method that scores CLT features by aggregating gradient\-weighted write contributions, and FAP\-Synergy, a synergy\-aware reranking procedure\. We evaluate pruning using KL\-divergence behavior retention and assess interpretation quality with FADE\-style metrics\. Across IOI and Doc\-String, across budgetsK∈\{50,100,200,400,800\}K\\in\\\{50,100,200,400,800\\\}, and across FAP, FAP\-Synergy, Activation\-Magnitude, and ACDC\-style pruning, the FAP family consistently achieves the best or near\-best fidelity, with FAP\-Synergy providing its clearest gains in strict\-budget regimes\. On IOI with CLTs for Llama\-3\.2\-1B and Gemma\-2\-2B, pruning toK=100K\{=\}100features matches the KL fidelity that random selection from the active feature set requires≈4​k\\approx 4\\text\{k\}features to achieve \(≈40×\\approx 40\\timescompression\), enabling≈40×\\approx 40\\timesfewer interpretation/evaluation calls while substantially reducing low\-quality features\. Our code is available at[https://github\.com/Qinhao\-Chen/PIE\-Pipeline](https://github.com/Qinhao-Chen/PIE-Pipeline)\.

Machine Learning, ICML

## 1Introduction

Mechanistic interpretability aims to interpret model behavior by identifying internal components and causal pathways that mediate computation from inputs to outputs\(Olah,[2022](https://arxiv.org/html/2604.16889#bib.bib5)\)\. This program has gained urgency as large language models \(LLMs\) are deployed in increasingly high\-stakes settings where transparency, auditing, and targeted intervention matter \(e\.g\., safety and biomedical decision support\)\(Templetonet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib24); Yanget al\.,[2022](https://arxiv.org/html/2604.16889#bib.bib58); Bandet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib57); Chenet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib64)\)\. Recent progress has moved from neuron\-level anecdotes toward scalable*feature\- and circuit\-level*accounts, enabled by sparse feature bases such as sparse autoencoders \(SAEs\) and replacement\-model approaches such as cross\-layer transcoders \(CLTs\)\(Brickenet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib23); Ameisenet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib31); Conmyet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib14); Syedet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib49)\)\.

![Refer to caption](https://arxiv.org/html/2604.16889v1/x1.png)Figure 1:The PIE Framework: A CLT\-Native End\-to\-End Pipeline\. We propose a three\-stage framework to enable scalable interpretability:Stage I \(Prune\)filters the massive search space of CLT feature\-write edges into a sparse ”Causal Core” using Feature Attribution Patching \(FAP\) and its synergy\-aware variant \(FAP\-Synergy\), which reranks boundary features based on pairwise interactions\.Stage II \(Interpret\)generates natural language descriptions only for the retained features using exemplar\-based prompting, drastically reducing interpretation costs\.Stage III \(Evaluate\)performs a dual\-objective assessment: quantifying behavioral fidelity \(KL Divergence, PCR\) and measuring interpretation quality via automated metrics \(Clarity, Purity, Responsiveness\)\.However, despite rapid advances in circuit discovery, a central bottleneck remains:which internal units should we spend interpretation budget on?Modern feature dictionaries can contain hundreds of thousands to millions of features, while downstream auto\-interpretation and evaluation pipelines are expensive per feature\(Billset al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib26); Pauloet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib25); Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55); Boggustet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib50)\)\. In practice, an end\-to\-end interpretability workflow must allocate compute carefully: explaining*everything*is infeasible, and explaining uninformative features is wasteful\.

#### Motif1: pruning is necessary for scalable auto\-interpretation, and CLTs demand a CLT\-native framework\.

Existing automatic interpretation pipelines largely focus on SAE features or neuron\-like units and typically evaluate uniformly sampled features, which leads to substantial redundancy: many features are weakly causal, noisy, or irrelevant to the downstream behavior of interest\(Billset al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib26); Pauloet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib25); Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55); Boggustet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib50)\)\. This motivates a simple principle:*not all features are worth explaining; prune first, interpret later\.*

At the same time, CLT replacement models represent circuits using cross\-layer feature writes that are not naturally captured by SAE\-only pipelines\(Ameisenet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib31); Lindseyet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib56)\)\. We therefore argue that scalable mechanistic interpretability requires the first CLT\-native auto\-interpretation framework that explicitly connects: pruning→\\rightarrowautomatic interpretation→\\rightarrowinterpretation evaluation\. This paper establishes such a framework, enabling end\-to\-end measurement of how pruning decisions shape both behavior fidelity and downstream explainability metrics\.

#### Motif2: mainstream pruning is insufficient because it ignores synergy and sign\.

A second bottleneck lies in how pruning is performed\. Most mainstream pruning methods are edge\-centric and/or assume approximately additive importance, selecting high\-magnitude contributors and discarding the rest\(Syedet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib49); Hannaet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib74)\)\. However, CLT circuits can exhibit two properties that violate these assumptions: \(i\)*non\-additive interactions*, where components that appear weak in isolation matter jointly \(“synergy”\); and \(ii\)*signed influence*, where both strengthening and weakening contributions are necessary for calibrated behavior\. Such effects are well\-known in causal tracing / patching settings, and we empirically verify their presence in CLT feature space in our ablations \(Appendix[E](https://arxiv.org/html/2604.16889#A5)and Sec\.[4](https://arxiv.org/html/2604.16889#S4)\)\.

Ignoring these effects can systematically under\-estimate important components and produce circuits that are behaviorally degraded or interpretability\-unstable\.

#### Our approach: Feature Attribution Patching \(FAP\) and its extensions\.

To address both motifs, we introduce a CLT\-native pruning family that operates directly on*features*rather than module\-level edges\. Our base method, Feature Attribution Patching \(FAP\), scores CLT feature occurrences using a fast, patch\-grounded first\-order estimate based on cross\-layer write differences dotted with cached downstream gradients \(an attribution patching analogue in CLT feature space\)\. We then propose FAP\-Signal, a*signal\-aware*variant that preserves both strengthening and weakening features by allocating budget separately to positive and negative contributions\. We also study a sign\-aware ablation, FAP\-Signal, which enforces a minimum allocation to both positive\- and negative\-scoring features; details and results are in Appendix[E](https://arxiv.org/html/2604.16889#A5)\.

This paper makes four contributions:

- •CLT\-native prune\-interpret\-eval framework\. We present the first end\-to\-end framework connecting CLT\-native pruning to automatic interpretation and interpretation evaluation, explicitly motivated by the redundancy of SAE\-style interpretation pipelines when applied at scale\.
- •Feature Attribution Patching \(FAP\)\. We propose an efficient feature\-level attribution patching method for CLT circuits that scores sparse cross\-layer feature writes with a single gradient pass, enabling large\-scale pruning\.
- •Synergy\-aware pruning\.We introduceFAP\-Synergy, a boundary interaction reranking method that improves fidelity at fixed budget\. We additionally evaluate a sign\-aware ablation,FAP\-Signal, in Appendix[E](https://arxiv.org/html/2604.16889#A5)\.
- •Comprehensive evaluation protocol\. We evaluate pruning using KL\-divergence behavior retention and assess interpretability impact via interpretation metrics including clarity, purity, and responsiveness\(Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55); Pauloet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib25); Boggustet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib50)\), enabling systematic quantification of how pruning affects explainability\.

## 2Related Work

#### Attribution patching and edge pruning\.

Causal circuit discovery methods often formalize the search space as a computation graph and aim to recover a sparse subgraph sufficient for reproducing a behavior under intervention\. Early work such as ACDC performs iterative edge search using patching\-based causal tests\(Conmyet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib14)\), but this can require many forward passes and becomes expensive at scale\. Attribution Patching\-style methods accelerate this process by computing proxy importance scores using gradients and validating them with patching, substantially reducing search cost while retaining causal grounding\(Syedet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib49)\)\. Our work inherits the patching\-first philosophy but changes the unit of analysis: rather than pruning*module\-to\-module edges*, we prune*CLT feature writes*, which are the natural causal objects in a cross\-layer replacement model\.

#### Feature\-based interpretability and CLT\-native circuit tracing\.

A recurring theme in mechanistic interpretability is that individual neurons are often polysemantic, motivating learned sparse feature bases as more stable semantic units\(Brickenet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib23)\)\. Toy models of superposition and related training phenomena provide a conceptual basis for why feature directions, rather than single neurons, may serve as the correct minimal units for interpretation\(Elhageet al\.,[2022](https://arxiv.org/html/2604.16889#bib.bib65); Henighanet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib66); Poweret al\.,[2022](https://arxiv.org/html/2604.16889#bib.bib67)\)\. Replacement\-model approaches further enable causal validation by allowing controlled interventions on feature activity and information flow\. In particular, CLT\-based circuit tracing represents computation as sparse cross\-layer feature writes and supports end\-to\-end causal experiments in which feature contributions are patched, ablated, or amplified\(Ameisenet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib31); Dunefskyet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib76)\)\. Large\-scale resources such as Neuronpedia’s circuit graphs operationalize these ideas in public tooling and highlight the need for methods that operate directly on CLT\-native objects\(Lindseyet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib56)\)\. Our method is designed explicitly for this setting: we score and prune CLT feature occurrences by aggregating their cross\-layer writes, rather than reducing the problem back to neuron\- or SAE\-only views\.

#### Automated interpretation and evaluation of feature descriptions\.

A complementary line of work aims to reduce human effort by generating interpretations of internal units automatically\. Neuron\- and feature\-explainer systems prompt an LLM with activation traces or exemplar contexts to propose a description of what a unit represents\(Billset al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib26); Pauloet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib25); Lin,[2023](https://arxiv.org/html/2604.16889#bib.bib27)\)\. Output\-centric and behavior\-conditioned approaches push toward hypotheses tied more directly to model outputs, aiming to reduce purely correlational labels\(Gur\-Ariehet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib43)\)\. However, natural\-language interpretations can be verbose, inconsistent, and difficult to falsify experimentally, motivating more structured or executable description formats\(Huanget al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib30)\)\. Recent work proposes semantic regex\-style DSLs that produce compact, checkable descriptions and enable systematic evaluation through detection and fuzzing protocols\(Boggustet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib50)\)\. In parallel, evaluation frameworks such as FADE formalize interpretation quality using metrics like clarity, purity, responsiveness, and faithfulness, including causal validation via patching\(Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55)\)\. While these systems advance feature description quality, they typically assume a large pool of candidate units and do not address the upstream question of*which features should be explained*\.

## 3Method: The PIE Framework

We introducePIE\(Prune, Interpret, Evaluate\), a CLT\-native framework designed to resolve the efficiency bottleneck in mechanistic interpretability\. While SAE features offer semantic sparsity, interpreting millions of features is computationally prohibitive\. Our framework rests on the premise that*pruning must precede interpretation*: by identifying the sparse subset of Cross\-Layer Transcoder \(CLT\) features causally relevant to a task, we can allocate interpretation budgets where they matter most\.

The framework consists of three stages: \(1\)Pruningvia Feature Attribution Patching \(FAP\) and its variants to select a minimal, high\-fidelity circuit; \(2\)Automatic Interpretationof the retained features; and \(3\)Evaluationof both behavioral fidelity and explanation quality\.

### 3\.1Problem Setup: CLT\-Native Features

We analyze a subject modelMMutilizing a Cross\-Layer Transcoder \(CLT\) as a replacement model\. Unlike standard transformers where edges connect modules \(e\.g\., Attention Head→\\toMLP\), a CLT decomposes computation into sparse feature activations that write directly across layers\.

Let a featureffin the CLT dictionary have activationaf​\(x,t\)a\_\{f\}\(x,t\)at token positionttfor inputxx\. This feature contributes a vectorWf\(s\)∈ℝdm​o​d​e​lW^\{\(s\)\}\_\{f\}\\in\\mathbb\{R\}^\{d\_\{model\}\}to a downstream residual stream sitess\. We define the fundamental unit of analysis as thefeature\-write edge: the specific contribution of featureffto sitessat positiontt\. Pruning involves selecting a subset of featuresℱ∗\\mathcal\{F\}^\{\*\}\(where\|ℱ∗\|≪\|ℱt​o​t​a​l\|\|\\mathcal\{F\}^\{\*\}\|\\ll\|\\mathcal\{F\}\_\{total\}\|\) such that the replacement model maintains low KL\-divergence from the original subject modelMM\.

### 3\.2Pruning via Feature Attribution Patching \(FAP\)

To scale pruning to millions of features, we cannot afford iterative ablation \(e\.g\., ACDC\)\. Instead, we proposeFeature Attribution Patching \(FAP\), a first\-order approximation method adapted for CLT feature space\.

#### Gradient\-Weighted Write Attribution\.

We estimate feature importance by combining activation differences with downstream gradients\. Given a clean inputxcleanx\_\{\\text\{clean\}\}and a corrupted inputxcorrx\_\{\\text\{corr\}\}, letΔ​af​\(t\)=af​\(xclean,t\)−af​\(xcorr,t\)\\Delta a\_\{f\}\(t\)=a\_\{f\}\(x\_\{\\text\{clean\}\},t\)\-a\_\{f\}\(x\_\{\\text\{corr\}\},t\)be the activation difference for featureff\. We compute the gradient of a target metricℒ\\mathcal\{L\}\(e\.g\., logit difference or negative KL\) with respect to the receiver site activationhs​\(t\)h\_\{s\}\(t\), denoted as∇hs​\(t\)ℒ\\nabla\_\{h\_\{s\}\(t\)\}\\mathcal\{L\}\. The FAP score for featureffis the dot product of its weighted write vector and the gradient:

SFAP​\(f\)=∑t,s\(Δ​af​\(t\)⋅Wf\(s\)\)⊤​∇hs​\(t\)ℒS\_\{\\text\{FAP\}\}\(f\)=\\sum\_\{t,s\}\\left\(\\Delta a\_\{f\}\(t\)\\cdot W^\{\(s\)\}\_\{f\}\\right\)^\{\\top\}\\nabla\_\{h\_\{s\}\(t\)\}\\mathcal\{L\}\(1\)This score efficiently estimates how much featureffcontributes to restoring the clean behavior from the corrupted state\. We retain the top\-KKfeatures with the highest magnitude scores\|SFAP​\(f\)\|\|S\_\{\\text\{FAP\}\}\(f\)\|\. This acts as a rapid, low\-cost filter to discard relevant noise\.

### 3\.3Addressing Magnitude Failure Modes

While vanilla FAP is efficient, magnitude\-based pruning suffers from two distinct failure modes in CLTs:*synergy ignorance*\(missing features that only matter deeply in combination\) and*sign cancellation*\(discarding balancing inhibitory features\)\. We introduce two specialized variants to address these\.

#### FAP\-Synergy: Boundary Interaction Reranking\.

Magnitude pruning often fails at the ”pruning boundary” where the threshold separating kept and pruned features\. Features just below the cutoff may be individually weak but highly synergistic with features already selected\. FAP\-Synergy addresses this via aboundary rerankingprocedure\. We define a set of ”Core” featuresℱcore\\mathcal\{F\}\_\{\\text\{core\}\}\(highest scores\) and ”Boundary” candidatesℱbound\\mathcal\{F\}\_\{\\text\{bound\}\}\(scores near the cutoff\)\. For a candidatefb∈ℱboundf\_\{b\}\\in\\mathcal\{F\}\_\{\\text\{bound\}\}and a partnerfc∈ℱcoref\_\{c\}\\in\\mathcal\{F\}\_\{\\text\{core\}\}, we estimate pairwise synergy via patching:

Syn​\(fb,fc\)=ℳ​\(\{fb,fc\}\)−ℳ​\(\{fb\}\)−ℳ​\(\{fc\}\)\\text\{Syn\}\(f\_\{b\},f\_\{c\}\)=\\mathcal\{M\}\(\\\{f\_\{b\},f\_\{c\}\\\}\)\-\\mathcal\{M\}\(\\\{f\_\{b\}\\\}\)\-\\mathcal\{M\}\(\\\{f\_\{c\}\\\}\)\(2\)whereℳ\\mathcal\{M\}represents the metric recovery \(e\.g\., logit restoration\) relative to the baseline\. If a boundary feature exhibits strong positive synergy with core features, its score is boosted \(Sf′=Sf\+λ⋅SynS^\{\\prime\}\_\{f\}=S\_\{f\}\+\\lambda\\cdot\\text\{Syn\}\), effectively rescuing high\-information features that vanilla FAP would discard\. This method aims to lower the KL divergence of the pruned circuit without increasing the feature budget\.

We additionally evaluate a sign\-aware budget allocation variant \(FAP\-Signal\) to mitigate sign cancellation; see Appendix[E](https://arxiv.org/html/2604.16889#A5)\.

### 3\.4Automatic Interpretation & Evaluation

Once features are pruned, we generate natural language explanations using an LLM \(e\.g\., GPT\-5\.2\) prompted with max\-activating exemplars\(Billset al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib26)\)\. Crucially, because we interpret only the pruned set, we drastically reduce API costs compared to dense sweeps\. We evaluate the quality of the discovery process using two categories of metrics\.

#### Behavioral Fidelity Metrics\.

We measure how well the pruned feature setℱ∗\\mathcal\{F\}^\{\*\}preserves the original model mechanics:

- •KL Divergence:The KL divergence between the subject model’s output distribution and the replacement model restricted toℱ∗\\mathcal\{F\}^\{\*\}, measured at the last token\. Lower is better\.
- •Prediction Change Rate:The frequency with which the argmax token prediction changes after pruning\.

#### Interpretability Metrics\.

To ensure that FAP selects meaningfully interpretable units \(rather than just polysemantic error\-correcting terms\), we evaluate the generated explanations using the FADE framework metrics\(Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55)\):

- •Clarity:Can an independent auditor LLM generate synthetic samples that activate the feature based*only*on the description? \(Measured via Gini coefficient of activations on synthetic vs\. control data\)\.
- •Purity:When presented with real dataset examples, can the explanation distinguish high\-activating samples from low\-activating ones? \(Measured via Average Precision\)\.
- •Responsiveness:Do natural samples that align with the description consistently trigger the feature? \(Measured via Gini coefficient on rated natural samples\)\.

We benchmark these metrics against a baseline of1000 Random Featuresto quantify the interpretability gain provided by our pruning strategies\.

### 3\.5Practical Efficiency Considerations

FAP is practical at CLT scales: computingSFAPS\_\{\\text\{FAP\}\}uses cached activations plus a small number of backward passes forgs​\(t\)g\_\{s\}\(t\), and is typically far cheaper than iterative edge\-search methods \(e\.g\., ACDC\(Conmyet al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib14)\)\)\. Synergy reranking is limited to a small boundary set to keep pairwise tests tractable, and the sign\-aware variant \(FAP\-Signal; Appendix[E](https://arxiv.org/html/2604.16889#A5)\) adds negligible overhead by splitting the budget by score sign\.

## 4Experiments

We evaluate the PIE framework on the Indirect Object Identification \(IOI\) task\(Wanget al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib72)\)to assess whether FAP\-based pruning preserves behavioral fidelity while yielding features with high intrinsic interpretability\.

### 4\.1Experimental Setup

#### Models & Tasks\.

We apply our framework to two open\-weights models,Gemma\-2\-2B\(Teamet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib69)\)andLlama\-3\.2\-1B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2604.16889#bib.bib70)\), utilizing pre\-trained Cross\-Layer Transcoders \(CLTs\)\(Hannaet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib71)\)\. We evaluate on two tasks:Indirect Object Identification \(IOI\)\(Wanget al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib72)\)andDoc\-String, a structural Python code task used in prior circuit\-discovery work\. We use IOI for the full end\-to\-end interpretability pipeline and use both IOI and Doc\-String for behavioral\-fidelity sweeps across budgets\. Detailed model specifications are provided in Appendix[B](https://arxiv.org/html/2604.16889#A2)\.

#### Pruning Methods and Budgets\.

We compare four feature\-selection methods:

- •FAP:selects top features by magnitude of gradient\-weighted write attribution\.
- •FAP\-Synergy:reranks boundary features using pairwise synergy estimation \(λ=3\\lambda=3, top\-25% boundary\)\. The hyperparameter selection procedure is reported in Appendix[A](https://arxiv.org/html/2604.16889#A1)\.
- •Activation\-Magnitude:ranks features by clean\-prompt activation magnitude\.
- •ACDC\-Style Pruning:a feature\-level adaptation of targeted activation patching that estimates importance via perturbation\.

For behavioral\-fidelity sweeps, we evaluate budgetsK∈\{50,100,200,400,800\}K\\in\\\{50,100,200,400,800\\\}\. For end\-to\-end interpretability evaluation, we use the main budgetK=100K\{=\}100\.

#### Random Active\-Set Comparison\.

To quantify compression, we also compare against random selection from the prompt\-active feature set rather than from the full CLT dictionary\. This baseline is used to measure how many active features are needed to match the fidelity achieved by PIE at small budgets\.

### 4\.2Evaluation Pipeline

To validate PIE, we employ a three\-stage protocol that separates the discovery distribution from the evaluation distribution\. We report the full interpretability pipeline on IOI atK=100K\{=\}100, and report behavioral\-fidelity sweeps across two tasks and multiple budgets to characterize pruning performance across regimes\.

1\. Pruning \(Task\-Specific\)\.We prune the CLT on the IOI dataset\. We measure behavioral fidelity viaKL DivergenceandPrediction Change Rate\(PCR\) on the IOI validation set to quantify how well the sparse circuit reproduces the original model’s task performance\.

2\. Interpretation \(Feature\-Centric\)\.For the retained features, we generate natural language explanations using the Max\-Act protocol\(Billset al\.,[2023](https://arxiv.org/html/2604.16889#bib.bib26)\)\. We utilize the pre\-computed activation history from Circuit Tracer to retrieve max\-activating exemplars efficiently\.

3\. Evaluation \(Generalization\)\.We evaluate the explanation quality \(Clarity, Purity, Responsiveness\) on a held\-out dataset of 2 million Wikipedia sentences\(Sentence Transformers,[2024](https://arxiv.org/html/2604.16889#bib.bib73)\)\. Crucially, evaluating on Wikipedia rather than IOI or Circuit Tracer pre\-computed activation history ensures that the explanations capture the features’ general semantics, not just their task\-specific utility\. Implementation details for the scoring LLMs and sample sizes are detailed in Appendix[B](https://arxiv.org/html/2604.16889#A2)\.

## 5Results

We evaluate PIE on Llama\-3\.2\-1B and Gemma\-2\-2B using the IOI task\. We compare FAP variants \(K=100K=100\) against a baseline of randomly sampled features\.

### 5\.1The Signal\-to\-Noise Interpretability Gap

A primary hypothesis of this work is that task\-relevant features are inherently more interpretable than the general population\. We quantify this by measuring the distribution of interpretation quality metrics \(Clarity, Purity, Responsiveness\)\.

#### Reduction of Uninformative Units\.

We formally define a feature asuninformativeor ”noisy” if it fails to meet a minimal quality threshold in either description generation or validation\. Specifically, a feature is classified as uninformative if its Responsiveness<0\.5<0\.5AND Purity<0\.5<0\.5\.

As shown in Table[1](https://arxiv.org/html/2604.16889#S5.T1), random sampling yields a high rate of such low\-quality features \(46\.7% for Llama\)\. FAP\-based pruning significantly filters this noise, reducing the uninformative rate by approximately9% absolute\(Llama\) and3\.5% absolute\(Gemma\)\. This confirms that PIE does not just select for magnitude; it systematically selects for semantic distinctness\.

Table 1:Interpretability quality of FAP\-pruned circuits \(K=100K=100\) vs\. Random baseline\. We report Mean±\\pmStd for quality metrics\.Uninformative Ratedenotes the percentage of features where Responsiveness<0\.5<0\.5and Purity<0\.5<0\.5\. FAP\-Synergy consistently maintains high interpretability while minimizing uninformative units\.MethodClarity\(↑\\uparrow\)Purity\(↑\\uparrow\)Responsiveness\(↑\\uparrow\)Uninformative Rate\(↓\\downarrow\)Llama\-3\.2\-1BRandom \(1k\)0\.585±0\.3200\.585\\pm 0\.3200\.326±0\.3410\.326\\pm 0\.3410\.455±0\.3640\.455\\pm 0\.36446\.70%FAP \(Base\)0\.627±0\.0320\.627\\pm 0\.0320\.477±0\.0420\.477\\pm 0\.0420\.465±0\.0410\.465\\pm 0\.04137\.98%FAP\-Synergy0\.627±0\.032\\mathbf\{0\.627\}\\pm 0\.0320\.477±0\.042\\mathbf\{0\.477\}\\pm 0\.0420\.465±0\.041\\mathbf\{0\.465\}\\pm 0\.04137\.95%Gemma\-2\-2BRandom \(1k\)0\.606±0\.3170\.606\\pm 0\.3170\.360±0\.2210\.360\\pm 0\.2210\.475±0\.2550\.475\\pm 0\.25537\.60%FAP \(Base\)0\.635±0\.0360\.635\\pm 0\.0360\.524±0\.0370\.524\\pm 0\.0370\.534±0\.0410\.534\\pm 0\.04134\.05%FAP\-Synergy0\.635±0\.0360\.635\\pm 0\.0360\.524±0\.037\\mathbf\{0\.524\}\\pm 0\.0370\.535±0\.041\\mathbf\{0\.535\}\\pm 0\.04134\.04%We report correlations and distributions among Clarity, Purity, and Responsiveness in Appendix[D](https://arxiv.org/html/2604.16889#A4)\.

### 5\.2Behavioral Fidelity and Pruning Efficiency

We evaluate pruning quality across two tasks, five budgets, and four feature\-selection methods\. Table[2](https://arxiv.org/html/2604.16889#S5.T2)reports mean last\-token KL across IOI and Doc\-String for Llama\-3\.2\-1B and Gemma\-2\-2B\. The main pattern is consistent across settings: the FAP family achieves the lowest KL across most regimes, while FAP\-Synergy is most helpful when the feature budget is tight\.

Table 2:Behavioral fidelity across tasks and budgets\.We report mean last\-token KL \(lower is better\)\. Across both IOI and Doc\-String, the FAP family consistently outperforms Activation\-Magnitude pruning\. ACDC\-style pruning is competitive in the strictest budget regime but degrades at largerKK, while FAP\-Synergy provides its clearest gains at low budgets and converges with base FAP asKKincreases\.#### Method comparison\.

Activation\-Magnitude pruning is consistently weaker than the FAP family across both tasks and all budgets, indicating that clean activation size alone is not sufficient to recover the causal core\. ACDC\-style pruning performs well only in the smallest\-budget regime, but then plateaus asKKincreases, suggesting that single\-feature perturbation captures the top few highly salient features but does not rank the long tail of medium\-importance features well in redundant CLT circuits\.

#### Budget dependence of FAP\-Synergy\.

FAP\-Synergy is most useful in low\-budget regimes \(K=50,100K\{=\}50,100\), where selection near the pruning boundary is most competitive\. For example, on IOI, FAP\-Synergy reduces KL from 1\.33 to 1\.22 on Llama\-3\.2\-1B and from 0\.91 to 0\.82 on Gemma\-2\-2B atK=50K\{=\}50\. As the budget increases, the feature ranking stabilizes and the difference between FAP and FAP\-Synergy becomes negligible\. This suggests a simple practical guideline: FAP\-Synergy is most valuable when the extracted circuit must remain very sparse, while FAP offers a simpler alternative when larger budgets are acceptable\.

#### Compression relative to active\-feature random selection\.

Figure[2](https://arxiv.org/html/2604.16889#S5.F2)compares PIE against random selection from the prompt\-active feature set\. This comparison answers a different question from Table[2](https://arxiv.org/html/2604.16889#S5.T2): rather than comparing pruning methods directly, it measures how many active features are needed for random selection to match the fidelity achieved by PIE atK=100K\{=\}100\. In both models, PIE reaches a fidelity level that random active\-set selection only achieves at roughly4​k4\\text\{k\}features, yielding an approximate40×40\\timescompression\.

02,0002\{,\}0004,0004\{,\}000055101015152020Number of Features Kept \(KK\)KL Divergence \(nats\)Gemma RandomGemma FAP \(K=100K\{=\}100\)Llama RandomLlama FAP \(K=100K\{=\}100\)Figure 2:Fidelity Efficiency Gap\.We plot the KL divergence of the Random baseline as the budgetKKincreases \(solid lines\)\. The dashed lines represent the KL achieved by FAP using onlyK=100K\{=\}100features\. FAP achieves with 100 features what Random selection achieves with≈\\approx4,000 features, demonstrating a compression factor of≈\\approx40x on the active feature set\.

### 5\.3Case Study: Rescuing Synergistic Components

To illustrate the mechanism of FAP\-Synergy, we analyze a specific ”boundary” feature in Llama\-3\.2\-1B that would have been discarded by standard magnitude pruning but was successfully rescued by our interaction\-aware reranking\.

#### The Hidden Architect: Orthography meets Semantics\.

We examineFeature L0\.2703, interpreted as a ”J\-initial token detector” \(activates on “J”, “Java”, “Jacob”\)\. Under base FAP, this feature fell slightly below the Top\-KKthreshold \(SFAPS\_\{\\text\{FAP\}\}magnitude ranking\) and was slated for removal\. However, FAP\-Synergy identified strong pairwise interactions with retained ”Core” features, boosting its score to safe retention\.

#### Positive Synergy: Constructive Amplification\.

Our analysis shows that L0\.2703 provides crucial upstream support for higher\-level semantic features\. In particular, we identifyL5\.8201\(“Given Name”detector\) as the8th strongest positive\-synergy partnerof L0\.2703 \(Synergy≈\+0\.156\\approx\+0\.156\): the model appears to use the low\-level orthographic cue \(“starts with J”\) from Layer 0 to amplify the confidence of the mid\-level semantic signal \(“is a name”\) at Layer 5\. As a result, when both features are present, the model’s handling of names like‘‘Jacob’’exceeds what either feature achieves alone, whereas pruning the L0 trigger makes the L5 detector less reliable on this subset of names\.

#### Negative Synergy: Managing Redundancy\.

Conversely, the framework also identifiedFeature L0\.5905\(specific ”Jacob” detector\) as the1st highest negative synergy partner\(S​y​n​e​r​g​y≈−0\.375Synergy\\approx\-0\.375\)\. This strong negative interaction indicates redundancy: the specific‘‘Jacob’’feature and the general‘‘J\-initial’’feature likely encode overlapping evidence for the token‘‘Jacob’’\. From an interpretability standpoint, this suggests either that L0\.5905 is an overfitted backup of L0\.2703 introduced during CLT reconstruction, or that activating L0\.5905 creates a shortcut that suppresses L0\.2703’s contribution in the Jacob context\. By rescuing L0\.2703, PIE preserves the circuit’scompositionalstructure \(Orthography→\\rightarrowSemantics\), rather than yielding a disjoint set of isolated semantic detectors\.

## 6Discussion

#### PIE makes CLT interpretability*budgetable*\.

A central obstacle in end\-to\-end circuit analysis is not discovering candidate units, but deciding*which*units are worth spending scarce interpretation budget on\. PIE reframes this as a*budgeted pipeline*: \(i\) prune a large candidate set down to a small subset that preserves behavior, \(ii\) spend expensive auto\-interpretation and evaluation only on that subset, and \(iii\) report both behavioral fidelity and interpretability outcomes under a fixed budget\. This is practically important because explanation and evaluation are the dominant cost in modern interpretability workflows \(LLM\-based description, counterfactual prompting, and FADE\-style scoring\)\(Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55); Boggustet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib50)\)\. By turning “interpret everything” into “interpret the*right*KK,” PIE enables systematic comparisons across methods and models under a consistent interpretation budget\.

#### Why the∼\\sim40×\\timescompression gap matters\.

Even when random baselines sample from the active feature set, they require thousands of features to match the behavioral preservation achieved by PIE withK=100K\{=\}100\. This gap indicates that the behaviorally relevant computation is concentrated in a much smaller*causal core*, while many active features are redundant, weakly coupled, or act as backups that do not materially affect the target behavior\. From a systems perspective, this compression is the primary driver of feasibility: reducing the interpretation target from𝒪​\(103−104\)\\mathcal\{O\}\(10^\{3\}\\\!\\\!\-\\\!\\\!10^\{4\}\)to𝒪​\(102\)\\mathcal\{O\}\(10^\{2\}\)features directly translates to large savings in LLM calls, tokens, and human inspection time\. From a scientific perspective, it sharpens mechanistic hypotheses: if the circuit can be summarized by a small set of features, we can more plausibly map them to functional roles and test those roles causally\.

#### The relatively “small” KL win from synergy reranking\.

Synergy\-aware reranking yields modest but consistent improvements in behavioral fidelity relative to magnitude\-only pruning\. It should be noted that this regime is intentionally*boundary\-local*: reranking only perturbs the final selection among near\-threshold candidates, so large KL swings are neither expected nor desirable\. The value of synergy reranking is therefore not solely in absolute KL reduction, but in improving the*interpretability–fidelity tradeoff*: among similarly faithful pruned sets, synergy tends to favor feature combinations that better preserve coordinated computation \(e\.g\., complementary roles rather than redundant backups\)\. This also motivates synergy diagnostics \(e\.g\., sign\- and interaction\-aware statistics\) as an analysis tool: they offer a handle on when two features jointly matter beyond their marginal contributions, which magnitude\-based criteria systematically ignore\.

#### Mechanistic lessons from the case study\.

The L0\.2703 case study illustrates how PIE supports*mechanistic*claims rather than only aggregate metrics\. First, it highlights that circuits can be compositional across layers: features may play distinct roles \(e\.g\., orthographic or structural cues upstream, semantic selection downstream\) that must be preserved jointly to maintain behavior\. Second, it exposes failure modes of magnitude\-only selection: highly active features can function as “backup” or “shortcut” pathways that partially compensate for other features but obscure the true causal decomposition\. By surfacing both positive and negative interaction partners, the synergy view provides qualitative evidence that the retained set is not merely a collection of strong individual detectors, but a coordinated subset that better matches the circuit’s functional structure\. These case studies are thus best read as*mechanistic validation*that complements the global compression and fidelity results\.

#### Synergy may be under\-measured by “feature\-isolated” evaluation\.

The case study suggests that synergy\-aware reranking’s effect is*structural*: it preferentially retains*complementary*features whose joint presence preserves coordinated computation, rather than substitutable “backup” features that match behavior only marginally\. In other words, synergy helps preserve*interactions*, in which the way features compose into a circuit, even when their marginal, feature\-by\-feature contributions look similar at the pruning boundary\. This distinction may be weakly expressed in our present evaluation setting because our interpretability pipeline largely treats features as*isolated*units \(explained and scored independently\), which is not fully “circuit\-friendly” and can under\-reward interaction\-preserving selections\. The mechanistic lessons therefore motivate an important direction for future work: developing a*CLT\-native*evaluation pipeline that explicitly evaluates*sets*of features and their composition \(e\.g\., interaction\-aware interventions, group\-level counterfactuals, and structure\-sensitive explanation scoring\)\. Such an evaluation would better capture the value of synergy\-preserving selection and enable a clearer understanding of when and why synergy improves circuit faithfulness beyond what marginal metrics reveal\.

## 7Conclusion

We introduced PIE, the first end\-to\-end CLT\-native interpretability framework that makes mechanistic analysis*budgetable*by explicitly connecting feature pruning, automatic interpretation, and FADE\-style evaluation under a fixed interpretation budget\. At the core of PIE is Feature Attribution Patching \(FAP\), a patch\-grounded, gradient\-weighted write attribution method that rapidly filters CLT feature space, together with FAP\-Synergy, which improves boundary selection by reranking near\-threshold features using pairwise interaction signals\.

Empirically, on IOI with CLTs for Llama\-3\.2\-1B and Gemma\-2\-2B, PIE isolates a sparse “causal core”: retaining onlyK=100K\{=\}100features achieves behavioral fidelity that a strong random baseline requires*thousands*of active features to match, yielding an approximate∼\\sim40×\\timescompression even within the active feature set and correspondingly large reductions in downstream interpretation/evaluation calls\. Across broader sweeps on both IOI and Doc\-String, and against stronger baselines including Activation\-Magnitude and ACDC\-style pruning, the FAP family remains consistently more robust, while FAP\-Synergy provides its clearest gains in strict\-budget regimes\. Looking forward, these results motivate morecircuit\-awareevaluation protocols that score*sets*of interacting features \(rather than isolated units\), as well as broader scaling studies across tasks, budgets, and CLT training regimes\.

## Impact Statement

Our work aims to make mechanistic interpretability scalable and economically viable\. By enabling the pruning of Cross\-Layer Transcoder \(CLT\) features, the PIE framework significantly reduces the computational and financial costs associated with automated model interpretation\. This advances the goal of AI transparency, making it feasible to audit large models for safety\-critical behaviors without analyzing millions of redundant parameters\. Furthermore, by lowering the resource requirements for circuit discovery, this work promotes inclusivity in the research community, allowing entities with smaller compute budgets to participate in safety research\. However, we acknowledge the risk of over\-reliance on pruned circuits; aggressive pruning might obscure subtle, distributed computations necessary for full behavioral robustness\. Researchers should treat pruned interpretations as high\-signal approximations rather than exhaustive descriptions of model psychology\.

## References

- E\. Ameisen, J\. Lindsey, A\. Pearce, W\. Gurnee, N\. L\. Turner, B\. Chen, C\. Citro, D\. Abrahams, S\. Carter, B\. Hosmer, J\. Marcus, M\. Sklar, A\. Templeton, T\. Bricken, C\. McDougall, H\. Cunningham, T\. Henighan, A\. Jermyn, A\. Jones, A\. Persic, Z\. Qi, T\. B\. Thompson, S\. Zimmerman, K\. Rivoire, T\. Conerly, C\. Olah, and J\. Batson \(2025\)Circuit tracing: revealing computational graphs in language models\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2025/attribution-graphs/methods.html)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p2.2),[§1](https://arxiv.org/html/2604.16889#S1.p1.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- S\. S\. Band, A\. Yarahmadi, C\. Hsu, M\. Biyari, M\. Sookhak, R\. Ameri, I\. Dehzangi, A\. T\. Chronopoulos, and H\. Liang \(2023\)Application of explainable artificial intelligence in medical health: a systematic review of interpretability methods\.Informatics in Medicine Unlocked40,pp\. 101286\.External Links:[Document](https://dx.doi.org/10.1016/j.imu.2023.101286),[Link](https://doi.org/10.1016/j.imu.2023.101286)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1)\.
- S\. Bills, N\. Cammarata, D\. Mossing, H\. Tillman, L\. Gao, G\. Goh, I\. Sutskever, J\. Leike, J\. Wu, and W\. Saunders \(2023\)Language models can explain neurons in language models\.Note:OpenAI \(project page\)External Links:[Link](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.p2.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2604.16889#S3.SS4.p1.1),[§4\.2](https://arxiv.org/html/2604.16889#S4.SS2.p3.1)\.
- A\. Boggust, D\. Ren, Y\. Assogba, D\. Moritz, A\. Satyanarayan, and F\. Hohman \(2025\)Semantic regexes: auto\-interpreting llm features with a structured language\.CoRRabs/2510\.06378\.External Links:2510\.06378,[Link](https://arxiv.org/abs/2510.06378)Cited by:[4th item](https://arxiv.org/html/2604.16889#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.p2.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2604.16889#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.htmlCited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Chen, X\. Wang, Z\. Yao, Y\. Bai, L\. Hou, and J\. Li \(2025\)Towards understanding safety alignment: a mechanistic perspective from safety neurons\.Note:arXiv preprintarXiv:2406\.14144External Links:[Link](https://arxiv.org/abs/2406.14144)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1)\.
- A\. Conmy, A\. N\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-Alonso \(2023\)Towards automated circuit discovery for mechanistic interpretability\.InNeurIPS 2023,External Links:2304\.14997,[Link](https://arxiv.org/abs/2304.14997)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px1.p1.1),[§3\.5](https://arxiv.org/html/2604.16889#S3.SS5.p1.2)\.
- J\. Dunefsky, P\. Chlenski, and N\. Nanda \(2024\)Transcoders find interpretable LLM feature circuits\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=J6zHcScAo0)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen, R\. Grosse, S\. McCandlish, J\. Kaplan, D\. Amodei, M\. Wattenberg, and C\. Olah \(2022\)Toy models of superposition\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2022/toy\_model/index\.html](https://transformer-circuits.pub/2022/toy_model/index.html)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2604.16889#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Gur\-Arieh, R\. Mayan, C\. Agassy, A\. Geiger, and M\. Geva \(2025\)Enhancing automated interpretability with output\-centric feature descriptions\.InProceedings of the Annual Meeting of the ACL,pp\. 5757–5778\.External Links:[Link](https://aclanthology.org/2025.acl-long.288/)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Hanna, S\. Pezzelle, and Y\. Belinkov \(2024\)Have faith in faithfulness: going beyond circuit overlap when finding model mechanisms\.External Links:2403\.17806,[Link](https://arxiv.org/abs/2403.17806)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px2.p1.1)\.
- M\. Hanna, M\. Piotrowski, J\. Lindsey, and E\. Ameisen \(2025\)Circuit\-tracer\.Note:[https://github\.com/safety\-research/circuit\-tracer](https://github.com/safety-research/circuit-tracer)Cited by:[§4\.1](https://arxiv.org/html/2604.16889#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Henighan, S\. Carter, T\. Hume, N\. Elhage, R\. Lasenby, S\. Fort, N\. Schiefer, and C\. Olah \(2023\)Superposition, memorization, and double descent\.Note:Transformer Circuits ThreadAccessed 2025\-12\-14External Links:[Link](https://transformer-circuits.pub/2023/toy-double-descent/index.html)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Huang, A\. Geiger, K\. D’Oosterlinck, Z\. Wu, and C\. Potts \(2023\)Rigorously assessing natural language explanations of neurons\.InBlackboxNLP Workshop @ EMNLP,pp\. 317–331\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.blackboxnlp-1.24),[Link](https://doi.org/10.18653/v1/2023.blackboxnlp-1.24)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Lin \(2023\)Neuronpedia: interactive reference and tooling for analyzing neural networks\.Note:WebsiteExternal Links:[Link](https://www.neuronpedia.org/)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Lindsey, E\. Ameisen, N\. Nanda, S\. Shabalin, M\. Piotrowski, T\. McGrath, M\. Hanna, O\. Lewis, C\. Tigges, J\. Merullo, C\. Watts, G\. Paulo, J\. Batson, L\. Gorton, E\. Simon, M\. Loeffler, C\. McDougall, and J\. Lin \(2025\)The circuits research landscape: results and perspectives\.Neuronpedia\.External Links:[Link](https://neuronpedia.org/graph/info)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p2.2),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Olah \(2022\)Mechanistic interpretability, variables, and the importance of interpretable bases\.Note:Transformer Circuits essayExternal Links:[Link](https://www.transformer-circuits.pub/2022/mech-interp-essay)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1)\.
- OpenAI \(2026\)API pricing\.Note:[https://openai\.com/api/pricing/](https://openai.com/api/pricing/)Accessed: 2026\-01\-28Cited by:[Appendix F](https://arxiv.org/html/2604.16889#A6.p1.4)\.
- G\. Paulo, A\. Mallen, C\. Juang, and N\. Belrose \(2024\)Automatically interpreting millions of features in large language models\.CoRRabs/2410\.13928\.External Links:2410\.13928,[Link](https://doi.org/10.48550/arXiv.2410.13928),[Document](https://dx.doi.org/10.48550/arXiv.2410.13928)Cited by:[4th item](https://arxiv.org/html/2604.16889#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.p2.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Power, Y\. Burda, H\. Edwards, I\. Babuschkin, and V\. Misra \(2022\)Grokking: generalization beyond overfitting on small algorithmic datasets\.External Links:2201\.02177,[Link](https://arxiv.org/abs/2201.02177)Cited by:[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Puri, A\. Jain, E\. Golimblevskaia, P\. Kahardipraja, T\. Wiegand, W\. Samek, and S\. Lapuschkin \(2025\)FADE: why bad descriptions happen to good features\.External Links:2502\.16994,[Link](https://arxiv.org/abs/2502.16994)Cited by:[Appendix D](https://arxiv.org/html/2604.16889#A4.SS0.SSS0.Px2.p1.2),[4th item](https://arxiv.org/html/2604.16889#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.p2.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2604.16889#S3.SS4.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2604.16889#S6.SS0.SSS0.Px1.p1.1)\.
- Sentence Transformers \(2024\)Cited by:[1st item](https://arxiv.org/html/2604.16889#A2.I3.i1.p1.1),[§4\.2](https://arxiv.org/html/2604.16889#S4.SS2.p4.1)\.
- A\. Syed, C\. Rager, and A\. Conmy \(2024\)Attribution patching outperforms automated circuit discovery\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 407–416\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.25/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.25)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2604.16889#S1.p1.1),[§2](https://arxiv.org/html/2604.16889#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé, J\. Ferret, P\. Liu, P\. Tafti, A\. Friesen, M\. Casbon, S\. Ramos, R\. Kumar, C\. L\. Lan, S\. Jerome, A\. Tsitsulin, N\. Vieillard, P\. Stanczyk, S\. Girgin, N\. Momchev, M\. Hoffman, S\. Thakoor, J\. Grill, B\. Neyshabur, O\. Bachem, A\. Walton, A\. Severyn, A\. Parrish, A\. Ahmad, A\. Hutchison, A\. Abdagic, A\. Carl, A\. Shen, A\. Brock, A\. Coenen, A\. Laforge, A\. Paterson, B\. Bastian, B\. Piot, B\. Wu, B\. Royal, C\. Chen, C\. Kumar, C\. Perry, C\. Welty, C\. A\. Choquette\-Choo, D\. Sinopalnikov, D\. Weinberger, D\. Vijaykumar, D\. Rogozińska, D\. Herbison, E\. Bandy, E\. Wang, E\. Noland, E\. Moreira, E\. Senter, E\. Eltyshev, F\. Visin, G\. Rasskin, G\. Wei, G\. Cameron, G\. Martins, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Batra, H\. Dhand, I\. Nardini, J\. Mein, J\. Zhou, J\. Svensson, J\. Stanway, J\. Chan, J\. P\. Zhou, J\. Carrasqueira, J\. Iljazi, J\. Becker, J\. Fernandez, J\. van Amersfoort, J\. Gordon, J\. Lipschultz, J\. Newlan, J\. Ji, K\. Mohamed, K\. Badola, K\. Black, K\. Millican, K\. McDonell, K\. Nguyen, K\. Sodhia, K\. Greene, L\. L\. Sjoesund, L\. Usui, L\. Sifre, L\. Heuermann, L\. Lago, L\. McNealus, L\. B\. Soares, L\. Kilpatrick, L\. Dixon, L\. Martins, M\. Reid, M\. Singh, M\. Iverson, M\. Görner, M\. Velloso, M\. Wirth, M\. Davidow, M\. Miller, M\. Rahtz, M\. Watson, M\. Risdal, M\. Kazemi, M\. Moynihan, M\. Zhang, M\. Kahng, M\. Park, M\. Rahman, M\. Khatwani, N\. Dao, N\. Bardoliwalla, N\. Devanathan, N\. Dumai, N\. Chauhan, O\. Wahltinez, P\. Botarda, P\. Barnes, P\. Barham, P\. Michel, P\. Jin, P\. Georgiev, P\. Culliton, P\. Kuppala, R\. Comanescu, R\. Merhej, R\. Jana, R\. A\. Rokni, R\. Agarwal, R\. Mullins, S\. Saadat, S\. M\. Carthy, S\. Cogan, S\. Perrin, S\. M\. R\. Arnold, S\. Krause, S\. Dai, S\. Garg, S\. Sheth, S\. Ronstrom, S\. Chan, T\. Jordan, T\. Yu, T\. Eccles, T\. Hennigan, T\. Kocisky, T\. Doshi, V\. Jain, V\. Yadav, V\. Meshram, V\. Dharmadhikari, W\. Barkley, W\. Wei, W\. Ye, W\. Han, W\. Kwon, X\. Xu, Z\. Shen, Z\. Gong, Z\. Wei, V\. Cotruta, P\. Kirk, A\. Rao, M\. Giang, L\. Peran, T\. Warkentin, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, D\. Sculley, J\. Banks, A\. Dragan, S\. Petrov, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, S\. Borgeaud, N\. Fiedel, A\. Joulin, K\. Kenealy, R\. Dadashi, and A\. Andreev \(2024\)Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[§4\.1](https://arxiv.org/html/2604.16889#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones, H\. Cunningham, N\. L\. Turner, C\. McDougall, M\. MacDiarmid, C\. D\. Freeman, T\. R\. Sumers, E\. Rees, J\. Batson, A\. Jermyn, S\. Carter, C\. Olah, and T\. Henighan \(2024\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1)\.
- K\. R\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt \(2023\)Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NpsVSN6o4ul)Cited by:[§B\.1](https://arxiv.org/html/2604.16889#A2.SS1.p1.2),[§4\.1](https://arxiv.org/html/2604.16889#S4.SS1.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2604.16889#S4.p1.1)\.
- G\. Yang, Q\. Ye, and J\. Xia \(2022\)Unbox the black\-box for the medical explainable ai via multi\-modal and multi\-centre data fusion: a mini\-review, two showcases and beyond\.Information Fusion77,pp\. 29–52\.External Links:ISSN 1566\-2535,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.inffus.2021.07.016),[Link](https://www.sciencedirect.com/science/article/pii/S1566253521001597)Cited by:[§1](https://arxiv.org/html/2604.16889#S1.p1.1)\.

## Appendix AHyperparameter Selection for FAP\-Synergy

#### Baseline\.

We use the ordinary FAP setting \(synergy weightλ=0\\lambda=0\) as the baseline on IOI withN=500N\{=\}500,K=100K\{=\}100\. The baseline metrics are: mean last\-token KL=1\.1399=1\.1399, std=0\.5393=0\.5393, prediction\-change rate=0\.446=0\.446\. All results below are reported as*deltas relative to this baseline*\.

#### Selection rule\.

We select hyperparameters by*minimizing mean last\-token KL*\. Even sub\-percent improvements are meaningful in interpretability pruning studies, where improvements are typically incremental yet consistent across large prompt sets\.

### A\.1Sensitivity Analysis

We conducted a comprehensive sweep over the synergy weightλ∈\{1,…,5\}\\lambda\\in\\\{1,\\dots,5\\\}and boundary percentb​p∈\{20,…,45\}bp\\in\\\{20,\\dots,45\\\}to identify the optimal configuration for FAP\-synergy\. The results are summarized in Table[3](https://arxiv.org/html/2604.16889#A1.T3)and visualized in Figure[3](https://arxiv.org/html/2604.16889#A1.F3)\.

As shown in the table, the global optimum \(lowestΔ\\Deltamean KL\) is achieved atλ=3\\lambda=3with a boundary percent ofb​p=25bp=25\. The data reveals a consistent trend across all testedλ\\lambdavalues: increasing the boundary percent beyondb​p=25bp=25degrades performance \(e\.g\., atb​p=40bp=40, the improvement drops significantly to≈−0\.758\\approx\-0\.758milli\-KL compared to\>−1\.0\>\-1\.0atb​p=25bp=25\)\. This suggests that widening the reranking window too far introduces noise or dilutes the high\-synergy pairs with less relevant features\. Furthermore, increasingλ\\lambdato 4 or 5 yields nearly identical results toλ=3\\lambda=3at the optimalb​pbp, but does not surpass it\. Consequently, we selectλ=3,b​p=25\\lambda=3,bp=25as the most robust configuration\.

### A\.2Delta table \(vs\. baseline\)

Table 3:Sweep results reported as deltas vs\. theλ=0\\lambda=0baseline \(negative is better\)\.Δ​KL\\Delta\\mathrm\{KL\}andΔ​std\\Delta\\mathrm\{std\}are shown in*milli\-KL*\(i\.e\.,×103\\times 10^\{3\}\) for readability\. The selected setting isλ=3\\lambda=3, bp=25=25\.
### A\.3Scatter plot \(delta vs\. baseline\)

202025253030353540404545−1\.2\-1\.2−1\-1−0\.8\-0\.8−0\.6\-0\.6−0\.4\-0\.4−0\.2\-0\.200\.20\.2boundary percent \(bp\)Δ\\Deltamean last\-token KL \(×103\\times 10^\{3\}\)baseline \(λ=0\\lambda=0\)λ=1\\lambda=1λ=2\\lambda=2λ=3\\lambda=3λ=4\\lambda=4λ=5\\lambda=5selectedFigure 3:Sweep scatter plot reported asΔ\\Deltamean KL vs\. baseline \(λ=0\\lambda=0\)\.

## Appendix BImplementation Details

### B\.1Models and Resources

We utilize the following public checkpoints:

- •Gemma\-2\-2B CLT:mntss/clt\-gemma\-2\-2b\-426k\(Dictionary size: 426k\)\.
- •Llama\-3\.2\-1B CLT:mntss/clt\-llama\-3\.2\-1b\-524k\(Dictionary size: 524k\)\.

The IOI dataset is generated using the template structure fromWanget al\.\([2023](https://arxiv.org/html/2604.16889#bib.bib72)\), consisting of 2000 unique prompts with varying names and objects\.

### B\.2Interpretation and Evaluation Protocol

#### Step 2: Interpretation Generation\.

We usegpt\-5\.2as the explainer model\.

- •Exemplars:We provide 40 max\-activating examples per feature, retrieved from the Circuit Tracer activation cache\.
- •Highlighting:Tokens are highlighted if their activation value exceeds 65% of the max activation in the sequence\.

#### Step 3: FADE Evaluation\.

We usegpt\-5\-minias the auditor model to compute the metrics defined in Section[3\.4](https://arxiv.org/html/2604.16889#S3.SS4)\.

- •Evaluation Dataset:We draw samples from the\(Sentence Transformers,[2024](https://arxiv.org/html/2604.16889#bib.bib73)\)dataset\.
- •Clarity:The auditor generates 15 synthetic positive and negative examples based solely on the description\. We measure the Gini coefficient of the feature’s activations on these synthetic batches\.
- •Purity and Responsiveness:We evaluate the feature on a retrieved set ofNe​v​a​l=250N\_\{eval\}=250real examples from Wikipedia \(distinct from the exemplar set\)\.

## Appendix CPruning Dynamics and Budget Effects

#### ACDC\-style pruning behavior\.

In our feature\-level ACDC adaptation, single\-feature perturbations are informative only for a relatively small set of top features\. We observe a low average number of non\-zero ACDC effects per prompt \(185\.18 for Gemma\-2\-2B and 254\.54 for Llama\-3\.2\-1B\), after which the ranking effectively degrades toward random selection\. This is consistent with CLT redundancy and compensatory behavior under single\-feature ablations\. FAP is also substantially cheaper computationally, requiring one forward/backward pass rather than more than4,0004\{,\}000forward passes per prompt\.

#### Cost\-equivalent view of FAP\-Synergy\.

Although the absolute KL differences between FAP and FAP\-Synergy can appear small, they are meaningful in strict\-budget regimes where interpretation cost scales linearly with the number of retained features\. On IOI, base FAP atK=75K\{=\}75yields a KL of 1\.22 on Llama\-3\.2\-1B and 0\.81 on Gemma\-2\-2B, while FAP\-Synergy atK=50K\{=\}50achieves 1\.22 and 0\.82 respectively\. Thus, in low\-budget settings, FAP\-Synergy can match the fidelity of a larger base\-FAP circuit while reducing the downstream interpretation budget by roughly 33%\.

#### Failure mode under over\-pruning\.

Very small budgets can remove features that carry important structural information even when they reduce computational cost\. One illustrative case isFeature L4\.16331, which functions as a conjunction detector linking two coordinated noun phrases, most often two person names\. For the prompt,“Then, Arthur and Ruby had a long argument, and afterwards Ruby said to Arthur,”this feature activates on the“and”betweenArthurandRuby, providing a grammatical anchor for the coordinated\-name structure\. AtK=100K\{=\}100, both FAP and FAP\-Synergy retain this feature, preserving this structural cue in the extracted circuit\. AtK=50K\{=\}50, however, the feature is dropped entirely\. The resulting circuit is still cheaper, but it loses a meaningful piece of grammatical structure, weakening the structural interpretability of the extracted mechanism\. This example highlights a central risk of aggressive pruning: lower budgets may preserve coarse behavioral fidelity while discarding features that contribute important compositional or syntactic roles\.

## Appendix DMetric Correlations and Distributions

In this section, we analyze the distribution of feature quality scores and the relationship between behavioral fidelity and interpretability\.

#### Signal\-to\-Noise Gap\.

As shown in Figure[6](https://arxiv.org/html/2604.16889#A4.F6), there is a distinct ”signal\-to\-noise” gap between random sampling and our method\. Random features cluster near zero across all metrics, whereas FAP variants consistently select features in the high\-interpretability regime\.

#### Feature Correlations\.

We investigate the relationship between Clarity, Purity, and Responsiveness within the FAP\-selected set\. Our analysis reveals a robust positive correlation between Responsiveness and Purity, with Pearsonr≈0\.71r\\approx 0\.71for Gemma\-2\-2B andr≈0\.73r\\approx 0\.73for Llama\-3\.2\-1B \(Figure[4](https://arxiv.org/html/2604.16889#A4.F4)\)\. This strong correlation suggests that PIE selects features with jointly high Purity and Responsiveness, i\.e\., concept\-specific activations that remain stable across natural data; FADE notes that when descriptions/features are well aligned, interpretability metrics tend to rise together\(Puriet al\.,[2025](https://arxiv.org/html/2604.16889#bib.bib55)\)\.

#### Prompt Fidelity vs\. Interpretability\.

We further ask whether ”better” circuits \(those with lower KL divergence\) yield more interpretable features\. We compute the Pearson correlation between the prompt\-level KL divergence and the mean interpretability scores of the retained features \(Figure[5](https://arxiv.org/html/2604.16889#A4.F5)\)\. ForClarity, we observe a significant negative correlation \(Llamar≈−0\.36r\\approx\-0\.36, Gemmar≈−0\.17r\\approx\-0\.17;p<0\.001p<0\.001\)\. Since lower KL indicates better fidelity, this result implies thatprompts where the pruned circuit functions well tend to contain features that are easier to explain\. Interestingly, Purity and Responsiveness show a weaker or slightly inverse relationship with KL \(Llamar≈0\.15r\\approx 0\.15for Purity\), suggesting that while functional fidelity strongly predicts theclarityof the mechanism, themonosemanticity\(purity\) of features may depend more on the specific semantic content of the prompt \(e\.g\., presence of specific named entities\) rather than the circuit’s overall error rate\.

![Refer to caption](https://arxiv.org/html/2604.16889v1/feature_corr_Gemma.png)\(a\)Gemma\-2\-2B \(r≈0\.71r\\approx 0\.71\)\.
![Refer to caption](https://arxiv.org/html/2604.16889v1/feature_corr_Llama.png)\(b\)Llama\-3\.2\-1B \(r≈0\.73r\\approx 0\.73\)\.

Figure 4:Feature Purity vs\. Responsiveness\.Scatter plots for FAP\-selected features\. We observe a strong positive correlation \(r\>0\.7r\>0\.7\) across both models, confirming that PIE selects features that are both precise \(Pure\) and sensitive \(Responsive\)\.![Refer to caption](https://arxiv.org/html/2604.16889v1/fidelity_quality_Gemma.jpg)\(a\)Gemma\-2\-2B: Prompt Fidelity vs\. Quality\.
![Refer to caption](https://arxiv.org/html/2604.16889v1/fidelity_quality_Llama.jpg)\(b\)Llama\-3\.2\-1B: Prompt Fidelity vs\. Quality\.

Figure 5:Does Fidelity Predict Interpretability?We plot the KL Divergence of each prompt \(x\-axis, lower is better\) against the mean interpretability metrics of the pruned features \(y\-axis\)\. ForMean Clarity\(left panels\), we observe a consistent negative slope, indicating that circuits with higher behavioral fidelity \(lower KL\) are composed of features that are easier to explain\.![Refer to caption](https://arxiv.org/html/2604.16889v1/x2.png)\(a\)Gemma\-2\-2B\.
![Refer to caption](https://arxiv.org/html/2604.16889v1/x3.png)\(b\)Llama\-3\.2\-1B\.

Figure 6:The Signal\-to\-Noise Gap\.Violin plots comparing Clarity, Purity, and Responsiveness for Random vs\. FAP variants\. Random sampling includes many low\-quality features \(scores near 0\), while FAP selects features in the high\-interpretability regime with low variance\.

## Appendix EFAP\-Signal: Sign\-Aware Budget Allocation \(Ablation\)

#### Motivation: sign cancellation in magnitude pruning\.

Magnitude\-based attribution can under\-select features with negative \(inhibitory\) contributions when positive features dominate the score distribution\. In mechanistic circuits, such “brake” components may be necessary to preserve calibrated behavior and to prevent systematic over\-correction under pruning\.

#### Definition\.

LetSFAP​\(f\)S\_\{\\text\{FAP\}\}\(f\)be the base Feature Attribution Patching score \(Eq\. \(1\)\)\. FAP\-Signal enforces a minimum allocation to both positive\- and negative\-scoring features at a fixed budgetKK\. For a user\-chosen ratioγ∈\(0,0\.5\)\\gamma\\in\(0,0\.5\):

1. 1\.Select the topγ​K\\gamma Kfeatures from\{f:SFAP​\(f\)\>0\}\\\{f:S\_\{\\text\{FAP\}\}\(f\)\>0\\\}by score magnitude\.
2. 2\.Select the topγ​K\\gamma Kfeatures from\{f:SFAP​\(f\)<0\}\\\{f:S\_\{\\text\{FAP\}\}\(f\)<0\\\}by score magnitude\.
3. 3\.Fill the remaining\(1−2​γ\)​K\(1\-2\\gamma\)Kslots by the largest\|SFAP​\(f\)\|\|S\_\{\\text\{FAP\}\}\(f\)\|among all remaining features\.

This procedure preserves a signed “floor” of inhibitory features while retaining the simplicity and speed of the base top\-KKselection\.

#### Experimental setting\.

We evaluate FAP\-Signal under the same protocol as the main paper: IOI pruning withK=100K\{=\}100, followed by automated interpretation and FADE\-style evaluation on a held\-out Wikipedia distribution \(Appendix[B](https://arxiv.org/html/2604.16889#A2)\)\. We useγ=0\.25\\gamma=0\.25throughout\.

#### Empirical outcome on IOI\.

In our IOI experiments, FAP\-Signal does not yield a consistent improvement over base FAP in either behavioral fidelity \(KL, PCR\) or interpretability metrics \(Clarity, Purity, Responsiveness\)\. Table[4](https://arxiv.org/html/2604.16889#A5.T4)shows that FAP\-Signal matches base FAP to within noise on KL/PCR\. Table[5](https://arxiv.org/html/2604.16889#A5.T5)reports the three semantic\-efficiency metrics introduced in Sec\.[H](https://arxiv.org/html/2604.16889#A8): Semantic Cost Efficiency \(SCE\), Signal\-to\-Divergence Ratio \(SDR\), and Risk\-Adjusted Responsiveness \(RAR\)\. Across both models, FAP\-Signal does not improve over base FAP and remains slightly below FAP\-Synergy\.

Table 4:Fidelity ablation for FAP\-Signal\.Same setup as Table[2](https://arxiv.org/html/2604.16889#S5.T2)in the main text, but including the sign\-aware variant\.Table 5:Efficiency metrics including FAP\-Signal \(restricted to metrics defined in the main text\)\.We report Semantic Cost Efficiency \(SCE\), Signal\-to\-Divergence Ratio \(SDR; dB, closer to0is better\), and Risk\-Adjusted Responsiveness \(RAR\) as defined in Sec\.[H](https://arxiv.org/html/2604.16889#A8)\. Across both models, FAP\-Signal does not improve over base FAP, while FAP\-Synergy provides the best overall efficiency\.
#### Interpretation\.

These results suggest that, for IOI on the evaluated CLTs, sign cancellation is not the dominant failure mode of base FAP atK=100K\{=\}100\. In contrast, boundary interactions \(synergy\) appear to be the more salient source of pruning error, motivating the main focus on FAP\-Synergy\.

## Appendix FEconomic Analysis

Our PIE pipeline incurs API cost only in the*Interpret*and*Evaluate*stages\. Under our experimental configuration, theexplanationphase \(GPT\-5\.2\) consumes approximately4,0004\{,\}000input tokens \(system prompt \+ 40 max\-activating exemplars\) and produces≈200\\approx 200output tokens*per feature*\. The subsequentevaluationphase \(GPT\-5 mini, used for both synthetic clarity generation and purity rating\) is more data\-intensive, averaging22,65022\{,\}650input tokens and4,0004\{,\}000output tokens per feature\. Under standard pricing\(OpenAI,[2026](https://arxiv.org/html/2604.16889#bib.bib75)\), this yields a total estimated cost of

cfeat≈$​0\.0235per interpreted feature\.c\_\{\\text\{feat\}\}\\approx\\mathdollar 0\.0235\\quad\\text\{per interpreted feature\.\}
#### Prompt\-level budgeting \(active set vs\.K=100K\{=\}100\)\.

A rigorous alternative to interpreting*all*CLT features is to interpret only those that are*active*for the prompt\. In our setting, the mean number of active feature occurrences per prompt is4,1884\{,\}188for Llama\-3\.2\-1B and5,1905\{,\}190for Gemma\-2\-2B\.

Interpreting the full active set for a*single*prompt would therefore cost

CostLlama, active/prompt\\displaystyle\\text\{Cost\}\_\{\\text\{Llama, active/prompt\}\}≈4,188⋅cfeat≈$​98\.42,\\displaystyle\\approx 4\{,\}188\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 98\.42,\(3\)CostGemma, active/prompt\\displaystyle\\text\{Cost\}\_\{\\text\{Gemma, active/prompt\}\}≈5,190⋅cfeat≈$​121\.97\.\\displaystyle\\approx 5\{,\}190\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 121\.97\.\(4\)In contrast, PIE interprets only a fixed budget ofK=100K\{=\}100features, costing

CostK=100≈100⋅cfeat≈$​2\.35\.\\text\{Cost\}\_\{K=100\}\\approx 100\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 2\.35\.\(5\)Thus, even when comparing against the*strong*baseline that restricts attention to active features, PIE reduces interpretation/evaluation spend by a factor of4,188100≈41\.9×\\frac\{4\{,\}188\}\{100\}\\approx 41\.9\\times\(Llama\) and5,190100≈51\.9×\\frac\{5\{,\}190\}\{100\}\\approx 51\.9\\times\(Gemma\), aligning with the observed≈40×\\approx 40\\timesfidelity efficiency gap on the active set\.

#### Dataset\-level budgeting \(global reuse across 2,000 prompts\)\.

The prompt\-level view is conservative because it does not exploit reuse: across a dataset, the same features appear repeatedly, so we can cache interpretations and only pay once per*unique*feature\. Concretely, when aggregating acrossN=2000N\{=\}2000prompts, the number of*unique*features that ever appear askeptafter pruning is approximately4,4004\{,\}400for Llama and4,0004\{,\}000for Gemma\. Under this global accounting, the total interpretation\+evaluation cost becomes

CostLlama, global kept\\displaystyle\\text\{Cost\}\_\{\\text\{Llama, global kept\}\}≈4,400⋅cfeat≈$​103\.40,\\displaystyle\\approx 4\{,\}400\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 103\.40,\(6\)CostGemma, global kept\\displaystyle\\text\{Cost\}\_\{\\text\{Gemma, global kept\}\}≈4,000⋅cfeat≈$​94\.00\.\\displaystyle\\approx 4\{,\}000\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 94\.00\.\(7\)By comparison, a naive global sweep that attempts to interpret the*entire*CLT dictionary would require evaluating

\|ℱLlama\|=16×32,768=524,288,\|ℱGemma\|=26×16,384=425,984\.\\begin\{split\}\|\\mathcal\{F\}\_\{\\text\{Llama\}\}\|&=16\\times 32\{,\}768=524\{,\}288,\\\\ \|\\mathcal\{F\}\_\{\\text\{Gemma\}\}\|&=26\\times 16\{,\}384=425\{,\}984\.\\end\{split\}\(8\)which would cost

CostLlama, full dict\\displaystyle\\text\{Cost\}\_\{\\text\{Llama, full dict\}\}≈524,288⋅cfeat≈$​12,320\.77,\\displaystyle\\approx 524\{,\}288\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 12\{,\}320\.77,\(9\)CostGemma, full dict\\displaystyle\\text\{Cost\}\_\{\\text\{Gemma, full dict\}\}≈425,984⋅cfeat≈$​10,010\.62\.\\displaystyle\\approx 425\{,\}984\\cdot c\_\{\\text\{feat\}\}\\approx\\mathdollar 10\{,\}010\.62\.\(10\)Therefore, from a global perspective PIE reduces the evaluation burden by524,2884,400≈119×\\frac\{524\{,\}288\}\{4\{,\}400\}\\approx 119\\timeson Llama and425,9844,000≈106×\\frac\{425\{,\}984\}\{4\{,\}000\}\\approx 106\\timeson Gemma, translating to savings on the order of∼$​12\.2​k\{\\sim\}\\mathdollar 12\.2\\text\{k\}and∼$​9\.9​k\{\\sim\}\\mathdollar 9\.9\\text\{k\}for a single end\-to\-end run\.

Scenario\#FeaturesCostvs\. BudgetedReductionPrompt\-level \(single prompt\)Llama active set4,188$98\.42$2\.35 \(K=100K\{=\}100\)41\.9×41\.9\\timesGemma active set5,190$121\.97$2\.35 \(K=100K\{=\}100\)51\.9×51\.9\\timesGlobal \(2,000 prompts; unique features\)Llama kept \(unique\)4,400$103\.40full dict: $12,320\.77119×119\\timesGemma kept \(unique\)4,000$94\.00full dict: $10,010\.62106×106\\timesTable 6:Economic impact of budgeting\.Usingcfeat≈$​0\.0235c\_\{\\text\{feat\}\}\\approx\\mathdollar 0\.0235per feature, a fixed interpretation budget \(and global reuse across prompts\) yields large cost reductions relative to interpreting the full active set per prompt or sweeping the entire CLT dictionary\.

## Appendix GLimitations

Our results should be interpreted in light of several practical limitations\.

#### Interpretability evaluation remains concentrated atK=100K\{=\}100\.

We report broader behavioral\-fidelity sweeps overK∈\{50,100,200,400,800\}K\\in\\\{50,100,200,400,800\\\}and across two tasks, but our full end\-to\-end interpretation evaluation \(including FADE\-style metrics and economic analysis\) remains centered on the mainK=100K\{=\}100setting\. As a result, while the pruning robustness story now extends beyond a single budget, we do not yet report a full scaling study of downstream interpretability quality across the entire budget range\.

#### No CLT training; reliance on public replacement checkpoints\.

We do not train Cross\-Layer Transcoders \(CLTs\) ourselves; instead, we use public CLT checkpoints released with Circuit Tracer tooling \(Gemma\-2\-2B CLT and Llama\-3\.2\-1B CLT\)\. Consequently, our claims are conditional on the quality and representational coverage of these replacement models\. In particular, pruning behavior and feature semantics may differ for CLTs trained with different data, objectives, sparsity regimes, or architectures\.

#### Scope of causal claims\.

PIE is designed to isolate a minimal circuit that reproduces behavior under the replacement\-model intervention, but this does not automatically imply that every retained feature corresponds to a unique mechanistic “part” in a human\-interpretable decomposition\. Feature redundancy, polysemanticity, and interaction effects can persist even after pruning, and the replacement\-model abstraction may miss mechanisms not captured by the CLT basis\.

## Appendix HSemantic Efficiency and Signal Analysis

While standard metrics show parity, the true advantage of FAP\-Synergy lies in itsefficiency—the amount of interpretable signal retained per unit of behavioral degradation\. Because Synergy actively repairs circuit boundaries to lower KL divergence, it minimizes the ”cost” of pruning\. To quantify this, we introduce three efficiency\-oriented metrics:

- •Semantic Cost Efficiency \(SCE\):A yield metric defined asClarity\+PurityKL⋅PCR\\frac\{\\text\{Clarity\}\+\\text\{Purity\}\}\{\\text\{KL\}\\cdot\\text\{PCR\}\}\. This rewards methods that maximize semantics while strictly minimizing the joint compounded cost of model degradation\.
- •Signal\-to\-Divergence Ratio \(SDR\):A decibel\-scale metric defined as10⋅log10⁡\(ResponsivenessKL\)10\\cdot\\log\_\{10\}\(\\frac\{\\text\{Responsiveness\}\}\{\\text\{KL\}\}\)\. This separates the interpretable signal from the behavioral noise\.
- •Risk\-Adjusted Responsiveness \(RAR\):Defined asResponsivenessKL⋅PCR\\frac\{\\text\{Responsiveness\}\}\{\\text\{KL\}\\cdot\\text\{PCR\}\}, isolating the responsiveness gain normalized by risk\.

As shown in Table[7](https://arxiv.org/html/2604.16889#A8.T7), FAP\-Synergy dominates across all efficiency metrics\. For Llama\-3\.2\-1B, Synergy improves SCE by over900 pointsand RAR by over500 pointscompared to the Base method\. This confirms that Synergy is not just selecting random features; it is selecting features that provide a higher return on interpretation investment by enforcing tighter behavioral bounds\.

Table 7:Semantic Efficiency Analysis\.By viewing KL and PCR as costs, we observe that FAP\-Synergy yields significantly higher interpretability per unit of behavioral degradation\.SCEmeasures total semantic yield,SDRmeasures signal\-to\-noise in dB \(closer to 0 is better\), andRARmeasures responsiveness relative to risk\. Best results are bolded\.