通过少量精确测量修正归因补丁的偏差

# Correcting Attribution Patching with a Few Exact Measurements
Source: [https://arxiv.org/html/2610.27393](https://arxiv.org/html/2610.27393)
Minxing Zheng11footnotemark:1Affiliation:Carnegie Mellon UniversityJialu WangAffiliation:University of California, Santa Cruz

###### Abstract

Mechanistic interpretability asks which language model components matter for a behavior, and activation patching answers by changing one component’s activation and measuring its effect, at a cost growing with components and prompts\. Attribution patching estimates these effects from one gradient pass, but its estimates can be far off, and verifying each one restores the cost\. Can the structure of these errors reduce this cost? We proposeGuided\-AP, which uses a few exact measurements to reveal attribution patching’s errors and correct unmeasured estimates\. On a single prompt, many components’ errors share one sign, so a small exact sample yields one correction for the rest\. Over many prompts of one task, components are misestimated similarly, so a few measured prompts show what to verify and how to correct each component’s average effect\. Our analysis shows that choosing what to verify needs error size, while correction also needs sign\. On a single prompt, the correction lowers error by up to 26% across three behaviors at the compute of verifying the largest estimates and also on larger models\. Over many prompts, learned corrections cut the error of average effects on factual completion by 75% to 79%, repaying their cost in long analyses\.

## 1Introduction

Mechanistic interpretability seeks to explain how a language model’s internal computations produce its behavior\. To study behaviors such as resolving a reference or recalling a fact, researchers measure how internal components, such as attention heads or MLP neurons, influence the model’s prediction\. One widely used approach for measuring this influence is activation patching \(AP\)\([Wang et al\., 2023](https://arxiv.org/html/2610.27393#bib.bib25);[Zhang and Nanda, 2024](https://arxiv.org/html/2610.27393#bib.bib29)\)\. It replaces a component’s activation with one from a contrasting input and measures the resulting change in a prediction score, such as the logit difference between two answers\.

![Refer to caption](https://arxiv.org/html/2610.27393v1/shared_errors_overview_v5.png)Figure 1:Guided\-APlearns shared attribution errors\. Within a prompt, a small exact sample corrects unmeasured estimates; across prompts, learned error profiles choose what to verify and correct remaining estimates and average effects\. Final estimates are exact where verified and corrected elsewhere\.However, AP is expensive across many components and prompts, because each measurement is a separate intervention\. To reduce this cost, attribution patching \(AtP\) approximates the same effects using gradients from a single backward pass\([Nanda, 2023](https://arxiv.org/html/2610.27393#bib.bib20);[Syed et al\., 2024](https://arxiv.org/html/2610.27393#bib.bib24)\)\. This efficiency comes at the cost of accuracy, because its estimates can be far from the measured effects\([Zhang and Wang, 2026](https://arxiv.org/html/2610.27393#bib.bib30)\)\. Verifying every estimate with AP would remove the error but restore the original cost, so the challenge is to improve the estimates without exhaustive measurement\. Under a limited budget, verifying the components with the largest estimated effects is a strong baseline, but a large effect does not mean a large error\. Errors can concentrate in a few components and recur across prompts, as illustrated in Figure[2](https://arxiv.org/html/2610.27393#S3.F2)\. Errors can also persist when estimates are averaged across prompts\. The central question of this paper is whether knowing the structure of these errors makes attribution analysis more effective and more efficient\.

Our answer builds on the fact that an exact measurement reveals both a component’s true effect and the error of its cheap estimate\. When such errors are similar across estimates, a few measurements can repair estimates that were never measured\. Each measurement therefore improves more than one estimate\. It replaces its own estimate with the true effect, and the error it reveals improves the estimates left unmeasured\. Each error belongs to one component on one prompt, so we can look at the errors from two perspectives \(Figure[1](https://arxiv.org/html/2610.27393#S1.F1)\)\. Our method,Guided\-AP, applies the same estimator in both settings\. On a single prompt, the errors of many components share one sign, so one shift improves many estimates at once\. We measure a small, representative sample and shift every unmeasured estimate by the median of the sampled errors, the shift with the lowest mean absolute error on the sample\. The rest of the budget verifies the largest estimated effects\. Over many prompts, each component’s error tends to recur\. A few measured prompts give each component an error profile\. Later prompts reuse it to choose verifications and correct each component’s average effect\.Guided\-APlearns from exact AP measurements\.

On a single prompt, the correction needs summary statistics of the error distribution \(e\.g\., its median\), and a small exact sample on that prompt estimates them\. Centering estimates without measuring their errors does not reproduce this gain\. With the exact sample, the correction lowers error by up to 26% at the same compute\. Over many prompts, the errors left after averaging mostly share one sign, so a few calibration prompts can estimate each component’s average error\. A correction learned from exact measurements cuts the error of these averages on factual completion by 75% to 79% relative to uncorrected averages, with no measurement on the analyzed prompts\. Using the same measurements directly as average effects is 6 to 15 times worse, so the gain comes from modeling the errors\. Choosing what to verify needs only the size of each error\. In our cost projections, the correction pays for itself within a few thousand prompts, and learned selection needs longer analyses, up to tens of thousands of prompts\. The single\-prompt gains carry to larger models and finer components, up to a 32B model with tens of thousands of components per layer, and on some tasks the correction reaches a fixed accuracy target several times faster than verification alone\.

We make three contributions\.

- •A framework of budgeted attribution analysis\.We study attribution analysis under a fixed budget of exact measurements that must serve both verification and correction, and we compare methods by the error left at equal compute\.
- •Analysis of error structures\.We show that attribution errors are shared within a single prompt and recur over many prompts, and which error statistic selection and correction each need\.
- •Scalability to finer components and larger models\.We show that a fixed budget of exact measurements remains useful as models grow larger and their components finer, where each measurement must go further\.

## 2Related Work

#### Causal attribution and circuit discovery\.

Activation patching localizes behavior through interventions whose results depend on corruption, component definitions, and output metrics\([Wang et al\., 2023](https://arxiv.org/html/2610.27393#bib.bib25);[Zhang and Nanda, 2024](https://arxiv.org/html/2610.27393#bib.bib29)\)\. Automated circuit discovery reduces search costs\([Conmy et al\., 2023](https://arxiv.org/html/2610.27393#bib.bib5);[Syed et al\., 2024](https://arxiv.org/html/2610.27393#bib.bib24)\); benchmarks and a shared task evaluate localization across tasks\([Mueller et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib19);[Arad et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib1);[Hanna et al\., 2024](https://arxiv.org/html/2610.27393#bib.bib10)\), and feature\-level methods reach tens of thousands of components\([Marks et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib18)\)\. We estimate isolated component effects and their dataset means; low effect\-vector error does not imply faithfulness of a jointly intervened circuit\.

#### Correcting attribution estimates\.

Attribution patching linearizes the intervention\([Nanda, 2023](https://arxiv.org/html/2610.27393#bib.bib20)\)\. Existing methods correct estimates within a prompt\. AtP∗targets specific failure modes\([Kramár et al\., 2024](https://arxiv.org/html/2610.27393#bib.bib16)\), relevance patching replaces gradients with propagation coefficients\([Jafari et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib12)\), interaction\-aware backpropagation fixes gradient flow through softmax and normalization\([Edin et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib6)\), path\-based attribution integrates along the intervention\([Sundararajan et al\., 2017](https://arxiv.org/html/2610.27393#bib.bib23);[Janizek et al\., 2021](https://arxiv.org/html/2610.27393#bib.bib13)\), and second\-order patching correction expands effects at the endpoint\([Zhang and Wang, 2026](https://arxiv.org/html/2610.27393#bib.bib30)\)\. We learn error structure from exact measurements and reuse it across components within a prompt or across prompts for a component\. Our base estimate is AtP\.

#### Efficient error estimation\.

Randomized diagonal estimators approximate the diagonal of a matrix from matrix\-vector products\([Hutchinson, 1989](https://arxiv.org/html/2610.27393#bib.bib11);[Bekas et al\., 2007](https://arxiv.org/html/2610.27393#bib.bib3);[Baston and Nakatsukasa, 2022](https://arxiv.org/html/2610.27393#bib.bib2);[Epperly et al\., 2024](https://arxiv.org/html/2610.27393#bib.bib7)\), and Hessian\-vector products make this cheap for neural networks\([Pearlmutter, 1994](https://arxiv.org/html/2610.27393#bib.bib21);[Yao et al\., 2021](https://arxiv.org/html/2610.27393#bib.bib28)\); we use such sketches only as a comparison label source\. Allocating exact measurements to unreliable estimates follows active learning\([Cohn et al\., 1994](https://arxiv.org/html/2610.27393#bib.bib4);[Lewis and Gale, 1994](https://arxiv.org/html/2610.27393#bib.bib17)\)and selective prediction\([Geifman and El\-Yaniv, 2017](https://arxiv.org/html/2610.27393#bib.bib8)\); influence estimation at scale similarly uses cheap curvature approximations\([Koh and Liang, 2017](https://arxiv.org/html/2610.27393#bib.bib15);[Grosse et al\., 2023](https://arxiv.org/html/2610.27393#bib.bib9)\)\. We test whether reusing attribution errors within or across prompts improves accuracy after charging acquisition, fitting, and verification costs\.

## 3Proposed Method

### 3\.1Preliminaries

Consider a language model withNNcandidate components, a clean promptxx, and a contrasting promptx′x^\{\\prime\}\. Writeaia\_\{i\}andai′a\_\{i\}^\{\\prime\}for componentii’s activations on the two prompts andδi=ai′−ai\\delta\_\{i\}=a\_\{i\}^\{\\prime\}\-a\_\{i\}for its patch direction\. These activations may be scalars or vectors\. LetF⁡\(t\)F\(t\), fort∈ℝNt\\in\\mathbb\{R\}^\{N\}, be the prediction metric onxxwhen each component receives the additive shiftti​δit\_\{i\}\\delta\_\{i\}and later layers are recomputed;F⁡\(0\)F\(0\)is the clean run\.

To isolate the effect of each component, letvi∈ℝNv\_\{i\}\\in\\mathbb\{R\}^\{N\}denote the coordinate unit vector withiith entry11and00elsewhere, and define the isolated AP effect of componentii, denotedΔi\\Delta\_\{i\}, and its AtP approximationgig\_\{i\}as

Δi=F⁡\(vi\)−F⁡\(0\),gi=∂iF⁡\(0\)\.\\Delta\_\{i\}=F\(v\_\{i\}\)\-F\(0\),\\qquad g\_\{i\}=\\partial\_\{i\}F\(0\)\.We callei=Δi−gie\_\{i\}=\\Delta\_\{i\}\-g\_\{i\}the AtP residual, and evaluate an estimateΔ^\\widehat\{\\Delta\}of\(Δ1,…,ΔN\)\(\\Delta\_\{1\},\\ldots,\\Delta\_\{N\}\)by its mean absolute error,MAE⁡\(Δ^\)=N−1​∑i=1N\|Δi−Δ^i\|\\operatorname\{MAE\}\(\\widehat\{\\Delta\}\)=N^\{\-1\}\\sum\_\{i=1\}^\{N\}\|\\Delta\_\{i\}\-\\widehat\{\\Delta\}\_\{i\}\|\.

Though accurate, exact AP requiresNNseparate interventions per prompt, with a cost linear in both the number of components and the number of prompts\. In comparison, AtP estimates allNNeffects from one backward pass, but its linear approximation misses the curvature of a finite patch, so its residuals can be large\([Zhang and Wang, 2026](https://arxiv.org/html/2610.27393#bib.bib30)\)\.

### 3\.2Learning Shared Attribution Errors

Given these two limitations, we seek a framework that balances estimation accuracy and computation\. Our key observation is that AtP residuals follow structured patterns\. First, a few components carry most of the error\. Second, on a single prompt, the residuals are biased away from zero\. Third, each component’s residual is consistent across prompts of the same task\. Figure[2](https://arxiv.org/html/2610.27393#S3.F2)demonstrates these patterns with a toy example on GPT\-2\.

![Refer to caption](https://arxiv.org/html/2610.27393v1/motivation_signed_1x2.png)Figure 2:AtP errors on GPT\-2\. \(a\) The top four components \(▼\\blacktriangledown\) carry 80% of the test error\. \(b\) Calibration versus test error for each component, with the top four highlighted\. The two errors are strongly correlated and consistently biased away from zero\.Motivated by these patterns, we propose to measure only a few well\-chosen components exactly, which removes most of the AtP error, and to use their residuals to correct the remaining unmeasured components\. Specifically, given a budget ofBBexact AP measurements per prompt, we aim to find a subsetSSof theNNcomponents with\|S\|=B\|S\|=Band construct the estimator

Δ^i=\{Δi,i∈S,gi\+ci,i∉S,\\widehat\{\\Delta\}\_\{i\}=\\begin\{cases\}\\Delta\_\{i\},&i\\in S,\\\\ g\_\{i\}\+c\_\{i\},&i\\notin S,\\end\{cases\}\(1\)wherecic\_\{i\}is learned from exact measurements\. Since measured components are exact, choosingSSandccto minimize the MAE ofΔ^\\widehat\{\\Delta\}gives

minS,c\\displaystyle\\min\_\{S,\\,c\}∑i=1N\|Δ^i−Δi\|=∑i∉S\|gi\+ci−Δi\|\\displaystyle\\sum\_\{i=1\}^\{N\}\|\\widehat\{\\Delta\}\_\{i\}\-\\Delta\_\{i\}\|=\\sum\_\{i\\notin S\}\|g\_\{i\}\+c\_\{i\}\-\\Delta\_\{i\}\|\(2\)s\.t\.\\displaystyle\\text\{s\.t\.\}S⊆\{1,…,N\},\|S\|=B\.\\displaystyle S\\subseteq\\\{1,\\ldots,N\\\},\\ \|S\|=B\.Equation[2](https://arxiv.org/html/2610.27393#S3.E2)requires the unknown residualsΔi−gi\\Delta\_\{i\}\-g\_\{i\}fori∉Si\\notin S\. The patterns above, however, can make these residuals predictable from measured ones\. Within a prompt, measured components reveal the shared offset of the rest; across prompts, a component’s residual on earlier prompts predicts its residual on new ones\. We therefore learn the corrections and the measured set from calibration data𝒟=\{\(Δp​i,gp​i\)\}\\mathcal\{D\}=\\\{\(\\Delta\_\{pi\},g\_\{pi\}\)\\\}, the exact AP measurements and AtP estimates of componentiion promptppavailable for fitting, from earlier prompts or from the test prompt itself, with residualsep​i=Δp​i−gp​ie\_\{pi\}=\\Delta\_\{pi\}\-g\_\{pi\}\. We fit the residual, and then the size of what the fitted correction leaves, by absolute loss over𝒟\\mathcal\{D\},

f^\\displaystyle\\widehat\{f\}=arg⁡minf⁡‖e−f‖1,\\displaystyle=\\arg\\min\_\{f\}\\\|e\-f\\\|\_\{1\},\(3\)h^\\displaystyle\\widehat\{h\}=arg⁡minh⁡‖\|e−f^\|−h‖1,\\displaystyle=\\arg\\min\_\{h\}\\big\\\|\|e\-\\widehat\{f\}\|\-h\\big\\\|\_\{1\},whereffandhhare evaluated at\(gp​i,i\)\(g\_\{pi\},i\)and may belong to any model class, such as a constant, a linear regression, or a neural network; in this paperffis a constant andhha log\-linear model \(Appendix[A](https://arxiv.org/html/2610.27393#A1)\)\. At test time, for a test promptp′p^\{\\prime\}, each componentiireceives a correction and a selection score,

c^p′​i=f^​\(gp′​i,i\),sp′​i=h^​\(gp′​i,i\)\.\\widehat\{c\}\_\{p^\{\\prime\}i\}=\\widehat\{f\}\(g\_\{p^\{\\prime\}i\},i\),\\qquad s\_\{p^\{\\prime\}i\}=\\widehat\{h\}\(g\_\{p^\{\\prime\}i\},i\)\.\(4\)The budget goes to the components with the highest scores, and the rest receive correctionsgp′​i\+c^p′​ig\_\{p^\{\\prime\}i\}\+\\widehat\{c\}\_\{p^\{\\prime\}i\}\. We call this procedureGuided\-AP\.

The performance ofGuided\-APdepends on what can be learned from measured residuals, since a measured residual helps unmeasured components only insofar as it reveals an error pattern they share\. The patterns in Figure[2](https://arxiv.org/html/2610.27393#S3.F2)suggest two axes of sharing: across components within one prompt, and across prompts of the same task for a given component\. These lead to two learning settings, which we callwithin\-prompt learningandacross\-prompt learning, respectively\. The former learns a prompt\-specific offset from other components measured on that prompt; the latter learns component\-specific error patterns from earlier prompts, informing both correction and verification on later prompts\. Within\-prompt learning needs no prior labels, whereas across\-prompt learning requires measurements from earlier prompts\. We describe each setting’s labels, correctionc^\\widehat\{c\}, and selection scoressbelow; Algorithm[1](https://arxiv.org/html/2610.27393#alg1)summarizes both\.

#### Within\-prompt learning\.

This setting learns a prompt\-specific offset from a few components measured on the same prompt, so it needs no prior labels but pays for its own\. The budget is split into a pilot setS1S\_\{1\}that supplies the labels and a verification setS2S\_\{2\}, with\|S1\|\+\|S2\|=B\|S\_\{1\}\|\+\|S\_\{2\}\|=B\. The pilot should represent the typical component, since the offset it estimates will be applied to all unmeasured ones\. Because residual size varies with attribution, we stratify by\|gp′​i\|\|g\_\{p^\{\\prime\}i\}\|, dividing the components intoKKequal groupsG1,…,GKG\_\{1\},\\ldots,G\_\{K\}by attribution rank and drawing the pilot equally from each,

S1=⋃k=1KUk,Uk⊆Gk,S\_\{1\}=\\bigcup\_\{k=1\}^\{K\}U\_\{k\},\\qquad U\_\{k\}\\subseteq G\_\{k\},\(5\)giving labels𝒟=\{\(Δp′​j,gp′​j\)\}j∈S1\\mathcal\{D\}=\\\{\(\\Delta\_\{p^\{\\prime\}j\},g\_\{p^\{\\prime\}j\}\)\\\}\_\{j\\in S\_\{1\}\}\. The correction is this offset, estimated by the pilot median, which minimizes the absolute error\([Koenker and Bassett, 1978](https://arxiv.org/html/2610.27393#bib.bib14)\)and is insensitive to its few large residuals,

c^p′​i=medianj∈S1⁡ep′​jfor all​i\.\\widehat\{c\}\_\{p^\{\\prime\}i\}=\\operatorname\{median\}\_\{j\\in S\_\{1\}\}e\_\{p^\{\\prime\}j\}\\quad\\text\{for all \}i\.\(6\)With at most one label per component, no component\-specific size model can be fit, so the selection score issp′​i=\|gp′​i\|s\_\{p^\{\\prime\}i\}=\|g\_\{p^\{\\prime\}i\}\|, the only available proxy for residual size, and the remaining budget verifies the highest scores outsideS1S\_\{1\}\. The pilot corrects the bulk; verification handles the tail\.

Algorithm 1Guided\-AP1:test prompt

p′p^\{\\prime\}, budget

BB, calibration data

𝒟\\mathcal\{D\}\(possibly empty\)

2:

gp′​i←∂iFp′​\(0\)g\_\{p^\{\\prime\}i\}\\leftarrow\\partial\_\{i\}F\_\{p^\{\\prime\}\}\(0\)for all

ii
3:if

𝒟=∅\\mathcal\{D\}=\\varnothingthen⊳\\trianglerightwithin\-prompt learning

4:

S1←S\_\{1\}\\leftarrowstratified sample by

\|gp′​i\|\|g\_\{p^\{\\prime\}i\}\|⊳\\trianglerightEq\.[5](https://arxiv.org/html/2610.27393#S3.E5)

5:

𝒟←\{\(Δp′​i,gp′​i\)\}i∈S1\\mathcal\{D\}\\leftarrow\\\{\(\\Delta\_\{p^\{\\prime\}i\},g\_\{p^\{\\prime\}i\}\)\\\}\_\{i\\in S\_\{1\}\}
6:

f^←\\widehat\{f\}\\leftarrowpilot median,

h^←\|gp′​i\|\\widehat\{h\}\\leftarrow\|g\_\{p^\{\\prime\}i\}\|⊳\\trianglerightEq\.[6](https://arxiv.org/html/2610.27393#S3.E6)

7:else⊳\\trianglerightacross\-prompt learning

8:

S1←∅S\_\{1\}\\leftarrow\\varnothing
9:

f^,h^←\\widehat\{f\},\\widehat\{h\}\\leftarrowfit on

𝒟\\mathcal\{D\}⊳\\trianglerightEq\.[3](https://arxiv.org/html/2610.27393#S3.E3)

10:

c^p′​i←f^​\(gp′​i,i\)\\widehat\{c\}\_\{p^\{\\prime\}i\}\\leftarrow\\widehat\{f\}\(g\_\{p^\{\\prime\}i\},i\),

sp′​i←h^​\(gp′​i,i\)s\_\{p^\{\\prime\}i\}\\leftarrow\\widehat\{h\}\(g\_\{p^\{\\prime\}i\},i\)for all

ii⊳\\trianglerightEq\.[4](https://arxiv.org/html/2610.27393#S3.E4)

11:

S2←S\_\{2\}\\leftarrowtop

B−\|S1\|B\-\|S\_\{1\}\|of

sp′​is\_\{p^\{\\prime\}i\}outside

S1S\_\{1\}
12:

S←S1∪S2S\\leftarrow S\_\{1\}\\cup S\_\{2\}
13:return

Δ^p′​i\\widehat\{\\Delta\}\_\{p^\{\\prime\}i\}with estimated

c^p′​i\\widehat\{c\}\_\{p^\{\\prime\}i\}⊳\\trianglerightEq\.[1](https://arxiv.org/html/2610.27393#S3.E1)

#### Across\-prompt learning\.

This setting learns from earlier prompts of the same task, so it needs labels but spends the whole test\-time budget on verification\. When a component’s residual recurs across prompts, as in Figure[2](https://arxiv.org/html/2610.27393#S3.F2), its residual on a new prompt can be predicted from earlier ones\. From these labels,f^\\widehat\{f\}in Equation[3](https://arxiv.org/html/2610.27393#S3.E3)learns a per\-component correction, andh^\\widehat\{h\}learns, from the component’s attribution and identity, the size of the residual that correction leaves\. Every component receives its correctionc^p′​i\\widehat\{c\}\_\{p^\{\\prime\}i\}and selection scoresp′​is\_\{p^\{\\prime\}i\}, and theBBcomponents with the highest scores are measured exactly\. The correction helps only when its labels are representative, meaning that each corrected component has enough labels measured uniformly across prompts\. Otherwise the correction can mislead and raise the error, and we setf^=0\\widehat\{f\}=0in this case\. For a component’s average effect over many test prompts, we can correct the averaged AtP estimate by adding the component’s mean labeled residual, with no exact measurement on these prompts\. Further details are in Appendix[A](https://arxiv.org/html/2610.27393#A1)\.

### 3\.3Theoretical Analysis

We ask when each setting beats verification alone and why selection needs its own model\. All results assume exact AP measurements; proofs are in Appendix[C](https://arxiv.org/html/2610.27393#A3)\. Within a prompt, the pilot buys a shared correction at the price of displaced verification\. LetUUbe the unmeasured components,AU​\(c\)=∑i∈U\|ei−c\|A\_\{U\}\(c\)=\\sum\_\{i\\in U\}\|e\_\{i\}\-c\|their error after a shiftcc, andmUm\_\{U\}their median, the best shift; letHU=AU​\(0\)−AU​\(mU\)H\_\{U\}=A\_\{U\}\(0\)\-A\_\{U\}\(m\_\{U\}\)be the gain of that shift,EU=AU​\(c^\)−AU​\(mU\)E\_\{U\}=A\_\{U\}\(\\widehat\{c\}\)\-A\_\{U\}\(m\_\{U\}\)the estimation cost of the pilot medianc^\\widehat\{c\}, andCCthe error removed by displaced verification\.

###### Theorem 1\(Within\-prompt gain\)\.

LetΔ^\\widehat\{\\Delta\}be the within\-prompt estimate with a stratified pilot ofnnlabels,n/Kn/Kfrom each ofKKequal\-size strata, andΔ^0\\widehat\{\\Delta\}^\{0\}the estimate that spends the same budget verifying the largest\|gi\|\|g\_\{i\}\|with no correction \(verify\-more, Section[4\.1](https://arxiv.org/html/2610.27393#S4.SS1)\)\. Then

N⁡\(MAE⁡\(Δ^0\)−MAE⁡\(Δ^\)\)=HU−EU−C,N\\big\(\\operatorname\{MAE\}\(\\widehat\{\\Delta\}^\{0\}\)\-\\operatorname\{MAE\}\(\\widehat\{\\Delta\}\)\\big\)=H\_\{U\}\-E\_\{U\}\-C,\(7\)and, over the pilot draw, for allt∈\(0,12\)t\\in\(0,\\tfrac\{1\}\{2\}\), with probability at least1−2​e−2​n​t21\-2e^\{\-2nt^\{2\}\},EU≤εU​\(t\)=max⁡\{AU​\(q⁡\(12−t\)\),AU​\(q⁡\(12\+t\)\)\}−AU​\(mU\)E\_\{U\}\\leq\\varepsilon\_\{U\}\(t\)=\\max\\\{A\_\{U\}\(q\(\\tfrac\{1\}\{2\}\-t\)\),A\_\{U\}\(q\(\\tfrac\{1\}\{2\}\+t\)\)\\\}\-A\_\{U\}\(m\_\{U\}\), whereqqis the quantile function of allNNresiduals\.

The pilot pays when the unmeasured residuals share a sign, soHUH\_\{U\}is large, and cluster near their median, soEUE\_\{U\}is small, by more thanCC\. We first analyze a component\-specific median correction learned from representative calibration labels\. This result concerns absolute error on individual prompts\. Correcting dataset averages instead requires estimating the mean residual\. For independent prompt residuals with quantile functionqqand medianmm, letR⁡\(c\)=𝔼​\|e−c\|R\(c\)=\\mathbb\{E\}\|e\-c\|andΓ=R⁡\(0\)−R⁡\(m\)\\Gamma=R\(0\)\-R\(m\)be the best shift’s gain\.

###### Theorem 2\(Median correction from representative labels\)\.

Letc^\\widehat\{c\}be the median ofnnlabels drawn independently from the component’s residual law, and assume𝔼​\|e\|<∞\\mathbb\{E\}\|e\|<\\infty\. For anyt∈\(0,12\)t\\in\(0,\\tfrac\{1\}\{2\}\), with probability at least1−2​e−2​n​t21\-2e^\{\-2nt^\{2\}\},

R⁡\(c^\)≤R⁡\(0\)−Γ\+2​t​w​\(t\),R\(\\widehat\{c\}\)\\leq R\(0\)\-\\Gamma\+2t\\,w\(t\),\(8\)wherew⁡\(t\)=q⁡\(12\+t\)−q⁡\(12−t\)w\(t\)=q\(\\tfrac\{1\}\{2\}\+t\)\-q\(\\tfrac\{1\}\{2\}\-t\)\.

The bound explains when recurring residual bias makes a median correction useful\. Magnitude\-guided acquisition need not satisfy its sampling assumption\. For dataset\-average correction, an offset improves a component’s estimate when it has the same sign as the average residual and less than twice its magnitude \(Appendix[C\.4](https://arxiv.org/html/2610.27393#A3.SS4)\)\. Selection asks a different question\. Equation[2](https://arxiv.org/html/2610.27393#S3.E2)verifies the components whose error after correction is largest, so a score must predict that error’s size\. Fix the corrections and letri=𝔼​\|ei−c^i\|r\_\{i\}=\\mathbb\{E\}\|e\_\{i\}\-\\widehat\{c\}\_\{i\}\|be componentii’s expected remaining error given what is known before measurement\.

###### Proposition 1\(Selection needs size\)\.

Ifmaxi⁡\|si−ri\|≤ϵ\\max\_\{i\}\|s\_\{i\}\-r\_\{i\}\|\\leq\\epsilon, selecting theBBhighest scores costs at most2​B​ϵ2B\\epsilonextra expected error over theBBhighestrir\_\{i\}\.

Scores must track the expected size of the remaining error, not the size of the expected error\. A component whose residual is large but changes sign hasf^\\widehat\{f\}near zero and largerir\_\{i\}\. This is whyhhis fit separately fromffin Equation[3](https://arxiv.org/html/2610.27393#S3.E3), and why a single prompt, which cannot fithh, falls back to\|g\|\|g\|\.

## 4Experiments

### 4\.1Experimental Setup

Figure 3:Warm runtime and relative AP error for within\-promptGuided\-APand the baselines\. Square size denotes the measurement budgetBB; hollow derivative markers denote larger MS\-HVP and IG settings\. Appendix[D\.3](https://arxiv.org/html/2610.27393#A4.SS3)gives the timing details\.#### Models, tasks, and prompts\.

For within\-promptGuided\-AP\(Section[4\.2](https://arxiv.org/html/2610.27393#S4.SS2)\), we study MLP neurons of Gemma\-2\-9B\([Gemma Team, 2024](https://arxiv.org/html/2610.27393#bib.bib26)\), Qwen3\-14B, and Qwen3\-32B\([Yang et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib27)\)\. For across\-promptGuided\-AP\(Section[4\.3](https://arxiv.org/html/2610.27393#S4.SS3)\), we study attention heads of Qwen3\-8B and Qwen2\.5\-7B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib22)\)\. Tasks: indirect\-object identification \(IOI\), factual completion, and subject\-verb agreement \(SVA\)\. Score: logit difference between correct and counterfactual tokens\. Evaluation prompts are held out; across\-prompt calibration uses separate prompts in one prespecified evaluation\. Figure[2](https://arxiv.org/html/2610.27393#S3.F2)uses separate exact effects from GPT\-2\.

#### Methods and baselines\.

Within\-promptGuided\-APuses 256 exact measurements per layer\. Across\-promptGuided\-APverifiesB∈\{16,32,64,128\}B\\in\\\{16,32,64,128\\\}heads per prompt and separately corrects dataset averages without measuring the evaluation prompts\. Our main baseline,verify\-more, spends the same cost on exact measurements of the components with the largest AtP magnitudes and leaves the rest uncorrected\. Across\-prompt labels are acquired on large attributions, so verification setsf^=0\\widehat\{f\}=0\(Section[3\.2](https://arxiv.org/html/2610.27393#S3.SS2.SSS0.Px2)\) and its comparison with verify\-more isolates selection\. An alternative calibration signal,ProbeCal, replaces the exact calibration labels with curvature\-probe sketches, randomized estimates of the Hessian diagonal\([Bekas et al\., 2007](https://arxiv.org/html/2610.27393#bib.bib3)\), and fits the size model and average correction from them \(Appendix[B](https://arxiv.org/html/2610.27393#A2)\)\. On Gemma\-2\-9B, we also compare with verify\-more on AtP∗estimates, selective HVP, MS\-HVP, singleton IG, and exhaustive AP \(Appendices[D\.3](https://arxiv.org/html/2610.27393#A4.SS3)and[F](https://arxiv.org/html/2610.27393#A6)\)\. All costs include pilot, calibration, and fitting\. We report measured warm runtime per layer within prompts, and calibration plus projected deployment cost across prompts\.

#### Metrics and uncertainty\.

Primary: per\-prompt effect\-vector MAE against exact FP32 AP \(Section[3\.1](https://arxiv.org/html/2610.27393#S3.SS1)\)\. Additional metrics: relative AP error \(summed absolute error over summed absolute effects\), dataset\-average error, and recall of the ten largest effects\. Error reduction: one minus the ratio of mean errors\. Intervals: paired group bootstraps, adjusted over the prespecified across\-prompt comparison family and descriptive within prompts \(Appendices[D](https://arxiv.org/html/2610.27393#A4),[F](https://arxiv.org/html/2610.27393#A6), and[I](https://arxiv.org/html/2610.27393#A9)\)\.

### 4\.2Within\-Prompt Learning

Within\-promptGuided\-APlowers Gemma\-2\-9B MAE by 17% to 26% across all three tasks at the same measured runtime as verify\-more, without earlier calibration prompts \(Table[1](https://arxiv.org/html/2610.27393#S4.T1)\)\. Top\-ten recall remains nearly perfect for both methods\.

Table 1:Within\-promptGuided\-APaccuracy and warm runtime\. MAE reductions compare with runtime\-matched verify\-more\. Groups is the number of evaluation prompts scored on all three tested MLP layers \(Appendix[D](https://arxiv.org/html/2610.27393#A4)\)\.#### Correction mechanism\.

Correcting unmeasured estimates drives the gain\. The fitted shift captures 96% to 97% of the best common shift’s benefit, outweighing displaced verification \(Theorem[1](https://arxiv.org/html/2610.27393#Thmtheorem1); Appendix[D\.5](https://arxiv.org/html/2610.27393#A4.SS5)\)\. Gains persist at equal measurement counts \(Appendix[D](https://arxiv.org/html/2610.27393#A4)\)\. FP64 readout rescoring of the same estimates retains 10% to 23% lower Gemma MAE and positive intervals on every larger\-model task \(Appendix[D\.5](https://arxiv.org/html/2610.27393#A4.SS5)\)\.

Figure 4:Within\-promptGuided\-APablations on Gemma\-2\-9B at 256 measurements: pilot size \(left\) and mean versus median \(right\)\. Values show MAE reductions versus verification alone\. Cohorts: Appendix[D\.4](https://arxiv.org/html/2610.27393#A4.SS4)\.
#### Controls and ablations\.

Measurements chosen for verification need not represent unmeasured errors\. Shifting the remaining estimates by the median residual of those measurements raises MAE in eight of nine settings\. The pilot lowers Gemma MAE by 17% to 24% relative to the better of this control and verification alone \(Table[2](https://arxiv.org/html/2610.27393#S4.T2)\)\.

Table 2:Within\-promptGuided\-APcontrols on Gemma\-2\-9B at the same 256\-measurement budget\. MAE is relative to verification alone; lower is better\.On Gemma,Guided\-APlowers group\-averaged MAE against runtime\-matched verify\-more in 96 of 100 IOI groups, 97 of 100 factual groups, and 98 of 100 SVA groups \(Appendix[D\.5](https://arxiv.org/html/2610.27393#A4.SS5)\)\.

A modest pilot captures most gains\. Pilots of 32 or 64 improve on 16, and 128 adds no consistent gain \(Figure[4](https://arxiv.org/html/2610.27393#S4.F4)a; Appendix Table[10](https://arxiv.org/html/2610.27393#A4.T10)\)\. Every tested pilot size lowers MAE on all three Gemma tasks, leaving flexibility in how the budget is divided between pilot measurements and exact verification\. The median beats the mean on identical labels in all nine settings; uniform and magnitude\-stratified samples are similar on Gemma \(Figure[4](https://arxiv.org/html/2610.27393#S4.F4)b; Appendix[D\.4](https://arxiv.org/html/2610.27393#A4.SS4)\)\.

### 4\.3Across\-Prompt Learning

We compare across\-promptGuided\-APwithProbeCal\(Section[4\.1](https://arxiv.org/html/2610.27393#S4.SS1)\)\. We assess verification against verify\-more at matched total cost, and dataset\-average correction against uncorrected AtP, without evaluation measurements for the latter\.

#### Verification allocation\.

Allocation needs transfer and reuse\. On Qwen3 IOI,Guided\-APlowers MAE by 12% against verify\-more at matched projected total cost, calibration included \(Table[3](https://arxiv.org/html/2610.27393#S4.T3)\)\.

Table 3:Across\-prompt MAE reductions relative to verify\-more at matched projected total cost, calibration included, for 10,000 prompts\. Bold and italic mark adjusted significant improvements and harms\.Where learning pays, projected break\-even at the smallest budget ranges from 1,900 to 24,000 prompts across the two calibration signals\. Shorter analyses favor additional verification; verify\-more also remains strong on factual completion at small budgets and on Qwen2\.5 IOI \(Appendices[G](https://arxiv.org/html/2610.27393#A7)and[I](https://arxiv.org/html/2610.27393#A9)\)\.

#### Dataset\-average correction\.

Persistent average errors can be corrected without measuring each new prompt\. On factual completion,Guided\-APlowers dataset\-average MAE by 75% to 79% on both models relative to uncorrected AtP \(Figure[5](https://arxiv.org/html/2610.27393#S4.F5); Appendix Table[15](https://arxiv.org/html/2610.27393#A6.T15)\)\. Post hoc cost comparisons give projected payback of 500 to 2,700 prompts where correction helps; shorter analyses favor verify\-more \(Appendix[H](https://arxiv.org/html/2610.27393#A8)\)\.

Learning residuals makes better use of calibration measurements\. Directly averaging the same AP effects produces 5\.8 to 14\.9 times more error than residual correction in every setting \(Appendix[H](https://arxiv.org/html/2610.27393#A8)\)\. On factual completion,Guided\-APlabels cover 13% to 18% of heads; the rest retain their AtP averages\.

Signed error transfer can fail\.Guided\-APraises Qwen2\.5 IOI dataset\-average MAE by 52%, consistent with selection bias \(Appendix[C\.4](https://arxiv.org/html/2610.27393#A3.SS4)\)\. SVA has no established correction gain, and neither correction establishes a dataset\-average recall gain\.

#### Calibration signals\.

ProbeCalhelps allocation where exact\-label learning is less effective, lowering factual MAE by 31% to 35% against verify\-more at matched projected total cost \(Table[3](https://arxiv.org/html/2610.27393#S4.T3)\)\. Its factual average correction reduces MAE by 46% against uncorrected AtP, less thanGuided\-AP\(Figure[5](https://arxiv.org/html/2610.27393#S4.F5)\); it helps Qwen2\.5 IOI averages whereGuided\-APharms them \(Appendix Table[15](https://arxiv.org/html/2610.27393#A6.T15)\)\.

#### Error structure and recall\.

Verification and average correction require different error information\. Factual errors recur \(Appendix[E](https://arxiv.org/html/2610.27393#A5)\), yet learned selection can lose at small budgets because magnitude ranking already captures the largest residuals\. SVA improves verification without an established mean\-correction gain\. Lower MAE also need not improve top\-ten recall \(Appendix[G](https://arxiv.org/html/2610.27393#A7)\)\.

Figure 5:Dataset\-average correction on the two models\. Ratios are uncorrected over corrected MAE, with adjusted intervals; the dotted line denotes equality\.

### 4\.4Scalability and Efficiency

We separate accuracy gains at a fixed measurement budget from runtime savings at an error target\.

#### Component count and accuracy\.

With the measurement budget fixed, the benefit persists at every tested component count across all three Gemma tasks, up to 14,336 neurons \(Appendix[D\.6](https://arxiv.org/html/2610.27393#A4.SS6)\)\.

Runtime savings depend on the required accuracy\. At a 1% mean relative AP error target,Guided\-APis 7\.9 times faster than verify\-more on Gemma IOI in measured warm runtime\. This runtime includes attribution, pilot measurements, and fitting on the same prompt; the gain requires no reuse across prompts\. Verification is cheaper on SVA, andGuided\-APmisses the factual target \(Table[4](https://arxiv.org/html/2610.27393#S4.T4); Appendix[D\.2](https://arxiv.org/html/2610.27393#A4.SS2)\)\.

Table 4:Warm cost at a 1% mean relative AP error target on Gemma\-2\-9B\. Italic errors exceed the target; checks count magnitude\-ranked AP measurements\.
#### Model size\.

The same budget yields 11% to 24% lower MAE on Qwen3\-14B and 14% to 18% on Qwen3\-32B at matched runtime, measuring only 1% of the latter’s neurons \(Table[1](https://arxiv.org/html/2610.27393#S4.T1); Figure[6](https://arxiv.org/html/2610.27393#S4.F6)a\)\. AtGuided\-AP’s achieved error, a post hoc comparison finds verify\-more takes 5\.9 to 14\.7 times longer in warm runtime across the three 32B tasks \(Figure[6](https://arxiv.org/html/2610.27393#S4.F6)b; Appendix Table[6](https://arxiv.org/html/2610.27393#A4.T6)\)\. This frontier comparison uses the same prompts and matches the correction’s observed error, separately from the prespecified targets\.

#### Comparison with derivative corrections\.

On the 30\-group Gemma comparison set,Guided\-APhas lower MAE than selective HVP, MS\-HVP, and singleton IG at their tested settings, taking about one to three warm seconds per layer \(Figure[3](https://arxiv.org/html/2610.27393#S4.F3); Appendix[D\.3](https://arxiv.org/html/2610.27393#A4.SS3)\)\. AtP remains fastest; exhaustive AP removes approximation error at greater cost\.

Figure 6:Scaling of the fixed budget\. Left, SVA MAE reductions at matched cost with descriptive intervals\. Right, measured 32B verification curves for factual completion and SVA \(lines\), within\-promptGuided\-AP\(stars\), and verification at the same error \(open circles\), on the same complete groups; IOI: Appendix Table[6](https://arxiv.org/html/2610.27393#A4.T6)\.

## 5Conclusion and Limitations

Attribution patching trades the accuracy of activation patching for speed\.Guided\-APuses a few exact measurements to improve estimates beyond the ones it verifies\. On a single prompt, the median error of a small exact sample corrects the unmeasured estimates, as long as enough budget remains to verify the largest ones\. Over many prompts of one task, a few measured prompts reveal each component’s recurring error\. Its size guides verification on later prompts, and its direction corrects average effects, which repays the cost of learning in long analyses\. Future work could combine the two settings and test them on sparse features, where activation patching is most costly\.

Our evaluation focuses on transformer language models and controlled language tasks\. Further evaluation on other architectures, component representations, and open\-ended language behaviors would clarify its broader applicability\. Accuracy and computational costs depend on the task, measurement budget, hardware, and implementation\. Activation patching measures effects relative to a chosen intervention and output metric\. These estimates describe individual components and can inform mechanistic explanations alongside tests of how components work together\.

## References

- Aradet al\.\(2025\)D\. Arad, Y\. Belinkov, H\. Chen, N\. Kim, H\. Mohebbi, A\. Mueller, G\. Sarti, and M\. TutekFindings of the BlackboxNLP 2025 shared task: localizing circuits and causal variables in language models\.InProceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, A\. Mueller, N\. Kim, H\. Mohebbi, H\. Chen, D\. Arad, and G\. Sarti \(Eds\.\),Suzhou, China,pp\. 543–552\.External Links:[Link](https://aclanthology.org/2025.blackboxnlp-1.32/),[Document](https://dx.doi.org/10.18653/v1/2025.blackboxnlp-1.32),ISBN 979\-8\-89176\-346\-3Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Baston and Nakatsukasa \(2022\)R\. A\. Baston and Y\. NakatsukasaStochastic diagonal estimation: probabilistic bounds and an improved algorithm\.External Links:2201\.10684,[Link](https://arxiv.org/abs/2201.10684)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Bekaset al\.\(2007\)C\. Bekas, E\. Kokiopoulou, and Y\. SaadAn estimator for the diagonal of a matrix\.Applied Numerical Mathematics57\(11\-12\),pp\. 1214–1229\.External Links:ISSN 0168\-9274,[Link](http://dx.doi.org/10.1016/j.apnum.2007.01.003),[Document](https://dx.doi.org/10.1016/j.apnum.2007.01.003)Cited by:[Appendix B](https://arxiv.org/html/2610.27393#A2.SS0.SSS0.Px1.p1.2),[Appendix B](https://arxiv.org/html/2610.27393#A2.p1.2),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.27393#S4.SS1.SSS0.Px2.p1.1)\.
- Cohnet al\.\(1994\)D\. Cohn, L\. Atlas, and R\. LadnerImproving generalization with active learning\.Machine Learning15\(2\),pp\. 201–221\.External Links:ISSN 1573\-0565,[Link](http://dx.doi.org/10.1007/BF00993277),[Document](https://dx.doi.org/10.1007/bf00993277)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Conmyet al\.\(2023\)A\. Conmy, A\. Mavor\-Parker, A\. Lynch, S\. Heimersheim, and A\. Garriga\-AlonsoTowards automated circuit discovery for mechanistic interpretability\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 16318–16352\.External Links:[Document](https://dx.doi.org/10.52202/075280-0719),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Edinet al\.\(2025\)J\. Edin, C\. L\. Christensen, R\. Csordás, T\. Ruotsalo, Z\. Wu, M\. Maistro, J\. Huang, and L\. MaaløeCorrecting gradient\-based circuit localization via interaction\-aware backpropagation\.External Links:2505\.17630,[Link](https://arxiv.org/abs/2505.17630)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Epperlyet al\.\(2024\)E\. N\. Epperly, J\. A\. Tropp, and R\. J\. WebberXTrace: making the most of every sample in stochastic trace estimation\.SIAM Journal on Matrix Analysis and Applications45\(1\),pp\. 1–23\.External Links:ISSN 1095\-7162,[Link](http://dx.doi.org/10.1137/23M1548323),[Document](https://dx.doi.org/10.1137/23m1548323)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\. 4878–4887\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Gemma Team \(2024\)Gemma TeamGemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[§4\.1](https://arxiv.org/html/2610.27393#S4.SS1.SSS0.Px1.p1.1)\.
- Grosseet al\.\(2023\)R\. Grosse, J\. Bae, C\. Anil, N\. Elhage, A\. Tamkin, A\. Tajdini, B\. Steiner, D\. Li, E\. Durmus, E\. Perez, E\. Hubinger, K\. Lukošiūtė, K\. Nguyen, N\. Joseph, S\. McCandlish, J\. Kaplan, and S\. R\. BowmanStudying large language model generalization with influence functions\.External Links:2308\.03296,[Link](https://arxiv.org/abs/2308.03296)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Hannaet al\.\(2024\)M\. Hanna, S\. Pezzelle, and Y\. BelinkovHave faith in faithfulness: going beyond circuit overlap when finding model mechanisms\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=TZ0CCGDcuT)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Hoeffding \(1963\)W\. HoeffdingProbability inequalities for sums of bounded random variables\.Journal of the American Statistical Association58\(301\),pp\. 13–30\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1963.10500830)Cited by:[§C\.1](https://arxiv.org/html/2610.27393#A3.SS1.p6.1)\.
- Hutchinson \(1989\)M\.F\. HutchinsonA stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines\.Communications in Statistics \- Simulation and Computation18\(3\),pp\. 1059–1076\.External Links:ISSN 1532\-4141,[Link](http://dx.doi.org/10.1080/03610918908812806),[Document](https://dx.doi.org/10.1080/03610918908812806)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Jafariet al\.\(2025\)F\. R\. Jafari, O\. Eberle, A\. Khakzar, and N\. NandaRelP: faithful and efficient circuit discovery in language models via relevance patching\.Note:NeurIPS 2025 Mechanistic Interpretability WorkshopExternal Links:2508\.21258,[Link](https://arxiv.org/abs/2508.21258)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Janizeket al\.\(2021\)J\. D\. Janizek, P\. Sturmfels, and S\. LeeExplaining explanations: axiomatic feature interactions for deep networks\.Journal of Machine Learning Research22\(104\),pp\. 1–54\.External Links:[Link](http://jmlr.org/papers/v22/20-1223.html)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Koenker and Bassett \(1978\)R\. Koenker and G\. BassettRegression quantiles\.Econometrica46\(1\),pp\. 33–50\.External Links:ISSN 0012\-9682,[Link](http://dx.doi.org/10.2307/1913643),[Document](https://dx.doi.org/10.2307/1913643)Cited by:[§3\.2](https://arxiv.org/html/2610.27393#S3.SS2.SSS0.Px1.p1.2)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. LiangUnderstanding black\-box predictions via influence functions\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1885–1894\.External Links:[Link](https://proceedings.mlr.press/v70/koh17a.html)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Kramáret al\.\(2024\)J\. Kramár, T\. Lieberum, R\. Shah, and N\. NandaAtP\*: an efficient and scalable method for localizing LLM behaviour to components\.External Links:2403\.00745,[Link](https://arxiv.org/abs/2403.00745)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Lewis and Gale \(1994\)D\. D\. Lewis and W\. A\. GaleA sequential algorithm for training text classifiers\.InSIGIR ’94: Proceedings of the Seventeenth Annual International ACM\-SIGIR Conference on Research and Development in Information Retrieval,W\. B\. Croft and C\. J\. van Rijsbergen \(Eds\.\),pp\. 3–12\.External Links:ISBN 9781447120995,[Link](http://dx.doi.org/10.1007/978-1-4471-2099-5_1),[Document](https://dx.doi.org/10.1007/978-1-4471-2099-5%5F1)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Markset al\.\(2025\)S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. MuellerSparse feature circuits: discovering and editing interpretable causal graphs in language models\.InThe Thirteenth International Conference on Learning Representations,pp\. 23888–23923\.External Links:[Link](https://openreview.net/forum?id=I4e82CIDxv)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Muelleret al\.\(2025\)A\. Mueller, A\. Geiger, S\. Wiegreffe, D\. Arad, I\. Arcuschin, A\. Belfki, Y\. S\. Chan, J\. F\. Fiotto\-Kaufman, T\. Haklay, M\. Hanna, J\. Huang, R\. Gupta, Y\. Nikankin, H\. Orgad, N\. Prakash, A\. Reusch, A\. Sankaranarayanan, S\. Shao, A\. Stolfo, M\. Tutek, A\. Zur, D\. Bau, and Y\. BelinkovMIB: a mechanistic interpretability benchmark\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 45069–45108\.External Links:[Link](https://proceedings.mlr.press/v267/mueller25a.html)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Nanda \(2023\)N\. NandaAttribution patching: activation patching at industrial scale\.Note:[https://www\.neelnanda\.io/mechanistic\-interpretability/attribution\-patching](https://www.neelnanda.io/mechanistic-interpretability/attribution-patching)Cited by:[§1](https://arxiv.org/html/2610.27393#S1.p2.1),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Pearlmutter \(1994\)B\. A\. PearlmutterFast exact multiplication by the Hessian\.Neural Computation6\(1\),pp\. 147–160\.External Links:ISSN 1530\-888X,[Link](http://dx.doi.org/10.1162/neco.1994.6.1.147),[Document](https://dx.doi.org/10.1162/neco.1994.6.1.147)Cited by:[Appendix B](https://arxiv.org/html/2610.27393#A2.p1.2),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Qwenet al\.\(2025\)Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Appendix F](https://arxiv.org/html/2610.27393#A6.p1.1),[§4\.1](https://arxiv.org/html/2610.27393#S4.SS1.SSS0.Px1.p1.1)\.
- Sundararajanet al\.\(2017\)M\. Sundararajan, A\. Taly, and Q\. YanAxiomatic attribution for deep networks\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 3319–3328\.External Links:[Link](https://proceedings.mlr.press/v70/sundararajan17a.html)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1)\.
- Syedet al\.\(2024\)A\. Syed, C\. Rager, and A\. ConmyAttribution patching outperforms automated circuit discovery\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 407–416\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.25/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.25)Cited by:[§1](https://arxiv.org/html/2610.27393#S1.p2.1),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NpsVSN6o4ul)Cited by:[§1](https://arxiv.org/html/2610.27393#S1.p1.1),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix F](https://arxiv.org/html/2610.27393#A6.p1.1),[§4\.1](https://arxiv.org/html/2610.27393#S4.SS1.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2021\)Z\. Yao, A\. Gholami, S\. Shen, M\. Mustafa, K\. Keutzer, and M\. MahoneyADAHESSIAN: an adaptive second order optimizer for machine learning\.Proceedings of the AAAI Conference on Artificial Intelligence35\(12\),pp\. 10665–10673\.External Links:ISSN 2159\-5399,[Link](http://dx.doi.org/10.1609/aaai.v35i12.17275),[Document](https://dx.doi.org/10.1609/aaai.v35i12.17275)Cited by:[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px3.p1.1)\.
- Zhang and Nanda \(2024\)F\. Zhang and N\. NandaTowards best practices of activation patching in language models: metrics and methods\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Hf17y6u9BC)Cited by:[§1](https://arxiv.org/html/2610.27393#S1.p1.1),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Wang \(2026\)L\. Zhang and J\. WangWhen attribution patching lies: diagnosis and a second\-order correction\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2606.09899)Cited by:[§1](https://arxiv.org/html/2610.27393#S1.p2.1),[§2](https://arxiv.org/html/2610.27393#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.27393#S3.SS1.p3.1)\.

## Appendix ALabel Acquisition and Fits

This appendix specifies label acquisition and the fitted correctionf^\\widehat\{f\}and size modelh^\\widehat\{h\}\. Acquired AP values and residuals are writtenΔ~i\\widetilde\{\\Delta\}\_\{i\}ande~i=Δ~i−gi\\widetilde\{e\}\_\{i\}=\\widetilde\{\\Delta\}\_\{i\}\-g\_\{i\}\. They equal the scoring referenceΔi\\Delta\_\{i\}and residualeie\_\{i\}in the primary analysis; Appendix[D\.5](https://arxiv.org/html/2610.27393#A4.SS5)distinguishes them for FP64 rescoring\.

### A\.1Within\-Prompt Learning

Pilot\.Sort components by increasing\|gi\|\|g\_\{i\}\|, breaking ties by component order, and partition them intoK=8K=8equal\-sized rank strata\. Independently permute each stratum and take one component per stratum in round\-robin order until the pilot is full \(Equation[5](https://arxiv.org/html/2610.27393#S3.E5)\)\. All evaluated populations are divisible by eight\.

Correction and selection\.Fit one offset, the median acquired pilot residual \(Equation[6](https://arxiv.org/html/2610.27393#S3.E6)\)\. Seth^=\|gp′​i\|\\widehat\{h\}=\|g\_\{p^\{\\prime\}i\}\|and use the remainingB−\|S1\|B\-\|S\_\{1\}\|measurements on the largest magnitudes outside the pilot\. Runtime includes both measurement sets, selection, fitting, and estimate assembly\.

### A\.2Across\-Prompt Learning

Label acquisition\.On each calibration prompt, sort components by decreasing\|g\|\|g\|\. Visit prompts in fixed round\-robin order, taking the next unacquired component until the label budget is exhausted\.

Size modelh^\\widehat\{h\}\.Each acquired label suppliesx=log⁡\(\|g\|\+η\)x=\\log\(\|g\|\+\\eta\)andy=log⁡\(\|e~−c^i\|\+η\)y=\\log\(\|\\widetilde\{e\}\-\\widehat\{c\}\_\{i\}\|\+\\eta\)withη=10−12\\eta=10^\{\-12\}, wherec^i\\widehat\{c\}\_\{i\}is the correction fit below; withf^=0\\widehat\{f\}=0, as in our experiments,y=log⁡\(\|e~\|\+η\)y=\\log\(\|\\widetilde\{e\}\|\+\\eta\)\. A pooled least\-squares fit gives slopeaaand interceptβ\\beta; componentiithen receives its own interceptβi=β\+wi​\(meani⁡\[y−a​x\]−β\)\\beta\_\{i\}=\\beta\+w\_\{i\}\(\\operatorname\{mean\}\_\{i\}\[y\-ax\]\-\\beta\)withwi=ni/\(ni\+5\)w\_\{i\}=n\_\{i\}/\(n\_\{i\}\+5\), wherenin\_\{i\}is its number of labels, and the pooled intercept when it has none\. The score is

h^​\(g,i\)=exp⁡\(a​log⁡\(\|g\|\+η\)\+βi\),\\widehat\{h\}\(g,i\)=\\exp\\big\(a\\log\(\|g\|\+\\eta\)\+\\beta\_\{i\}\\big\),a log\-linear model fit by least squares in log space as a surrogate for the absolute\-loss objective of Equation[3](https://arxiv.org/html/2610.27393#S3.E3)\. The score approximates expected remaining errorrir\_\{i\}, the target of Proposition[1](https://arxiv.org/html/2610.27393#Thmproposition1)\.

Correctionf^\\widehat\{f\}\.Equation[3](https://arxiv.org/html/2610.27393#S3.E3)restricted to one constant per component is minimized by the median of each component’s acquired residuals\. Our verification experiments setf^=0\\widehat\{f\}=0\(Section[4\.1](https://arxiv.org/html/2610.27393#S4.SS1)\)\.

Dataset averages\.Let𝒞i\\mathcal\{C\}\_\{i\}be the calibration prompts on which componentiiwas measured\. Its average correction is the mean acquired residual,

c¯i=\{\|𝒞i\|−1​∑p∈𝒞iep​i,\|𝒞i\|\>0,0,\|𝒞i\|=0,\\bar\{c\}\_\{i\}=\\begin\{cases\}\|\\mathcal\{C\}\_\{i\}\|^\{\-1\}\\sum\_\{p\\in\\mathcal\{C\}\_\{i\}\}e\_\{pi\},&\|\\mathcal\{C\}\_\{i\}\|\>0,\\\\ 0,&\|\\mathcal\{C\}\_\{i\}\|=0,\\end\{cases\}\(9\)with no shrinkage, and the corrected average isg¯i\+c¯i\\bar\{g\}\_\{i\}\+\\bar\{c\}\_\{i\}\. The target is the average residual over prompts, so it is estimated by an average; Lemma 2 states when such a correction helps\. For probes, Equation[10](https://arxiv.org/html/2610.27393#A2.E10)plays the same role\.

## Appendix BCurvature\-Probe Calibration

ProbeCalsupplies an alternative learning signal through shared curvature probes\. WhenFpF\_\{p\}is twice differentiable at the baseline, letGp=∇2Fp​\(0\)G\_\{p\}=\\nabla^\{2\}F\_\{p\}\(0\)be its Hessian in intervention coordinates for calibration promptpp\. With independent random sign vectorszp​r∈\{−1,1\}Nz\_\{pr\}\\in\\\{\-1,1\\\}^\{N\}, its sketch is

dp​i=1R​∑r=1Rzp​r,i​\(Gp​zp​r\)i\.d\_\{pi\}=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}z\_\{pr,i\}\(G\_\{p\}z\_\{pr\}\)\_\{i\}\.The randomized diagonal estimator of[Bekas et al\. \(2007\)](https://arxiv.org/html/2610.27393#bib.bib3)is computed with Hessian\-vector products\([Pearlmutter, 1994](https://arxiv.org/html/2610.27393#bib.bib21)\)\. Its expectation is baseline curvature; Equation[11](https://arxiv.org/html/2610.27393#A2.E11)gives its interaction\-dependent noise, and Equation[12](https://arxiv.org/html/2610.27393#A2.E12)distinguishes curvature from the finite AP residual\. ForJJcalibration prompts withRRprobes each, the signed average correction is

c¯iprobe=12​J​∑p=1Jdp​i,\\bar\{c\}\_\{i\}^\{\\rm probe\}=\\frac\{1\}\{2J\}\\sum\_\{p=1\}^\{J\}d\_\{pi\},\(10\)and the corrected average isg¯i\+c¯iprobe\\bar\{g\}\_\{i\}\+\\bar\{c\}\_\{i\}^\{\\rm probe\}\. Averaging retains the signs; a sketch estimates baseline curvature, not the complete finite\-AP residual\.ProbeCaluses the sketches as labels for the size model\. Every component\-prompt pair suppliesxp​i=log⁡\(\|gp​i\|\+η\)x\_\{pi\}=\\log\(\|g\_\{pi\}\|\+\\eta\)andyp​i=log⁡\(\|dp​i\|\+η\)y\_\{pi\}=\\log\(\|d\_\{pi\}\|\+\\eta\), wheredp​id\_\{pi\}averages the signed probes before the magnitude is taken\. A pooled least\-squares fit gives\(a,β\)\(a,\\beta\); a component with varying log attribution across prompts also receives its own pair\(ai,βi\)\(a\_\{i\},\\beta\_\{i\}\), shrunk toward the pooled values, and its score ish^​\(g,i\)=exp⁡\(ai​log⁡\(\|g\|\+η\)\+βi\)\\widehat\{h\}\(g,i\)=\\exp\(a\_\{i\}\\log\(\|g\|\+\\eta\)\+\\beta\_\{i\}\)\. A capacity\-matched AP\-only control, fit the same way on exact labels, separates the label source from the model class\. Probes are computed only on calibration prompts; no probe is run on evaluation prompts\.

#### Probe noise and the finite\-intervention remainder\.

For a fixed HessianGGand independent sign vectors,zi​\(G​z\)i=Gi​i\+∑j≠iGi​j​zi​zjz\_\{i\}\(Gz\)\_\{i\}=G\_\{ii\}\+\\sum\_\{j\\neq i\}G\_\{ij\}z\_\{i\}z\_\{j\}, so conditional onGG\([Bekas et al\., 2007](https://arxiv.org/html/2610.27393#bib.bib3)\)

𝔼⁡\[di∣G\]=Gi​i,Var⁡\(di∣G\)=1R​∑j≠iGi​j2,\\mathbb\{E\}\[d\_\{i\}\\mid G\]=G\_\{ii\},\\qquad\\operatorname\{Var\}\(d\_\{i\}\\mid G\)=\\frac\{1\}\{R\}\\sum\_\{j\\neq i\}G\_\{ij\}^\{2\},\(11\)andCov⁡\(di,dk∣G\)=Gi​k2/R\\operatorname\{Cov\}\(d\_\{i\},d\_\{k\}\\mid G\)=G\_\{ik\}^\{2\}/Rfori≠ki\\neq k\. Interactions between components set the noise of each sketch and correlate sketches across components, so they are not independent samples\. The sketch also targets only the curvature term of the residual\. For a twice differentiable pathfi​\(t\)=F⁡\(t​vi\)f\_\{i\}\(t\)=F\(tv\_\{i\}\), Taylor’s theorem gives

ei\\displaystyle e\_\{i\}=12​Gi​i\+ρi,\\displaystyle=\\tfrac\{1\}\{2\}G\_\{ii\}\+\\rho\_\{i\},\(12\)ρi\\displaystyle\\rho\_\{i\}=∫01\(1−t\)​\{fi′′​\(t\)−fi′′​\(0\)\}​dt,\\displaystyle=\\int\_\{0\}^\{1\}\(1\-t\)\\\{f\_\{i\}^\{\\prime\\prime\}\(t\)\-f\_\{i\}^\{\\prime\\prime\}\(0\)\\\}\\,dt,Thus the sketches supply a curvature\-based learning signal rather than exact finite\-intervention residual labels\.

## Appendix CMathematical Details

### C\.1Proof of Theorem[1](https://arxiv.org/html/2610.27393#Thmtheorem1)

We extend the identity to the runtime\-matched baseline of the experiments, which may measureB0≠BB\_\{0\}\\neq Bcomponents, and acquired AP valuesΔ~i\\widetilde\{\\Delta\}\_\{i\}that may differ from the referenceΔi\\Delta\_\{i\}\. FixggandΔ\\Deltaand lete=Δ−ge=\\Delta\-g\. The method measuresSSwith\|S\|=B\|S\|=Band shifts the unmeasured setUUbyc^\\widehat\{c\}; the baseline measuresVB0V\_\{B\_\{0\}\}, theB0B\_\{0\}largest\|gi\|\|g\_\{i\}\|, and leaves the rest atgig\_\{i\}\. LetDacq=∑i∈S\|Δ~i−Δi\|−∑i∈VB0\|Δ~i0−Δi\|D\_\{\\rm acq\}=\\sum\_\{i\\in S\}\|\\widetilde\{\\Delta\}\_\{i\}\-\\Delta\_\{i\}\|\-\\sum\_\{i\\in V\_\{B\_\{0\}\}\}\|\\widetilde\{\\Delta\}^\{0\}\_\{i\}\-\\Delta\_\{i\}\|be the difference in acquisition error, andC=∑i∈VB0\|ei\|−∑i∈S\|ei\|C=\\sum\_\{i\\in V\_\{B\_\{0\}\}\}\|e\_\{i\}\|\-\\sum\_\{i\\in S\}\|e\_\{i\}\|the verification opportunity cost\. For the decomposition reported in Appendix[D](https://arxiv.org/html/2610.27393#A4),CCsplits into an allocation term and a count term,

Calloc\\displaystyle C\_\{\\rm alloc\}=∑i∈VB\|ei\|−∑i∈S\|ei\|,\\displaystyle=\\sum\_\{i\\in V\_\{B\}\}\|e\_\{i\}\|\-\\sum\_\{i\\in S\}\|e\_\{i\}\|,Ccount\\displaystyle C\_\{\\rm count\}=∑i∈VB0\|ei\|−∑i∈VB\|ei\|,\\displaystyle=\\sum\_\{i\\in V\_\{B\_\{0\}\}\}\|e\_\{i\}\|\-\\sum\_\{i\\in V\_\{B\}\}\|e\_\{i\}\|,which isolate the change in whichBBcomponents are measured and the extra or fewer measurements affordable at matched runtime\.

The identity\.The summed errors of the two estimates are

L0\\displaystyle L\_\{0\}=∑i∉VB0\|ei\|\+∑i∈VB0\|Δ~i0−Δi\|,\\displaystyle=\\sum\_\{i\\notin V\_\{B\_\{0\}\}\}\|e\_\{i\}\|\+\\sum\_\{i\\in V\_\{B\_\{0\}\}\}\|\\widetilde\{\\Delta\}^\{0\}\_\{i\}\-\\Delta\_\{i\}\|,L1\\displaystyle L\_\{1\}=AU​\(c^\)\+∑i∈S\|Δ~i−Δi\|\.\\displaystyle=A\_\{U\}\(\\widehat\{c\}\)\+\\sum\_\{i\\in S\}\|\\widetilde\{\\Delta\}\_\{i\}\-\\Delta\_\{i\}\|\.Subtracting,L0−L1=AU​\(0\)−AU​\(c^\)−C−DacqL\_\{0\}\-L\_\{1\}=A\_\{U\}\(0\)\-A\_\{U\}\(\\widehat\{c\}\)\-C\-D\_\{\\rm acq\}, and adding and subtractingAU​\(mU\)A\_\{U\}\(m\_\{U\}\)gives

N⁡\(MAE0−MAE1\)=HU−EU−C−Dacq\.N\(\\operatorname\{MAE\}\_\{0\}\-\\operatorname\{MAE\}\_\{1\}\)=H\_\{U\}\-E\_\{U\}\-C\-D\_\{\\rm acq\}\.SinceAUA\_\{U\}is convex with a minimum at any median,HU,EU≥0H\_\{U\},E\_\{U\}\\geq 0\. When acquired values equal the reference,Dacq=0D\_\{\\rm acq\}=0and Equation[7](https://arxiv.org/html/2610.27393#S3.E7)follows\.□\\square

The estimation cost\.The bound onEUE\_\{U\}uses one fact about convex functions\.

Lemma 1\.Ifc^∈\[a,b\]\\widehat\{c\}\\in\[a,b\], thenAU​\(c^\)≤max⁡\{AU​\(a\),AU​\(b\)\}A\_\{U\}\(\\widehat\{c\}\)\\leq\\max\\\{A\_\{U\}\(a\),A\_\{U\}\(b\)\\\}\.

Proof\.AUA\_\{U\}is convex, and every point of\[a,b\]\[a,b\]is a convex combination of its endpoints\.□\\square

Letqqbe the quantile function of theNNresiduals andnnthe pilot size\. The pilot is drawn without replacement, equally fromKKequal\-sized strata \(Equation[5](https://arxiv.org/html/2610.27393#S3.E5)\), with independent draws across strata, so the number of pilot residuals belowq⁡\(12−t\)q\(\\tfrac\{1\}\{2\}\-t\)is a sum over strata of independent without\-replacement counts with mean at mostn⁡\(12−t\)n\(\\tfrac\{1\}\{2\}\-t\)\. Hoeffding’s inequality holds for sampling without replacement and for independent sums of such counts\([Hoeffding, 1963](https://arxiv.org/html/2610.27393#bib.bib31)\), so the pilot median falls belowq⁡\(12−t\)q\(\\tfrac\{1\}\{2\}\-t\)with probability at moste−2​n​t2e^\{\-2nt^\{2\}\}, and symmetrically aboveq⁡\(12\+t\)q\(\\tfrac\{1\}\{2\}\+t\)\. On the complementary event, Lemma 1 with\[a,b\]=\[q⁡\(12−t\),q⁡\(12\+t\)\]\[a,b\]=\[q\(\\tfrac\{1\}\{2\}\-t\),q\(\\tfrac\{1\}\{2\}\+t\)\]gives

EU≤max⁡\{AU​\(a\),AU​\(b\)\}−AU​\(mU\)≕εU​\(t\),E\_\{U\}\\leq\\max\\\{A\_\{U\}\(a\),A\_\{U\}\(b\)\\\}\-A\_\{U\}\(m\_\{U\}\)\\eqqcolon\\varepsilon\_\{U\}\(t\),the excess error of the worst shift in the band\. The probability statement concerns only the pilot median against the quantiles of allNNresiduals; Lemma 1 then applies to whichever unmeasured setUUthe pilot and verification leave\.□\\square

### C\.2Proof of Theorem[2](https://arxiv.org/html/2610.27393#Thmtheorem2)

The theorem assumesnnindependent labels from the component’s residual law and𝔼​\|e\|<∞\\mathbb\{E\}\|e\|<\\infty\. Labels acquired by attribution magnitude need not satisfy this, and the band is undefined fort≥12t\\geq\\tfrac\{1\}\{2\}\. Lete1,…,ene\_\{1\},\\ldots,e\_\{n\}be the labels andc^\\widehat\{c\}their median\.

*Step 1: sample median in the central band\.*Ifc^<q⁡\(12−t\)\\widehat\{c\}<q\(\\tfrac\{1\}\{2\}\-t\), at leastn/2n/2labels fall belowq⁡\(12−t\)q\(\\tfrac\{1\}\{2\}\-t\)\. Their number is binomial with success probability at most12−t\\tfrac\{1\}\{2\}\-t, so Hoeffding’s inequality bounds this event bye−2​n​t2e^\{\-2nt^\{2\}\}; similarly forc^\>q⁡\(12\+t\)\\widehat\{c\}\>q\(\\tfrac\{1\}\{2\}\+t\)\.

*Step 2: excess error inside the band\.*R⁡\(c\)=𝔼​\|e−c\|R\(c\)=\\mathbb\{E\}\|e\-c\|is convex with derivative2​Φ​\(c\)−12\\Phi\(c\)\-1almost everywhere, whereΦ\\Phiis the distribution function ofee, so forc^≥m\\widehat\{c\}\\geq m,

R⁡\(c^\)−R⁡\(m\)\\displaystyle R\(\\widehat\{c\}\)\-R\(m\)=∫mc^\(2​Φ​\(u\)−1\)​𝑑u\\displaystyle=\\int\_\{m\}^\{\\widehat\{c\}\}\\bigl\(2\\Phi\(u\)\-1\\bigr\)\\,du≤2​t​\(c^−m\)≤2​t​w​\(t\),\\displaystyle\\leq 2t\\,\(\\widehat\{c\}\-m\)\\leq 2t\\,w\(t\),since12≤Φ⁡\(u\)≤12\+t\\tfrac\{1\}\{2\}\\leq\\Phi\(u\)\\leq\\tfrac\{1\}\{2\}\+ton the band\. The casec^<m\\widehat\{c\}<mis symmetric\.

*Step 3\.*SinceR⁡\(m\)=R⁡\(0\)−ΓR\(m\)=R\(0\)\-\\Gamma, the two steps give Equation[8](https://arxiv.org/html/2610.27393#S3.E8)\. A union bound extends the event to allNNcomponents with probability at least1−2​N​e−2​n​t21\-2Ne^\{\-2nt^\{2\}\}\.□\\square

### C\.3Proof of Proposition[1](https://arxiv.org/html/2610.27393#Thmproposition1)

LetTTbe theBBcomponents with the largestrir\_\{i\}andSStheBBwith the largest scoressis\_\{i\}\. The expected error left bySSexceeds that left byTTby∑i∈Tri−∑i∈Sri\\sum\_\{i\\in T\}r\_\{i\}\-\\sum\_\{i\\in S\}r\_\{i\}\. Sinceri≤si\+ϵr\_\{i\}\\leq s\_\{i\}\+\\epsilononTTandri≥si−ϵr\_\{i\}\\geq s\_\{i\}\-\\epsilononSS,

∑i∈Tri−∑i∈Sri≤∑i∈Tsi−∑i∈Ssi\+2​B​ϵ≤2​B​ϵ,\\sum\_\{i\\in T\}r\_\{i\}\-\\sum\_\{i\\in S\}r\_\{i\}\\leq\\sum\_\{i\\in T\}s\_\{i\}\-\\sum\_\{i\\in S\}s\_\{i\}\+2B\\epsilon\\leq 2B\\epsilon,becauseSSmaximizes the score sum over sets of sizeBB\.□\\square

The oracle knows expected remaining errors before measurement\. A residual that changes sign can have a small fitted correction but large expected absolute error, so a correction alone need not provide a useful selection score\.

### C\.4Dataset Averages: Sign Condition and Selected Labels

Lemma 2 \(Sign condition\)\.For an averaged residuale¯i\\bar\{e\}\_\{i\}and correctionc¯i\\bar\{c\}\_\{i\},\|e¯i−c¯i\|<\|e¯i\|\|\\bar\{e\}\_\{i\}\-\\bar\{c\}\_\{i\}\|<\|\\bar\{e\}\_\{i\}\|if and only ifc¯i​e¯i\>0\\bar\{c\}\_\{i\}\\bar\{e\}\_\{i\}\>0and\|c¯i\|<2​\|e¯i\|\|\\bar\{c\}\_\{i\}\|<2\|\\bar\{e\}\_\{i\}\|\.

Proof\.Squaring, the inequality is equivalent toc¯i2<2​c¯i​e¯i\\bar\{c\}\_\{i\}^\{2\}<2\\bar\{c\}\_\{i\}\\bar\{e\}\_\{i\}, which requiresc¯i​e¯i\>0\\bar\{c\}\_\{i\}\\bar\{e\}\_\{i\}\>0and then reduces to\|c¯i\|<2​\|e¯i\|\|\\bar\{c\}\_\{i\}\|<2\|\\bar\{e\}\_\{i\}\|\.□\\square

Dataset averages require the mean residual, whereas per\-prompt absolute loss is minimized by a median\. Acquisition can bias the mean\. IfAiA\_\{i\}indicates measurement of componentii, the acquired\-label mean targets𝔼⁡\[ei∣Ai=1\]\\mathbb\{E\}\[e\_\{i\}\\mid A\_\{i\}=1\]rather than𝔼⁡\[ei\]\\mathbb\{E\}\[e\_\{i\}\]\. Section[4\.3](https://arxiv.org/html/2610.27393#S4.SS3.SSS0.Px2)evaluates whether the acquired residuals transfer to dataset averages\.

## Appendix DWithin\-Prompt Evaluation and Controls

### D\.1Cohort, Acquisition, and Aggregation

Within\-prompt evaluation comprises 1,662 layer evaluations and 542 complete model\-task\-prompt groups, with a fixedP=64P=64pilot insideB=256B=256measurements\. We intervene on all MLP neurons in a layer\. Gemma\-2\-9B has 14,336 neurons \(layers 10, 20, 30\), Qwen3\-14B has 17,408 \(9, 19, 29\), and Qwen3\-32B has 25,600 \(15, 31, 47\)\. Each task has 100 prompt groups\. Summaries include groups with all three layers; Gemma is complete, and available Qwen counts appear in Table[1](https://arxiv.org/html/2610.27393#S4.T1)\.

Relative AP error divides summed absolute estimation error by summed absolute reference effects for each vector\. MAE, dataset\-average error, and top\-ten recall retain their definitions in Section[3\.1](https://arxiv.org/html/2610.27393#S3.SS1)\.

We average three fixed pilot repetitions within each layer, then layers within each prompt group, then groups\. Error reduction is one minus the ratio of mean MAEs\. Descriptive 95% intervals use 20,000 paired group\-bootstrap draws \(seed 20261010\), holding fits and measured costs fixed; across\-prompt multiplicity adjustment is separate\. Gemma intervals are 14\.7% to 20\.1% \(IOI\), 23\.5% to 28\.9% \(factual\), and 17\.4% to 22\.5% \(SVA\)\.

The method and baseline acquire separate AP measurements in the same allocation\. Warm runtime includes attribution, acquisition, selection, fitting, and assembly, excluding loading, numerical validation, and reference collection\. The baseline uses the largest executed magnitude\-ranked batch prefix affordable at the method’s runtime; equal\-count comparisons are separate\. Gemma, Qwen3\-14B, and Qwen3\-32B use one, two, and four L40S GPUs, respectively, with timings matched within each hardware configuration\.

### D\.2Measured Cost at Specified Accuracy

The prespecified targets are 0\.5%, 1%, and 2% mean relative AP error\. On the main cohorts \(Table[1](https://arxiv.org/html/2610.27393#S4.T1)\), each measured magnitude\-ranked AP prefix retains its error and warm runtime, including attribution setup\. At the exhaustive endpoint, we use the cheaper separately measured full\-AP time, which omits attribution\.

For each target, select the cheapest measured prefix whose group\-averaged error meets it, applying the same count to all included layers and groups\. Compare with fixedP=64P=64,B=256B=256Guided\-APonly when both meet the target\. Aggregation follows Appendix[D](https://arxiv.org/html/2610.27393#A4); targets constrain the cohort mean rather than every prompt\.

Guided\-APCheapest measured verification prefix: checks / s per layerModelTaskGroupsErrors/layer2% target1% target0\.5% targetGemma\-2\-9BIOI1000\.94%2\.080 / 0\.272,416 / 16\.378,128 / 54\.16Gemma\-2\-9BFactual1001\.20%1\.300 / 0\.255,664 / 21\.849,904 / 38\.00Gemma\-2\-9BSVA1000\.71%1\.250 / 0\.2532 / 0\.435,472 / 20\.25Qwen3\-14BIOI521\.34%2\.9516 / 0\.768,016 / 73\.1212,608 / 114\.56Qwen3\-14BFactual560\.57%1\.670 / 0\.420 / 0\.422,144 / 10\.29Qwen3\-14BSVA530\.58%1\.640 / 0\.3916 / 0\.541,936 / 9\.30Qwen3\-32BIOI101\.09%6\.520 / 0\.876,016 / 126\.5515,552 / 324\.93Qwen3\-32BFactual311\.09%3\.5416 / 1\.156,736 / 67\.2215,920 / 157\.58Qwen3\-32BSVA400\.66%3\.550 / 0\.8416 / 1\.166,672 / 66\.88Table 5:Warm cost at all specified targets\. Zero checks denotes ordinary AtP\.Table[5](https://arxiv.org/html/2610.27393#A4.T5)reports every target and setting; a cost advantage requires both methods to meet the same target\.

Fixed\-estimate FP64 rescoring retains Gemma IOI error below 1% \(0\.95%\); verify\-more needs 2,064 checks and 14\.05 seconds, giving a 6\.74 time ratio with unchanged execution and timings\.

#### Matching the correction’s achieved error\.

This post hoc comparison matchesGuided\-AP’s achieved mean relative AP error rather than a prespecified target\. For each 32B task and reference, select the cheapest measured verification prefix attaining that error on the same complete groups, using one prefix count throughout \(Table[6](https://arxiv.org/html/2610.27393#A4.T6)\)\.

Table 6:Post hoc 32B accuracy and warm runtime atGuided\-AP’s achieved error\. Ratios divide verify\-more time byGuided\-APtime\. FP64 rescoring keeps the estimates and measured runtimes fixed\.Each paired group\-bootstrap draw recomputes the method’s mean error and the cheapest matching prefix \(20,000 draws, seed 20261010\)\. Descriptive time\-ratio intervals are 7\.9 to 19\.5, 10\.9 to 18\.2, and 4\.2 to 8\.1 for IOI, factual completion, and SVA; fixed\-estimate FP64 intervals are 9\.1 to 21\.3, 5\.4 to 17\.8, and 1\.5 to 4\.7\. Cohorts match Table[1](https://arxiv.org/html/2610.27393#S4.T1)\.

Relative error normalizes each vector before group averaging, so its prompt weighting differs from unnormalized MAE\. Matching achieved relative error is distinct from matching runtime or a prespecified accuracy target\.

### D\.3Measured Comparisons with Other Correction Methods

The comparison with other correction methods uses 30 complete prompt groups per Gemma task, each scored on all three layers, totaling 270 layer comparisons\. These groups overlap the main 100\-group sets \(Table[7](https://arxiv.org/html/2610.27393#A4.T7)\)\.

All selective methods use 256 components or AP measurements;Guided\-APincludes its 64\-label pilot\. AtP∗verification includes gradient\-drop ranking\. HVP and MS\-HVP correct magnitude\-selected components; MS\-HVP uses four or five steps\. Singleton IG uses four or eight Gauss\-Legendre points on isolated component paths\. Unselected components retain AtP\. No\-pilot centering shifts them by the negative median attribution\. Runtimes include ranking, operators, and assembly on the same hardware; reference collection is separate\.

Table 7:Measured comparator accuracy and warm seconds per layer on the complete Gemma comparator groups\. Error is relative AP error\. Parentheses identify MS\-HVP steps or IG quadrature points\.Against runtime\-matched verify\-more,Guided\-APlowers MAE by 16\.0%, 26\.3%, and 19\.7% on IOI, factual completion, and SVA in this comparator cohort\. Table[7](https://arxiv.org/html/2610.27393#A4.T7)gives all methods’ measured errors and runtimes\.

Figure[3](https://arxiv.org/html/2610.27393#S4.F3)uses fixed budget\-frontier cohorts: nine IOI, eight factual, and nine SVA Gemma groups, plus 45 SVA groups for 32B\. GemmaB=512B=512points reuse the first 512 measurements fromB=1024B=1024sequences with the same pilot; time combines the measured acquisition time of those prefixes with theB=256B=256setup, fit, and assembly overhead\. Derivative budgets are 64 and 256\. Hollow Gemma markers denote five\-step MS\-HVP and eight\-point IG; 32B uses endpoint HVP, four\-step MS\-HVP, and four\-point IG\.

### D\.4Within\-Prompt Controls and Ablations

Verification\-plus\-shift fits a median residual from the same 256 magnitude\-ranked AP measurements as verification alone, then shifts unmeasured estimates\.Guided\-APuses a 64\-label representative pilot inside that budget\. Mean\-versus\-median fits share labels\. Uniform and stratified sampling are replayed from measured AP tables, counting all selected measurements\. These controls use the main cohorts; the acquired pilot\-size study uses a separate cohort\.

Figure 7:Within\-promptGuided\-APablations on Gemma\-2\-9B at 256 measurements\. Panels \(a\) to \(c\) report MAE reductions relative to verification alone\. Panel \(d\) shows factual absolute\-error distributions before and after the common shift, scaled by the original root mean square error\.Table 8:Within\-promptGuided\-APMAE reductions at equal measurement counts\. Intervals are descriptive paired\-group 95% intervals\. FP64 scoring keeps the deployed FP32 estimates fixed\.Table 9:GemmaGuided\-APpilot ablations with replayed samples\. Entries give percent lower MAE than verification alone at 256 total measurements\.Figure[7](https://arxiv.org/html/2610.27393#A4.F7)a and Table[10](https://arxiv.org/html/2610.27393#A4.T10)use the acquired pilot\-size cohort below\. Panel \(b\) uses the measured pilot labels from 100 Gemma groups per task; panel \(c\) and Table[9](https://arxiv.org/html/2610.27393#A4.T9)replay samples on those groups\. All use the layer/group aggregation above and descriptive uncertainty\.

The acquired pilot\-size study uses 20 groups per task, all three layers, four pilot sizes, and three sampling seeds\. The same complete groups enter every size\. Pilots of 32 or 64 measurements capture most of the benefit \(Table[10](https://arxiv.org/html/2610.27393#A4.T10)\)\.

Table 10:Acquired pilot\-size ablation on Gemma\-2\-9B\. Entries are MAE reductions relative to verification alone, in percent, at 256 total measurements\. The same complete groups enter each pilot size\.AtP=64P=64, 10,000 paired group\-bootstrap draws give descriptive 95% intervals of 11\.4% to 24\.6% \(IOI\), 22\.2% to 32\.1% \(factual\), and 14\.5% to 25\.9% \(SVA\)\. Group wins are 19/20, 20/20, and 19/20; the main results use the separate 100\-group cohorts\.

Figure[7](https://arxiv.org/html/2610.27393#A4.F7)d pools factual residuals on their original RMS scale\. Fractions within 0\.5, 1, and 2 RMS use exact histogram bin boundaries\. The shifted distribution applies the pilot offset before exact replacements\. At least three quarters of residuals share a strict sign in 43%, 51%, and 47% of Gemma IOI, factual, and SVA prompt\-layer cases, respectively, measured from their quartiles\.

### D\.5Mechanism and Precision Checks

After predictions and selected sets are fixed, reference residuals supply the diagnostic best common shift on the unmeasured set\. Its realized correction benefit isHU−EUH\_\{U\}\-E\_\{U\}\.CallocC\_\{\\rm alloc\}compares the method with the equal\-count magnitude prefixVBV\_\{B\};CcountC\_\{\\rm count\}accounts for the runtime\-matched prefixVB0V\_\{B\_\{0\}\}\. Table[11](https://arxiv.org/html/2610.27393#A4.T11)normalizes terms by mean matched\-baseline MAE\.

Table 11:Theorem[1](https://arxiv.org/html/2610.27393#Thmtheorem1)’s decomposition on Gemma, in percentage points of matched\-baseline MAE\. Negative count cost means fewer baseline checks\.Primary acquisition discrepancy isDacq=0D\_\{\\rm acq\}=0\. The fitted median retains 96\.4%, 97\.0%, and 96\.1% of the best shift’s benefit on IOI, factual completion, and SVA\. On SVA, runtime matching contributes 1\.04 percentage points and displaced verification costs 1\.36; the equal\-count gain is 19\.1%\. Group\-averaged wins are 96/100, 97/100, and 98/100\.

FP64 readout checks retain the FP32 model body, hidden states, attribution vector, and runtime match\.*Fixed*scores deployed FP32 estimates against the FP64 reference;*refit*also uses FP64 acquired values for the pilot shift and measured coordinates\. Neither runs a full\-FP64 model \(Table[12](https://arxiv.org/html/2610.27393#A4.T12)\)\.

Table 12:Percent lower MAE under the primary reference and two FP64 sensitivity checks\. Fixed estimates and FP64\-acquired refits keep the budget match\.Fixed\-estimate 95% intervals are 7\.2% to 13\.1%, 20\.5% to 26\.3%, and 12\.6% to 18\.3% on Gemma IOI, factual, and SVA\. Qwen3\-14B SVA is 4\.1% \(0\.5% to 8\.1%\)\. For 32B, fixed\-estimate intervals are 8\.1% to 17\.7%, 3\.2% to 13\.1%, and 4\.4% to 11\.8%; primary intervals are 11\.6% to 22\.7%, 13\.6% to 21\.2%, and 10\.7% to 16\.7%\. Primary group wins are 9/10, 30/31, and 39/40, on the same cohorts as Table[1](https://arxiv.org/html/2610.27393#S4.T1)\.

### D\.6Fixed Budget across Component Populations

Nested populations contain 1,024, 4,096, or all neurons in a predetermined random order\. Each receivesP=64P=64,B=256B=256on the same groups, layers, and seeds as the full\-population study, with its own runtime\-matched verification prefix\. Population sizes are dependent views of each prompt group \(Table[13](https://arxiv.org/html/2610.27393#A4.T13)\)\.

Table 13:MAE reduction against runtime\-matched verify\-more at three predetermined component populations\. Full populations are 14,336, 17,408, and 25,600 neurons for the three models\.

## Appendix EError\-Structure Diagnostics

#### Probe noise grows with interactions\.

Defineκi=∑j≠iGi​j2/max⁡\(\|Gi​i\|,10−30\)\\kappa\_\{i\}=\\sqrt\{\\sum\_\{j\\neq i\}G\_\{ij\}^\{2\}\}/\\max\(\|G\_\{ii\}\|,10^\{\-30\}\), the ratio of off\-diagonal to diagonal curvature that sets the sketch noise in Equation[11](https://arxiv.org/html/2610.27393#A2.E11)\. Its median rises from 5\.8 to 77\.2 overN=32N=32to1,1521\{,\}152heads in Qwen3\-8B IOI \(Figure[8](https://arxiv.org/html/2610.27393#A5.F8)\)\. Separately, 41N/RN/Rsettings on ten held\-out IOI prompts give a median observed\-to\-predicted sketch\-MSE ratio of 1\.003 \(range 0\.945 to 1\.042\)\. Each point averages row\-level MSE ratios, and settings are dependent\.

Figure 8:Sketch\-noise diagnostics\. Left: median coupling across six model\-task settings\. Right: 41N/RN/Rmeans on Qwen3\-8B IOI; dashed line denotes equality and red line the median\. Points are dependent, not confidence intervals\.Table[14](https://arxiv.org/html/2610.27393#A5.T14)compares fixed calibration halves\. AP error correlations use heads measured in both halves; probe correlations use all heads\. Top\-16 overlap scores the model fitted on all 50 calibration groups separately on each half\. These describe calibration stability rather than held\-out allocation performance\.

Table 14:Stability between fixed calibration halves\. AP errorρ\\rhouses heads measured in both halves \(87 to 137 per setting\); probeρ\\rhouses all heads\. Top\-16 kept reports overlap for theGuided\-AP/ProbeCalmodel fitted on all 50 calibration groups, scored on each half\.For the dataset\-average effects, the absolute mean signed error across heads is 81% to 96% of the mean absolute error\. Equivalently, withg¯i\\bar\{g\}\_\{i\}andΔ¯i\\bar\{\\Delta\}\_\{i\}denoting the estimated and reference mean effects, the diagnostic is\|∑i\(g¯i−Δ¯i\)\|/∑i\|g¯i−Δ¯i\|\|\\sum\_\{i\}\(\\bar\{g\}\_\{i\}\-\\bar\{\\Delta\}\_\{i\}\)\|/\\sum\_\{i\}\|\\bar\{g\}\_\{i\}\-\\bar\{\\Delta\}\_\{i\}\|\. It describes agreement in error sign across heads after averaging prompts\.

## Appendix FConfirmation Protocol

Across\-prompt evaluation uses 50 calibration and 100 evaluation groups in each of six model\-task cells\. All 1,152 Qwen3\-8B heads \(revisionb968826d\) and 784 Qwen2\.5\-7B\-Instruct heads \(a09a3545\) are included\([Yang et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib27);[Qwen et al\., 2025](https://arxiv.org/html/2610.27393#bib.bib22)\)\. Collection uses H100, PyTorch 2\.9\.1, Transformers 4\.57\.1, CUDA 12\.6, FP32 eager attention, and disabled TF32\. Groups shared across models remain linked\. IOI equally mixes matched and template/entity\-shifted groups; exact and reversed contrasts do not overlap across splits\.

Arms comprise AtP ranking \(verify\-more\), random ranking,Guided\-AP, a cost\-matched guided prefix,ProbeCal, and a capacity\-matched AP\-only learner\. The reference\-error oracle is a diagnostic ceiling\. The five nonbaseline arms, six cells, four budgets, and three horizons define 360 comparisons\. AtP∗is evaluated within prompts \(Appendix[D\.3](https://arxiv.org/html/2610.27393#A4.SS3)\); all across\-prompt arms use AtP estimates\.

Guided\-APacquires 1,600 labels per cell by magnitude\-guided round robin\.ProbeCaluses eight Rademacher probes per calibration group, totaling 400 HVPs\. Fits and the capacity\-matched control follow Appendices[A](https://arxiv.org/html/2610.27393#A1)and[B](https://arxiv.org/html/2610.27393#A2)\.

The output metric is the final\-position logit difference between correct and counterfactual tokens, relative to the clean baseline\. Per\-prompt MAE is computed afterB∈\{16,32,64,128\}B\\in\\\{16,32,64,128\\\}verifications\. Dataset\-mean error averages effects over the 100 held\-out groups before taking absolute error; it is not average per\-prompt MAE\. Recall uses the ten largest absolute reference effects\.

Primary verification and scoring share a measured full AP table; separately acquired AP checks atB=16B=16andB=32B=32, with their cost included, confirm this shared table\. Total costs include calibration, fitting, transfer, model loading, attribution, and verification\. Analysis lengths of 100, 1,000, and 10,000 prompts project measured costs and observed errors; they are not additional evaluation samples\.

We use 20,000 paired group\-bootstrap draws with the prescribed groups and strata\. Two\-sided Bonferroni percentile intervals use tail probability0\.05/\(2⋅360\)0\.05/\(2\\cdot 360\), about 1\.39 draws per tail, and condition on the fitted calibration and measured costs\.

### F\.1Across\-Prompt Result Summaries

Table[3](https://arxiv.org/html/2610.27393#S4.T3)reports allocation at matched projected total cost, including calibration\. Table[15](https://arxiv.org/html/2610.27393#A6.T15)gives dataset\-average error ratios against uncorrected AtP and calibration coverage\.

Table 15:Dataset\-average correction\. Error ratio is uncorrected divided by corrected MAE\. Bold and italic mark adjusted significant improvements and harms\. Heads and coverage refer toGuided\-APcalibration\.

## Appendix GCost Frontiers for All Cells

Figures[9](https://arxiv.org/html/2610.27393#A7.F9),[10](https://arxiv.org/html/2610.27393#A7.F10), and[11](https://arxiv.org/html/2610.27393#A7.F11)cover all six across\-prompt cells, budgets, and projected analysis lengths\.

Figure 9:Across\-prompt evaluation on Qwen3 IOI\. Error against projected total cost at three analysis lengths, with markers for verification budgets and dotted curves for verify\-more, followed by the dataset\-mean correction ratios for all six cells with adjusted intervals\.Calibration cost favors verify\-more at 100 prompts throughout the grid\. At 1,000 prompts, 108 of 120 adjusted comparisons favor verify\-more and none favors learning\. At 10,000 prompts, the useful cells and budgets depend on the signal\.Guided\-APimproves Qwen3 IOI and both SVA cells atB=16B=16;ProbeCalalso improves factual allocation atB≥64B\\geq 64\. The figures retain the remaining comparisons and controls\.

Lower MAE need not improve recall\. One Qwen3 IOIProbeCalsetting reduces MAE by 11% while top\-ten recall falls from 99\.2% to 97\.0%\.

Projected break\-even ranges from 1,900 to 24,000 prompts atB=16B=16and 900 to 15,500 atB=128B=128\. Recall intervals usually include no change; some IOIB=16B=16settings reduce recall by 0\.03 to 0\.07\.

Figure 10:Cost\-error frontiers for three more cells at 100, 1,000, and 10,000 prompts\. Conventions follow Figure[9](https://arxiv.org/html/2610.27393#A7.F9)\.Figure 11:Cost\-error frontiers for the remaining two cells\. Conventions as in Figure[9](https://arxiv.org/html/2610.27393#A7.F9)\.
## Appendix HAdditional Controls on the Confirmation Cells

These post hoc controls use the same six cells and 20,000 paired bootstrap draws, with their analysis protocol fixed before computation\.

#### Dataset correction at matched total cost\.

Verify\-more uses the largest affordable magnitude\-ranked AP prefix\. It wins in all settings at 100 prompts and in ten of twelve at 1,000 prompts\. At 10,000 prompts it affords no additional checks, so the comparison is with uncorrected AtP\. Where correction improves that baseline, projected break\-even is 527 to 2,744 prompts\. The IOI exceptions appear in Table[15](https://arxiv.org/html/2610.27393#A6.T15)\.

#### Learning residuals versus averaging measurements\.

Using the same 1,600 queries directly as component\-average effects produces 5\.8 to 14\.9 times more error than residual correction \(Equation[9](https://arxiv.org/html/2610.27393#A1.E9)\) in all six cells, with every adjusted interval above one, and is worse than uncorrected AtP\.

#### Independently acquired verification\.

Separate AP forwards atB=16B=16andB=32B=32reproduce the primary values exactly in all six cells and six arms; all 75 testable prespecified comparisons retain their direction\.

## Appendix ICost Accounting and Numerical Scope

For methodmmunder a fixed loading and batching schedule, projected total cost is

Cm​\(T,B\)=Cminit\+T​cm​\(B\),C\_\{m\}\(T,B\)=C\_\{m\}^\{\\rm init\}\+Tc\_\{m\}\(B\),\(13\)whereCminitC\_\{m\}^\{\\rm init\}includes calibration, fitting, and the specified initialization costs\. The per\-prompt termcm​\(B\)c\_\{m\}\(B\)includes attribution, ranking, activation handling, transfer, and verification\.TTis a projected analysis length, not the number of evaluation prompts over which dataset averages are measured\. The prespecified analysis lengths are 100, 1,000, and 10,000 prompts, and the allocation budgets areB∈\{16,32,64,128\}B\\in\\\{16,32,64,128\\\}\.

For an across\-prompt policy withBBverifications, the matched verify\-more budget is

bm​\(T,B\)\\displaystyle b\_\{m\}\(T,B\)=maxb∈\{0,…,N\}⁡b\\displaystyle=\\max\_\{b\\in\\\{0,\\ldots,N\\\}\}bsubject to​C0​\(T,b\)≤Cm​\(T,B\),\\displaystyle\\text\{subject to \}C\_\{0\}\(T,b\)\\leq C\_\{m\}\(T,B\),HereC0C\_\{0\}includes attribution and magnitude\-ranked verification\. Measured batch padding and overhead determine the integer budget; unused compute is retained\. Within\-prompt matching uses measured runtime without prior calibration\. Dense curves are descriptive; adjusted inference uses the prespecified budgets and horizons\.

Equation[13](https://arxiv.org/html/2610.27393#A9.E13)assumes one initialization\. Break\-even horizons project measured cost components\.

On Qwen3 factual completion, 1,600 AP labels at batch size 16 took 13\.98 seconds \(0\.009 seconds/check\); 400 serial HVPs took 51\.77 seconds \(0\.13 seconds/HVP\)\. At these rates, one HVP costs about 15 exact checks\. Full calibration costs, including probes, fitting, transfer, and model loading, range from 384 to 1,050 seconds per cell and are included in the comparisons\.

Deployment includes initialization, calibration, fitting, model loading, gradients, activation handling, transfer, and verification; reference collection is separate\. Rates are specific to the measured hardware configuration\.

Interventions add directions to live activations, preserving upstream propagation\. Derivatives and references use FP32 eager execution; same\-hidden\-state FP64 readouts assess numerical sensitivity\.