Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces

# Unboxing LLM Outputs with Orthogonal Latent Spaces
Source: [https://arxiv.org/html/2605.12809](https://arxiv.org/html/2605.12809)
## Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces

Shixing Yu Electrical and Computer Engineering Cornell Tech sy774@cornell\.edu &Promit Ghosal Department of Statistics University of Chicago promit@uchicago\.edu &Kyra Gan Operations Research and Industrial Engineering Cornell Tech kyragan@cornell\.edu

###### Abstract

A critical step for reliable large language models \(LLMs\) use in healthcare is to attribute predictions to their training data, akin to a medical case study\. This requires token\-level precision: pinpointing not just which training examples influence a decision, but which tokens within them are responsible\. While*influence functions*offer a principled framework for this, prior work is restricted to*autoregressive*settings and relies on an implicit assumption of*token independence*, rendering their identified influences unreliable\. We introduce a flexible framework that infers*token\-level influence*through a latent mediation approach for*general prediction tasks*\. Our method attaches*sparse autoencoders*to any layer of a pretrained LLM to learn a basis of approximately independent latent features\. Unlike prior methods where influence decomposes additively across tokens, influence computed over latent features is inherently*non\-decomposable*\. To address this, we introduce a novel method using*Jacobian\-vector products*\. Token\-level influence is obtained by propagating latent attributions back to the input space via token activation patterns\. We scale our approach using efficient inverse\-Hessian approximations\. Experiments on medical benchmarks show our approach identifies sparse, interpretable sets of tokens that*jointly*influence predictions\. Our framework enhances trust and enables model auditing, generalizing to any high\-stakes domain requiring transparent and accountable decisions\.

## 1Introduction

The deployment of LLMs in high\-stakes domains like healthcare hinges on a critical and unmet requirement: the ability to audit a model’s reasoning by tracing its predictions directly to the evidence in its training data\. This need for verifiability is urgent, as LLMs are increasingly explored for clinical tasks such as diagnostic support and treatment planning, where errors can have severe consequences\(Singhalet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib18); Topol,[2019](https://arxiv.org/html/2605.12809#bib.bib31)\)\. Without this capability—akin to a clinician demanding the source for a medical decision—LLMs remain unverifiable black boxes\. Their tendency to hallucinate\(Jiet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib32)\)and their susceptibility to spurious correlations present in training data\(Oberst and Sontag,[2019](https://arxiv.org/html/2605.12809#bib.bib33)\)pose significant safety risks, undermining the trust required for clinical adoption\(Futomaet al\.,[2020](https://arxiv.org/html/2605.12809#bib.bib35); Ghassemiet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib34)\)\.

This fundamental need for evidence\-based reasoning is not adequately addressed by prevailing interpretability methods\. Techniques like Chain\-of\-Thought prompting generate rationales that are often post hoc justifications rather than faithful reflections of the model’s true decision process\(Turpinet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib36); Barezet al\.,[2025](https://arxiv.org/html/2605.12809#bib.bib28)\)\. Other popular approaches, such as attention visualization\(Wiegreffe and Pinter,[2019](https://arxiv.org/html/2605.12809#bib.bib38); Jain and Wallace,[2019](https://arxiv.org/html/2605.12809#bib.bib39)\)or gradient\-based feature attribution\(Sundararajanet al\.,[2017b](https://arxiv.org/html/2605.12809#bib.bib40)\), are limited to explaining a single forward pass of a model\. They operate within the context of a given input, providing no insight into how prior training experiences shaped the model’s fundamental behavioral patterns and knowledge\(Feldman and Zhang,[2020](https://arxiv.org/html/2605.12809#bib.bib37)\)\. This represents a critical limitation for clinical deployment, where the ability to pinpoint the exact training evidence behind a prediction—not just generate plausible\-sounding rationales—is essential for medical professionals to validate the model’s logic against established knowledge, fact\-check its conclusions, and ultimately build the trust required for adoption in safety\-critical settings\.

A principled framework for addressing this question lies in*influence functions*\(IFs\), a tool from robust statistics that explains how a model’s predictions depend on its training data\(Hampel,[1974](https://arxiv.org/html/2605.12809#bib.bib16)\)\. This approach treats the model as an empirical entity shaped by its dataset, enabling one to trace a final prediction back to influential training points\(Koh and Liang,[2017](https://arxiv.org/html/2605.12809#bib.bib9)\)\. Recent work has successfully scaled this approach to modern LLMs, demonstrating its potential to reveal generalization patterns by attributing influence down to the token level\(Grosseet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib24)\)\. However, a key limitation persists: the IF framework assumes independence among the components of the objective \(e\.g\., tokens in an autoregressive prediction task in prior work\)\. This assumption is necessary for influence scores to be meaningfully interpretable, as it ensures that the relative difference in influence between components is well\-defined\. In practice, the tokens within LLMs are highly correlated\. Thus, prior implementations, while powerful, produce influence estimates that are theoretically unsound and difficult to interpret\(Basu and Echenique,[2020](https://arxiv.org/html/2605.12809#bib.bib41); Tsimpoukelliet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib42)\)\.

We introduce a robust framework that infers*token\-level*influence on test predictions via latent mediation, enabling more reliable influence estimation\. Building on recent monosemanticity research\(Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30); Templetonet al\.,[2024b](https://arxiv.org/html/2605.12809#bib.bib862)\)and disentangled representation learning\(Wanget al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1086)\), our method leverages the fact that neural networks decompose into semantically meaningful, independent components\. Our method generalizes to*general prediction tasks*by propagating influence through disentangled latent spaces where features exhibit statistical independence, critical for reliable influence estimation\. Our contributions are fourfold:

1. 1\.Unified sample\- and feature\-level influence: We extend influence analysis beyond the isolated\-token paradigm of prior work to model the*joint influence of tokens*within training sample\-label pairs\. By propagating influence from latent features to input tokens through their joint activation patterns, we attribute predictions to specific token combinations in the training data while leveraging monosemantic structure\. Unlike methods treating neurons as atomic units, we recognize meaningful computation occurs at interpretable feature level spanning multiple neurons\.
2. 2\.Stable, independent feature extraction via sparse autoencoders \(SAEs\): We use SAEs\(Gaoet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib48); Cunninghamet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib25); Markset al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib26); Conget al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib43)\)as an interpretability component to produce sparse, approximately orthogonal latent features at an intermediate layer\. We then compute influence scores with respect to these latent features, improving the stability and interpretability of training\-data attributions\.
3. 3\.Scalable non\-decomposable influence estimation via derivative swapping: Latent\-level influence is holistically interdependent and lacks the additive decomposition of token\-level approaches\. While naive Jacobian\-vector product \(JVP\) evaluation requires an𝒪​\(dl\)\\mathcal\{O\}\(d\_\{l\}\)forward\-mode pass per feature, we exploit Clairaut’s Theorem to swap the derivative order\. This gradient\-derivative formulation restructures the computation into a single reverse\-mode pass, reducing complexity to𝒪​\(1\)\\mathcal\{O\}\(1\)and achieving a10×10\\timesto20×20\\timespractical speedup\.
4. 4\.Large\-scale empirical validation on medical and general reasoning: We evaluate our framework on 1B and 1\.5B parameter models \(Llama\-3\.2 and Qwen2\.5\) across multiple QA datasets\. Rigorous necessity and sufficiency ablations show our method isolates compact, highly influential circuits that systematically outperform activation magnitude and frequency baselines\. Moreover, heatmap visualizations over input tokens on fixed test samples suggest potential patterns in the model’s behavior, revealing that incorporating context during training leads to different behavior than not doing so; however, further investigation is needed to fully understand this behavioral difference\.

By unifying data\-level and feature\-level attribution, our approach offers a principled pathway toward transparent, trustworthy, and deployable LLMs for high\-stakes domains, with additional potential for large\-scale training data auditing and diagnostics, which we further discuss in Section[6](https://arxiv.org/html/2605.12809#S6)\. Section[3](https://arxiv.org/html/2605.12809#S3)introduces our notation and preliminaries on IF and JVP\. We then describe our method in Section[4](https://arxiv.org/html/2605.12809#S4)and evaluate its performance in Section[5](https://arxiv.org/html/2605.12809#S5)\. Additional related works are included in Appendix[2](https://arxiv.org/html/2605.12809#S2)\. The full pipeline is demonstrated in Figure[1](https://arxiv.org/html/2605.12809#S1.F1)\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/figure1_v2.png)Figure 1:Pipeline overview\.Overview of RepInfLLM\. A domain\-specific LLM is first finetuned, then SAEs are swept over intermediate layers \(25%–75%\) to select a representative latent space\. During inference, the selected SAE is inserted inline to map both training and test instances into shared sparse latents, enabling influence attribution directly in representation space\. The prediction follows the standard forward pass, while attribution is computed via a backward path that operates on disentangled latents using activation\-weighted scaling, avoiding token\-level entanglement\.
## 2Related Works

Interpretability in LLMInterpretability methods range from black\-box approaches like perturbation and sensitivity analysis\(Casalicchioet al\.,[2018](https://arxiv.org/html/2605.12809#bib.bib614); Ribeiroet al\.,[2016](https://arxiv.org/html/2605.12809#bib.bib457); Covertet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib604); Warstadtet al\.,[2020](https://arxiv.org/html/2605.12809#bib.bib599)\), to gradient\-based attribution methods\(Smilkovet al\.,[2017](https://arxiv.org/html/2605.12809#bib.bib616); Sundararajanet al\.,[2017b](https://arxiv.org/html/2605.12809#bib.bib40); Bachet al\.,[2015](https://arxiv.org/html/2605.12809#bib.bib603); Shrikumaret al\.,[2017](https://arxiv.org/html/2605.12809#bib.bib606); Selvarajuet al\.,[2016](https://arxiv.org/html/2605.12809#bib.bib607); Bilodeauet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib759)\), and concept\-based representations probing\(Belinkov,[2022](https://arxiv.org/html/2605.12809#bib.bib412); Kornblithet al\.,[2019](https://arxiv.org/html/2605.12809#bib.bib566); Bansalet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib788); Burnset al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib137); Zouet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib245); Arditiet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1013)\)\. More recent work in mechanistic interpretability focuses on reverse\-engineering internal model structures through circuit analysis\(Olahet al\.,[2018](https://arxiv.org/html/2605.12809#bib.bib126); Elhageet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib132),[2022b](https://arxiv.org/html/2605.12809#bib.bib133)\), and feature discovery\(Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30); Sharkey,[2022](https://arxiv.org/html/2605.12809#bib.bib204); Cunninghamet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib25); Denget al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib296)\)\. In addition to monosemanticity and disentanglement, this line of work has enabled analyses of motifs like induction heads or copy suppression\(Olssonet al\.,[2022](https://arxiv.org/html/2605.12809#bib.bib135); McDougallet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib99); Cammarataet al\.,[2020](https://arxiv.org/html/2605.12809#bib.bib128),[2021](https://arxiv.org/html/2605.12809#bib.bib123)\), universality\(Chanet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib562); Gurneeet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib138); Marchettiet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib856)\), and emergent world models\(Liet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib139); Nanda,[2023](https://arxiv.org/html/2605.12809#bib.bib145); Ivanitskiyet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib525); Karvonen,[2024](https://arxiv.org/html/2605.12809#bib.bib999); Shanahanet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib940); janus,[2022](https://arxiv.org/html/2605.12809#bib.bib360)\)\. Unlike these approaches, which often prioritize global model understanding, our method emphasizes actionable, testable attributions tailored for high\-stakes domains like healthcare, where rapid fact\-checking and validation of model decisions are critical for reliability and trust\.

Sparse Autoencoders and Independent FeaturesSAEs learn disentangled, interpretable features via sparsity constraints \(e\.g\., L1 penalty\), promoting statistical independence in latent representations\. This approach builds upon a long history of seeking independent data components, including classical linear methods like Principal Component Analysis \(PCA\)\(Jolliffe and Cadima,[2016](https://arxiv.org/html/2605.12809#bib.bib3)\)and Independent Component Analysis \(ICA\)\(Hyvärinen,[2013](https://arxiv.org/html/2605.12809#bib.bib2)\), as well as nonlinear probabilistic frameworks like Variational Autoencoders \(VAEs\)\(Kingma and Welling,[2014](https://arxiv.org/html/2605.12809#bib.bib1)\)\. However, SAEs offer a uniquely transparent and deterministic pathway to feature learning that balances sparsity and reconstruction fidelity\. They are widely used for mechanistic interpretability in LLMs\(Cunninghamet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib25); Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30); Templetonet al\.,[2024a](https://arxiv.org/html/2605.12809#bib.bib44); Markset al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib26)\), with variants includingkk\-sparse SAEs\(Makhzani and Frey,[2013](https://arxiv.org/html/2605.12809#bib.bib45)\), gated and JumpReLU SAEs\(Rajamanoharanet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib880)\), and TopK methods\(Gaoet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib48); Bussmannet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib49)\)\. Beyond language, SAEs extend to multimodal domainsSurkovet al\.\([2025](https://arxiv.org/html/2605.12809#bib.bib46)\), radiology and medical imaging\(Abdulaalet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib47)\), and reinforcement learning alignment\(Yinet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib4)\), demonstrating versatility across tasks\. Recent work shows that transcoders \(which approximate dense MLP behavior via wider, sparsely\-activating networks\) often match or exceed SAEs in interpretability and fidelity\(Dunefskyet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1089)\)\. Extending our framework to handle independent logits from a transcoder is promising but beyond the scope of this work\.

Monosemanticity and DisentanglementThe pursuit of monosemantic features, where neurons respond to single coherent concepts, represents a major focus in interpretability research\. This effort addresses the phenomenon of polysemanticity, explained through the superposition hypothesis\(Olahet al\.,[2018](https://arxiv.org/html/2605.12809#bib.bib126); Elhageet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib132),[2023](https://arxiv.org/html/2605.12809#bib.bib131); Scherliset al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib157); Henighanet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib130)\)\. Solutions include both architectural modifications such askk\-sparse autoencoders\(Makhzani and Frey,[2013](https://arxiv.org/html/2605.12809#bib.bib45)\), softmax linear units\(Elhageet al\.,[2022a](https://arxiv.org/html/2605.12809#bib.bib134); Rajamanoharanet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib880)\), as well as post\-hoc methods like SAEs\(Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30); Sharkey,[2022](https://arxiv.org/html/2605.12809#bib.bib204); Cunninghamet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib25); Denget al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib296)\)\. Studies have examined the linearity of representations\(Nanda,[2022](https://arxiv.org/html/2605.12809#bib.bib751); Engelset al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib860); O’Mahonyet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib265); Hendelet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib247); Toddet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib526); Hernandezet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib442); Chaninet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib669); Tiggeset al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1034); Arditiet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1013)\), identified counterexamples such as circular features\(Engelset al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib860)\)and non\-linear perspectives\(Blacket al\.,[2022](https://arxiv.org/html/2605.12809#bib.bib155)\)\. Geometry\-aware analyses show structured organization\(Parket al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib1028)\), and scaling studies\(Templetonet al\.,[2024a](https://arxiv.org/html/2605.12809#bib.bib44)\)suggest disentanglement improves with model size\. While these works aim for complete monosemanticity, our approach uses SAEs to obtain approximately independent features specifically to enable more reliable influence estimation, prioritizing practical interpretability over full disentanglement\.

## 3Preliminaries

Given a training dataset𝒟=\{zi=\(xi,yi\)\}i=1n\\mathcal\{D\}=\\\{z\_\{i\}=\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}i\.i\.d\. drawn from an unknown distribution, with inputxi∈𝒳x\_\{i\}\\in\\mathcal\{X\}and labelyi∈𝒴y\_\{i\}\\in\\mathcal\{Y\}\. A modelhθ:𝒳→𝒴h\_\{\\theta\}:\\mathcal\{X\}\\to\\mathcal\{Y\}with parametersθ∈ℝp\\theta\\in\\mathbb\{R\}^\{p\}is trained by minimizing the empirical riskθ^=arg⁡minθ⁡1n​∑i=1nℓ​\(hθ​\(xi\),yi\)\\hat\{\\theta\}=\\arg\\min\_\{\\theta\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(h\_\{\\theta\}\(x\_\{i\}\),y\_\{i\}\), whereℓ​\(⋅,⋅\)\\ell\(\\cdot,\\cdot\)is the loss function\.

#### Influence Functions \(IFs\)

In statistical estimation, the IF quantifies the sensitivity of an estimator to infinitesimal perturbations in the data, under the assumption that the data are independent\. This concept extends directly to machine learning, where the high\-dimensional “parameter” is the set of weightsθ^\\hat\{\\theta\}of a trained neural network—a complex function of the data shaped by the architecture, loss, and optimizer\. Once training is complete and the model parametersθ^\\hat\{\\theta\}are fixed, we can analyze their local sensitivity to individual training samples\. This is first formalized by the*response function*,θ^ϵ,ztrain\\hat\{\\theta\}\_\{\\epsilon,z\_\{\\text\{train\}\}\}, which describes what the optimal parameters*would be*if we were to infinitesimally upweight the loss \(byϵ\\epsilon\) on a specific pointztrain=\(xtrain,ytrain\)z\_\{\\text\{train\}\}=\(x\_\{\\text\{train\}\},y\_\{\\text\{train\}\}\)in the empirical risk\. This perturbed objective is defined as:

θ^ϵ,ztrain=arg⁡minθ⁡1n​∑i=1nℓ​\(hθ​\(xi\),yi\)\+ϵ​ℓ​\(hθ​\(xtrain\),ytrain\),\\hat\{\\theta\}\_\{\\epsilon,z\_\{\\text\{train\}\}\}=\\arg\\min\_\{\\theta\}\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\(h\_\{\\theta\}\(x\_\{i\}\),y\_\{i\}\)\+\\epsilon\\ell\(h\_\{\\theta\}\(x\_\{\\text\{train\}\}\),y\_\{\\text\{train\}\}\),\(1\)where the solution atϵ=0\\epsilon=0corresponds exactly to the original pre\-trained parameters:θ^0,ztrain=θ^\\hat\{\\theta\}\_\{0,z\_\{\\text\{train\}\}\}=\\hat\{\\theta\}\. The IF measures the sensitivity of these pre\-trained parameters by computing the first\-order Taylor approximation \(i\.e\., the derivative\) of the response function with respect toϵ\\epsilon, atθ^\\hat\{\\theta\}\. Under standard regularity conditions, this can be computed using the Implicit Function Theorem\(Krantz and Parks,[2002](https://arxiv.org/html/2605.12809#bib.bib29)\)\. LetHθ^=1n​∑i=1n∇θ2ℓ​\(hθ^​\(xi\),yi\)H\_\{\\hat\{\\theta\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\nabla^\{2\}\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{i\}\),y\_\{i\}\)be the Hessian of the empirical risk evaluated atθ^\\hat\{\\theta\}, then

IFθ^​\(ztrain\)=d​θ^ϵ,ztraind​ϵ\|ϵ=0=−Hθ^−1​∇θℓ​\(hθ^​\(xtrain\),ytrain\)\.\\mathrm\{IF\}\_\{\\hat\{\\theta\}\}\(z\_\{\\text\{train\}\}\)=\\frac\{d\\hat\{\\theta\}\_\{\\epsilon,z\_\{\\text\{train\}\}\}\}\{d\\epsilon\}\\bigg\|\_\{\\epsilon=0\}=\-H\_\{\\hat\{\\theta\}\}^\{\-1\}\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{\\text\{train\}\}\),y\_\{\\text\{train\}\}\)\.\(2\)Influential Training Samples on Test PredictionSinceIFθ^​\(ztrain\)\\mathrm\{IF\}\_\{\\hat\{\\theta\}\}\(z\_\{\\text\{train\}\}\)is a high\-dimensional vector, it is often difficult to interpret directly\. To obtain a more concrete measure, we convert this parameter\-space influence into a scalar quantity by measuring its effect on a specific model output\. This is done by projecting the influence vector onto the gradient of a chosen function, such as the loss or the logits for a test exampleztest=\(xtest,ytest\)z\_\{\\text\{test\}\}=\(x\_\{\\text\{test\}\},y\_\{\\text\{test\}\}\)\. Applying the Chain Rule, we can compute the scalar*influence*of upweightingztrainz\_\{\\text\{train\}\}on the loss atztestz\_\{\\text\{test\}\}as follows:

ℐ​\(ztrain,ztest\)=−∇θℓ​\(hθ^​\(xtest\),ytest\)⊤​Hθ^−1∇θℓ​\(hθ^​\(xtrain\),ytrain\)\.\\mathcal\{I\}\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)^\{\\top\}H\_\{\\hat\{\\theta\}\}^\{\-1\}\\\\ \\quad\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{\\text\{train\}\}\),y\_\{\\text\{train\}\}\)\.\(3\)This provides an interpretable measure to trace predictions back to influential training samples\.

Influential Tokens on Test Prediction in Autoregressive TasksIn*autoregressive*tasks, the loss function decomposes additively across tokens, which enables the direct computation of token\-level influence\. This additive structure permits the gradient and Hessian in the influence function to be similarly decomposed, allowing the influence of individual training tokens to be derived explicitly\. Let\{x1,⋯,xT\}\\\{x\_\{1\},\\cdots,x\_\{T\}\\\}to denote theTTtokens inxtrainx\_\{\\text\{train\}\}\. Then, Eq\. \([3](https://arxiv.org/html/2605.12809#S3.E3)\) can be rewritten as

ℐ​\(ztrain,ztest\)=−∇θℓ​\(hθ^​\(xtest\),ytest\)⊤​Hθ^−1∇θ​∑t=1Tℓ​\(hθ^​\(xt\),yt\)\.\\mathcal\{I\}\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)^\{\\top\}H\_\{\\hat\{\\theta\}\}^\{\-1\}\\\\ \\quad\\nabla\_\{\\theta\}\\sum\_\{t=1\}^\{T\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{t\}\),y\_\{t\}\)\.\(4\)Thus, the per\-token influence score is defined as\(Grosseet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib24)\):

ℐt​\(ztrain,ztest\)=−∇θℓ​\(hθ^​\(xtest\),ytest\)⊤​Hθ^−1∇θℓ​\(hθ^​\(xt\),yt\)\.\\mathcal\{I\}\_\{t\}\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)^\{\\top\}H\_\{\\hat\{\\theta\}\}^\{\-1\}\\\\ \\quad\\nabla\_\{\\theta\}\\ell\(h\_\{\\hat\{\\theta\}\}\(x\_\{t\}\),y\_\{t\}\)\.\(5\)

#### Jacobian\-Vector Products

This is a key technical tool that we use\. Given a functionF:ℝn→ℝmF:\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}^\{m\}and a directionv∈ℝnv\\in\\mathbb\{R\}^\{n\}, the JVP atx∈ℝnx\\in\\mathbb\{R\}^\{n\}is the*directional derivative*ofFFatxxalongvv:

JVP​\(F,x,v\)=dd​ε​F​\(x\+ε​v\)\|ε=0=JF​\(x\)​v,\\mathrm\{JVP\}\(F,x,v\)\\;=\\left\.\\frac\{d\}\{d\\varepsilon\}F\(x\+\\varepsilon v\)\\right\|\_\{\\varepsilon=0\}\\;=\\;J\_\{F\}\(x\)\\,v,\(6\)whereJF​\(x\)J\_\{F\}\(x\)is the Jacobian ofFFatxx\. Intuitively, it answers the question:*“If I nudge the input by an infinitesimal stepε​v\\varepsilon v, how does the output change to first order?”*

Modern automatic differentiation libraries \(e\.g\., PyTorch, JAX, TensorFlow\) can compute JVPs directly without materializing the full Jacobian\. Instead, they propagate the perturbationvvforward through each primitive operation \(forward\-mode AD\), making JVPs scalable to high\-dimensional functions such as deep neural networks\.

## 4Methodology

We now detail our framework that infers*token\-level*influence on test predictions via a latent mediation approach, enabling more reliable influence estimation for general prediction tasks\. This section presents the core components of our approach: 1\) augmenting LLMs with SAEs to obtain more interpretable latent representations \(Section[4\.1](https://arxiv.org/html/2605.12809#S4.SS1)\), 2\) computing influence scores over these latent features rather than directly on input tokens \(Section[4\.2](https://arxiv.org/html/2605.12809#S4.SS2)\), and 3\) efficiently implementing this computation via Jacobian\-vector products \(Section[4\.4](https://arxiv.org/html/2605.12809#S4.SS4)\) while maintaining the ability to propagate attributions back to the input space\. Figure[2](https://arxiv.org/html/2605.12809#S4.F2)provides an overview of the complete framework\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/pipeline_v6.png)Figure 2:Framework overview\.Traditional influence functions operate in the input space, assuming token independence and decomposable losses\. Our method introduces a sparse autoencoder at an intermediate layer, splitting the model into upstream and downstream parts\. Influence is then computed at the representation level using JVPs, enabling stable per\-feature attributions and linking test predictions to interpretable sparse features\.### 4\.1Augmenting LLM with Sparse Autoencoders for Independent Features

We followBrickenet al\.\([2023](https://arxiv.org/html/2605.12809#bib.bib30)\)andGaoet al\.\([2024](https://arxiv.org/html/2605.12809#bib.bib48)\)to define a sparse autoencoder that maps inputxl∈ℝdx^\{l\}\\in\\mathbb\{R\}^\{d\}at layerllinto a sparse latent coder∈ℝhr\\in\\mathbb\{R\}^\{h\}through

r\\displaystyle r=σ​\(Wenc​\(xl−bpre\)\+benc\),\\displaystyle=\\sigma\(W\_\{\\text\{enc\}\}\(x^\{l\}\-b\_\{\\text\{pre\}\}\)\+b\_\{\\text\{enc\}\}\),\(7\)x~l\\displaystyle\\tilde\{x\}^\{l\}=Wdec​r\+bpre,\\displaystyle=W\_\{\\text\{dec\}\}r\+b\_\{\\text\{pre\}\},\\vskip\-5\.0pt\(8\)whereWenc∈ℝh×dW\_\{\\text\{enc\}\}\\in\\mathbb\{R\}^\{h\\times d\},benc∈ℝhb\_\{\\text\{enc\}\}\\in\\mathbb\{R\}^\{h\},Wdec∈ℝd×hW\_\{\\text\{dec\}\}\\in\\mathbb\{R\}^\{d\\times h\}, andbpre∈ℝdb\_\{\\text\{pre\}\}\\in\\mathbb\{R\}^\{d\}\. The nonlinearityσ​\(⋅\)\\sigma\(\\cdot\)is ReLU in classical settings\(Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30)\)and TopK in modern settings\(Gaoet al\.,[2024](https://arxiv.org/html/2605.12809#bib.bib48)\)\.

Letrjr\_\{j\}be the activation of featurejj, forming the basis for our feature\-level influence analysis\.

### 4\.2Influence Functions on Latent Representation

Classical influence functions applied directly to correlated text tokens \(Eq\. \([5](https://arxiv.org/html/2605.12809#S3.E5)\)\) are problematic due to strong sequential dependencies\. To address this, we compute influence on*latent features*rather than raw tokens\. Consider an intermediate activationh\(l\)∈ℝdlh^\{\(l\)\}\\in\\mathbb\{R\}^\{d\_\{l\}\}at layerll, which may correspond to the output of an attention head, MLP block, or residual stream in a transformer\(Elhageet al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib132)\)\. Mechanistic interpretability studies suggest that such representations often encode semantically meaningful features causally linked to final predictions\(Wanget al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib82); Menget al\.,[2022](https://arxiv.org/html/2605.12809#bib.bib52)\)\. To attribute influence to individual neurons \(latent coordinates\) within the representationrtrainr\_\{\\text\{train\}\}\(corresponding toh\(l\)h^\{\(l\)\}\), we must relate neuron\-level effects directly to the*training gradient*that appears in influence functions\.

As illustarated in Figure[2](https://arxiv.org/html/2605.12809#S4.F2), we split the model parameters into two parts:θ=\(θ1,θ2\)\\theta=\(\\theta\_\{1\},\\theta\_\{2\}\), whereθ1=\{θ:θ<l\}\\theta\_\{1\}=\\\{\\theta:\\theta\_\{<l\}\\\}comprises all parameters*up to and including layerll*, mapping a raw input sequencexxto an intermediate representationr=hθ1​\(x\)r=h\_\{\\theta\_\{1\}\}\(x\);θ2=\{θ:θ\>l\}\\theta\_\{2\}=\\\{\\theta:\\theta\_\{\>l\}\\\}comprises the remaining parameters*after layerll*, mappingrrto the final prediction\. By fixingθ1\\theta\_\{1\}and treatingrras the input, the IF can be computed with respect toθ2\\theta\_\{2\}alone—effectively attributing influence to the latent representationrrrather than the original tokens\.

Letrtrain=hθ1​\(xtrain\)r\_\{\\text\{train\}\}=h\_\{\\theta\_\{1\}\}\(x\_\{\\text\{train\}\}\)andrtest=hθ1​\(xtest\)r\_\{\\text\{test\}\}=h\_\{\\theta\_\{1\}\}\(x\_\{\\text\{test\}\}\)denote the latent representation of the training and testing inputs,xtrainx\_\{\\text\{train\}\}andxtestx\_\{\\text\{test\}\}, respectively\. Define the corresponding latent\-space data points asztrainr=\(rtrain,ytrain\)z\_\{\\text\{train\}\}^\{r\}=\(r\_\{\\text\{train\}\},y\_\{\\text\{train\}\}\)andztestr=\(rtest,ytest\)z\_\{\\text\{test\}\}^\{r\}=\(r\_\{\\text\{test\}\},y\_\{\\text\{test\}\}\)\.

The*representation\-level influence function*is defined as:

ℐ​\(ztrainr,ztestr\)=−∇θ2ℓ​\(hθ2​\(rtest\),ytest\)⏟=⁣:gtest⊤​Hθ2−1∇θ2ℓ​\(hθ2​\(rtrain\),ytrain\)⏟=⁣:gtrain,\\mathcal\{I\}\(z^\{r\}\_\{\\text\{train\}\},z^\{r\}\_\{\\text\{test\}\}\)=\-\\left\.\\underbrace\{\\nabla\_\{\\theta\_\{2\}\}\\ell\(h\_\{\\theta\_\{2\}\}\(r\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)\}\_\{=:\\,g\_\{\\text\{test\}\}\}\\right\.^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}\\\\ \\quad\\underbrace\{\\nabla\_\{\\theta\_\{2\}\}\\ell\(h\_\{\\theta\_\{2\}\}\(r\_\{\\text\{train\}\}\),y\_\{\\text\{train\}\}\)\}\_\{=:\\,g\_\{\\text\{train\}\}\},\(9\)whereHθ2H\_\{\\theta\_\{2\}\}is the Hessian of the training loss w\.r\.t\.θ2\\theta\_\{2\}\.

A crucialphilosophical asymmetrynow arises: IFs measure how a*training point*affects the loss on a*test point*\. The test point serves as a fixed evaluation context—we care about its loss, but we do not attribute influence to its internal structure\. Consequently, while bothztrainrz^\{r\}\_\{\\text\{train\}\}andztestrz^\{r\}\_\{\\text\{test\}\}are latent representations, our goal is to decompose the training\-side gradientgtraing\_\{\\text\{train\}\}into contributions from individual latent coordinates ofrtrainr\_\{\\text\{train\}\}\. In contrast,gtestg\_\{\\text\{test\}\}requires no decomposition; it is obtained by a single backward pass throughθ2\\theta\_\{2\}after computingrtestr\_\{\\text\{test\}\}viaθ1\\theta\_\{1\}\. To make this asymmetry explicit in notation, we use the notationℐr​\(ztrainr,ztest\)\\mathcal\{I\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\):

ℐr​\(ztrainr,ztest\)=ℐ​\(ztrainr,\(hθ1​\(xtest\),ytest\)\)=ℐ​\(ztrainr,ztestr\),\\mathcal\{I\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\\mathcal\{I\}\(z^\{r\}\_\{\\text\{train\}\},\(h\_\{\\theta\_\{1\}\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)\)=\\mathcal\{I\}\(z^\{r\}\_\{\\text\{train\}\},z^\{r\}\_\{\\text\{test\}\}\),\(10\)where the superscriptrronℐ\\mathcal\{I\}signals that attribution targets the*training representation*\. The core challenge lies in attributinggtraing\_\{\\text\{train\}\}to individual latent coordinates ofrtrainr\_\{\\text\{train\}\}in a manner that reflects their actual computational attribution,222Decomposinggtestg\_\{\\text\{test\}\}would answer a different question \(e\.g\., “which test features make it sensitive to training data”\)\. The HessianHθ2H\_\{\\theta\_\{2\}\}depends solely on training data, and the perturbation is applied to the training point; thus,rtestr\_\{\\text\{test\}\}’s internal composition is irrelevant for the attribution we seek\.which we articulate in Section[4\.3](https://arxiv.org/html/2605.12809#S4.SS3)\. Now, by mapping these influential features back to the specific words that*activate*them, our explanations more faithfully capture the model’s internal reasoning—moving beyond isolated token attributions toward coherent, feature\-driven interpretability\.

### 4\.3Feature\-Level Attribution via Integrated Gradients

Recall that our goal is to attribute the training\-side gradient to individual latent coordinates ofrtrainr\_\{\\text\{train\}\}\. To make this dependence explicit and to evaluate it at different inputs, we define a functionGGthat any intermediate representationrrto the corresponding gradient onθ2\\theta\_\{2\}:

G​\(r\)=∇θ2ℓ​\(hθ2​\(r\),ytrain\),G\(r\)\\;=\\;\\nabla\_\{\\theta\_\{2\}\}\\ell\\\!\\left\(h\_\{\\theta\_\{2\}\}\(r\),y\_\{\\text\{train\}\}\\right\),\(11\)with the loss evaluated using the fixed training labelytrainy\_\{\\text\{train\}\}\. By construction,gtrain=G​\(rtrain\)g\_\{\\text\{train\}\}=G\(r\_\{\\text\{train\}\}\)\.

In contrast to token attribution in autoregressive tasks,G​\(r\)G\(r\)is a single vector within the high\-dimensional parameter space ofθ2\\theta\_\{2\}; consequently, it cannot be linearly separated into distinct components for individual neurons\. A naive approach would be to consider the directional derivative∂G/∂r\(j\)\\partial G/\\partial r^\{\(j\)\}, which measures how an infinitesimal perturbation to neuronjjaffects the gradient\. While this captures local sensitivity, it does not tell us how much that neuron’s*activation*contributes to the actual value ofG​\(r\)G\(r\)when transitioning from an inactive to an active state\. This issue is commonly addressed by defining contributions relative to a baseline representing the absence of the features \(e\.g\., integrated gradients\(Sundararajanet al\.,[2017a](https://arxiv.org/html/2605.12809#bib.bib612)\), Shapley values\(Lundberg and Lee,[2017](https://arxiv.org/html/2605.12809#bib.bib1093)\)\)\. For sparse representations learned by SAEs, the natural baseline isr0=𝟎r\_\{0\}=\\mathbf\{0\}, corresponding to the manifold where “all features are inactive\.” We therefore consider the changeΔ​G=G​\(rtrain\)−G​\(𝟎\)\\Delta G=G\(r\_\{\\text\{train\}\}\)\-G\(\\mathbf\{0\}\), which captures the effect of turning on the active features\.

To decomposeΔ​G\\Delta Ginto per\-neuron attributions, we adopt the axiomatic framework of integrated gradients\(Sundararajanet al\.,[2017a](https://arxiv.org/html/2605.12809#bib.bib612)\)\. This method attributes the output of a scalar model to its input features by integrating the gradients along a straight\-line path from a baseline to the realized representation\. We parametrize this straight\-line path byα∈\[0,1\]\\alpha\\in\[0,1\], and, abusing the notation, denote it asr​\(α\)r\(\\alpha\)\. Because our baseline is the origin \(r0=𝟎r\_\{0\}=\\mathbf\{0\}\), this straight\-line path is the scaling functionr​\(α\)=α​rtrainr\(\\alpha\)=\\alpha r\_\{\\text\{train\}\}\. By the chain rule of calculus, the rate of change of the gradient along this path is driven entirely by the representation itself:

dd​α​G​\(r​\(α\)\)=JG​\(α​rtrain\)​d​r​\(α\)d​α=JG​\(α​rtrain\)​rtrain\.\\frac\{d\}\{d\\alpha\}G\(r\(\\alpha\)\)=J\_\{G\}\(\\alpha r\_\{\\text\{train\}\}\)\\frac\{dr\(\\alpha\)\}\{d\\alpha\}=J\_\{G\}\(\\alpha r\_\{\\text\{train\}\}\)r\_\{\\text\{train\}\}\.\(12\)
Applying the fundamental theorem of calculus, we integrate this derivative to express the total gradient change:

G​\(rtrain\)−G​\(𝟎\)=∫01JG​\(α​rtrain\)​rtrain​𝑑α,G\(r\_\{\\text\{train\}\}\)\-G\(\\mathbf\{0\}\)=\\int\_\{0\}^\{1\}J\_\{G\}\(\\alpha r\_\{\\text\{train\}\}\)\\,r\_\{\\text\{train\}\}\\,d\\alpha,\(13\)whereJG​\(r\)∈ℝ\|θ2\|×dlJ\_\{G\}\(r\)\\in\\mathbb\{R\}^\{\|\\theta\_\{2\}\|\\times d\_\{l\}\}is the Jacobian ofGGwith respect torr\. Expanding the product coordinate\-wise yields an exact additive latent decomposition:

G​\(rtrain\)=G​\(𝟎\)\+∑j=1dlrtrain\(j\)​\(∫01JG​\(α​rtrain\)​ej​𝑑α\)\.G\(r\_\{\\text\{train\}\}\)=G\(\\mathbf\{0\}\)\+\\sum\_\{j=1\}^\{d\_\{l\}\}r\_\{\\text\{train\}\}^\{\(j\)\}\\left\(\\int\_\{0\}^\{1\}J\_\{G\}\(\\alpha r\_\{\\text\{train\}\}\)\\,e\_\{j\}\\,d\\alpha\\right\)\.\(14\)We provide a detailed proof of this decomposition in Appendix[D\.1](https://arxiv.org/html/2605.12809#A4.SS1)\.

Computing the exact integral is expensive\. To obtain a scalable estimator, we approximate the integral by evaluating the Jacobian at a particular point along the path,r⋆r^\{\\star\}\. This represents a backward first\-order Taylor expansion from the realized activations, defining our per\-neuron latent contribution as:

Δ​Gj​\(r⋆\):=rtrain\(j\)​JG​\(r⋆\)​ej\.\\Delta G\_\{j\}\(r^\{\\star\}\)\\;:=\\;r\_\{\\text\{train\}\}^\{\(j\)\}\\,J\_\{G\}\(r^\{\\star\}\)\\,e\_\{j\}\.\(15\)
By choosingr⋆=rtrainr^\{\\star\}=r\_\{\\text\{train\}\}, this yields a computationally\-efficient approximation because it evaluates the Jacobian using activations already materialized during the standard forward pass\. Moreover, the multiplicative factorrtrain\(j\)r\_\{\\text\{train\}\}^\{\(j\)\}guarantees that contributions correctly vanish for inactive features, satisfying a desirable*activation*sensitivity property\.

###### Definition 4\.2\(Neuron\-Level Influence\)\.

Using Eq\. \([15](https://arxiv.org/html/2605.12809#S4.E15)\) to adjust for activation attribution, the influence score attributed to neuronjjin the training representationrtrainr\_\{\\text\{train\}\}is defined as

ℐjr​\(ztrainr,ztest\)=−gtest⊤​Hθ2−1​Δ​Gj​\(rtrain\),\\mathcal\{I\}\_\{j\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\,g\_\{\\text\{test\}\}^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}\\,\\Delta G\_\{j\}\(r\_\{\\text\{train\}\}\),\(16\)wheregtest=∇θ2ℓ​\(hθ1​\(xtest\),ytest\)g\_\{\\text\{test\}\}=\\nabla\_\{\\theta\_\{2\}\}\\ell\(h\_\{\\theta\_\{1\}\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)andHθ2H\_\{\\theta\_\{2\}\}is the downstream Hessian\.

This formulation attributes influence to a neuron only if \(i\) it is active in the representation \(rtrain\(j\)≠0r\_\{\\text\{train\}\}^\{\(j\)\}\\neq 0\) and \(ii\) its activation induces a change in the downstream parameter gradient, as quantified by the scaled Jacobian columnJG​\(rtrain\)​ejJ\_\{G\}\(r\_\{\\text\{train\}\}\)\\,e\_\{j\}\. The sign convention ensures that positive influence corresponds to improved train–test gradient alignment\. Consequently, the score provides a causal, influence\-based measure of each neuron’s contribution, offering a rigorous alternative to correlational metrics like raw activation magnitude\(Koh and Liang,[2017](https://arxiv.org/html/2605.12809#bib.bib9); Geigeret al\.,[2021](https://arxiv.org/html/2605.12809#bib.bib243)\)\.

### 4\.4Scaling Influence From JVPs to Constant\-Time Derivative Swapping

In Definition[4\.2](https://arxiv.org/html/2605.12809#S4.Thmtheorem2), the contribution of neuronjjrelies on the Jacobian columnJG​\(rtrain\)​ejJ\_\{G\}\(r\_\{\\text\{train\}\}\)\\,e\_\{j\}\. For LLMs, materializing the full JacobianJG∈ℝ\|θ2\|×dlJ\_\{G\}\\in\\mathbb\{R\}^\{\|\\theta\_\{2\}\|\\times d\_\{l\}\}is memory\-prohibitive\. A standard circumvention is to compute this column implicitly using a Jacobian\-vector product \(JVP\)\(Baydinet al\.,[2018](https://arxiv.org/html/2605.12809#bib.bib119)\)\. By the definition of the directional derivative, evaluating the Jacobian along the standard basis vectoreje\_\{j\}yields:

JG​\(rtrain\)​ej=dd​ε​G​\(rtrain\+ε​ej\)\|ε=0=JVP​\(G,rtrain,ej\)\.J\_\{G\}\(r\_\{\\text\{train\}\}\)\\,e\_\{j\}=\\left\.\\frac\{d\}\{d\\varepsilon\}G\(r\_\{\\text\{train\}\}\+\\varepsilon e\_\{j\}\)\\right\|\_\{\\varepsilon=0\}=\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\)\.\(17\)
Putting Eq\.s \([15](https://arxiv.org/html/2605.12809#S4.E15)\)\-\([17](https://arxiv.org/html/2605.12809#S4.E17)\) together, we obtain a computable form for the neuron\-level influence:

ℐjr​\(ztrainr,ztest\)=−gtest⊤​Hθ2−1​rtrain\(j\)​JVP​\(G,rtrain,ej\)\.\\mathcal\{I\}\_\{j\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\,g\_\{\\text\{test\}\}^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}\\,r\_\{\\text\{train\}\}^\{\(j\)\}\\,\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\)\.\(18\)Here, the JVP term quantifies the downstream gradient sensitivity, while the activation factor \(rtrain\(j\)r\_\{\\text\{train\}\}^\{\(j\)\}\) enforces that only active circuits contribute\. This directly connects data\-level influence attribution to feature\-level monosemanticity\(Brickenet al\.,[2023](https://arxiv.org/html/2605.12809#bib.bib30)\)\.

The Computational Bottleneck\.While Eq\. \([18](https://arxiv.org/html/2605.12809#S4.E18)\) avoids storing the full Jacobian, its evaluation requires computing a separate JVP for*every*active featurejj, making it computationally impractical\. BecauseGGmaps to the high\-dimensional parameter spaceθ2\\theta\_\{2\}, each JVP requires an expensive forward\-over\-reverse differentiation pass \(𝒪​\(dl\)\\mathcal\{O\}\(d\_\{l\}\)passes\)\. For SAEs where thousands of features may fire simultaneously across a sequence, iterating over neurons individually becomes computationally intractable\. We therefore need a method that computes all neuron influences in constant time, independent of the number of active features\.

Constant\-Time Influence via Derivative Swapping\.The key insight is to exchange the order of differentiation and summation \(via Clairaut’s Theorem\), enabling us to obtain all feature influences from a single backward pass\.

Letstest=Hθ2−1​gtests\_\{\\text\{test\}\}=H\_\{\\theta\_\{2\}\}^\{\-1\}g\_\{\\text\{test\}\}and defineP​\(rtrain\)=stest⊤​gtrain\.P\(r\_\{\\text\{train\}\}\)\\;=\\;s\_\{\\text\{test\}\}^\{\\top\}g\_\{\\text\{train\}\}\.Sincestests\_\{\\text\{test\}\}is constant w\.r\.t\.rtrainr\_\{\\text\{train\}\}, we have

∇rtrainP=∇rtrain\(stest⊤​gtrain\),\\nabla\_\{r\_\{\\text\{train\}\}\}P=\\nabla\_\{r\_\{\\text\{train\}\}\}\\Big\(s\_\{\\text\{test\}\}^\{\\top\}g\_\{\\text\{train\}\}\\Big\),\(19\)and the vector of latent influences is

ℐ→r​\(ztrainr,ztest\)=−\(∇rtrainP\)⊙rtrain,\\vec\{\\mathcal\{I\}\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\Big\(\\nabla\_\{r\_\{\\text\{train\}\}\}P\\Big\)\\odot r\_\{\\text\{train\}\},\(20\)reducing complexity from𝒪​\(dl\)\\mathcal\{O\}\(d\_\{l\}\)to𝒪​\(1\)\\mathcal\{O\}\(1\)backward passes \(compute∇θ2ℓ\\nabla\_\{\\theta\_\{2\}\}\\ell, then backpropagatePPtortrainr\_\{\\text\{train\}\}\)\.

This constant\-time formulation enables simultaneous computation of influence for all latent features and batch samples, allowing the method to scale to billion\-parameter models and sparse autoencoders with tens of thousands of features, which significantly improves the efficiency of the framework\. The full proof is provided in Appendix[D\.2](https://arxiv.org/html/2605.12809#A4.SS2)\. For speedup and memory\-saving analysis, we redirect readers to Appendix[C](https://arxiv.org/html/2605.12809#A3)\.

## 5Experiments

Table 1:Finetuning results across base models\.Δ\\Deltadenotes LoRA\-SFT minus baseline accuracy\.We evaluate on three multiple\-choice QA benchmarks: MedQAHanet al\.\([2023](https://arxiv.org/html/2605.12809#bib.bib1090)\), OpenBookQAMihaylovet al\.\([2018](https://arxiv.org/html/2605.12809#bib.bib1091)\), and CommonsenseQATalmoret al\.\([2019](https://arxiv.org/html/2605.12809#bib.bib1092)\)\. All experiments start from task\-finetuned models, and influence is measured with respect to the resulting task adapted model at inference time\. We use OpenBookQA and CommonsenseQA as the primary quantitative benchmark in the main text, and defer additional dataset results and full experimental details to the appendix \(Appendix[B](https://arxiv.org/html/2605.12809#A2)\)\. We study small open\-weight, instruction\-tuned LLMs from the Llama and Qwen families \(∼\\sim1B parameters\)\.

Setup and PipelineFor a given task, the pipeline proceeds as follows: \(1\) finetune a pretrained LLM on the task dataset \(Sec[5\.1](https://arxiv.org/html/2605.12809#S5.SS1)\); \(2\) insert an SAE at an intermediate layer and train it with frozen LLM weights to obtain sparse latent features \(Sec[5\.1](https://arxiv.org/html/2605.12809#S5.SS1)\); \(3\) pre\-filter candidate training examples using gradient similarity to reduce influence computation cost and evaluate pre\-filtering stability \(Sec[5\.3](https://arxiv.org/html/2605.12809#S5.SS3)\)\. \(4\) compute representation\-level influence scores at the SAE layer \(defined in Sec\.[4\.4](https://arxiv.org/html/2605.12809#S4.SS4)\) to rank training examples and latent features, and presents the quantitative evaluation \(Sec\.[5\.4](https://arxiv.org/html/2605.12809#S5.SS4)\)\.

Table 2:SAE insertion results on OpenbookQA\.Δ\\Deltadenotes\+SAEminus finetuned accuracy\.Table 3:Orthogonality statistics across representation spaces on OpenbookQA, Llama\-3\.2\-1B\. Lower is better for entanglement metrics, higher is better for rank and near\-orthogonality\.![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots/necessity_Llama3.2-1B-Instruct.png)\(a\)Llama\-3\.2\-1B\-Instruct \(necessity\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots/sufficiency_Llama3.2-1B-Instruct.png)\(b\)Llama\-3\.2\-1B\-Instruct \(sufficiency\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots/necessity_Qwen2.5-1.5B-Instruct.png)\(c\)Qwen2\.5\-1\.5B\-Instruct \(necessity\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots/sufficiency_Qwen2.5-1.5B-Instruct.png)\(d\)Qwen2\.5\-1\.5B\-Instruct \(sufficiency\)\.

Figure 3:Necessity and sufficiency tests on OpenbookQA\. For necessity, we rank and remove top\-kkSAE features using RepInf and three baselines, then reportΔ\\Deltalogit,Δ\\DeltaNLL, and flip rate after masking\. For sufficiency, we keep only top\-kkfeatures and report the retained correct logit and correct rate\.### 5\.1Task Performance After Finetuning and SAE Insertion

A basic requirement for practical auditing is that 1\) finetuning yields meaningful task\-specific updates and 2\) inserting the SAE layer does not significantly degrade the task performance\. We therefore report \(i\) pretrained and LoRA\-SFT accuracy across base models in Table[1](https://arxiv.org/html/2605.12809#S5.T1)and \(ii\) the accuracy change from inserting an SAE at the chosen layer in Table[2](https://arxiv.org/html/2605.12809#S5.T2)\. We additionally report full sweeps over SAE insertion layers across datasets and models in Appendix[A\.1](https://arxiv.org/html/2605.12809#A1.SS1), showing that performance is sensitive to the placement of the representation bottleneck\.

LoRA Supervised Finetuning\.Table[1](https://arxiv.org/html/2605.12809#S5.T1)shows that LoRA\-SFT yields substantial gains across all base models, with the largest improvements for Llama\-3\.2\-1B \(\+44\.2%/40\.22%\) and Llama\-3\.2\-1B\-IT \(\+30\.8%/21\.79%\)\. Qwen2\.5\-1\.5B starts from a stronger baseline and thus shows smaller but still meaningful gains\. These results confirm that finetuning produces large, measurable task specific updates, which is a prerequisite for downstream influence analysis\.

SAE insertion\.Table[2](https://arxiv.org/html/2605.12809#S5.T2)shows that inserting an SAE at an intermediate layer largely preserves finetuned performance: Qwen2\.5\-1\.5B\-IT drops by only 0\.8 points at its selected \(and best\) layer, whereas Llama\-3\.2\-1B\-IT drops by 2\.4 points at the selected layer\. However, the “best layer” results indicate that most of Llama’s degradation can be recovered by a nearby layer choice \(down to a 0\.2 point drop\), suggesting that the main cost comes from bottleneck placement rather than the SAE mechanism itself; layer selection is therefore a key hyperparameter for balancing interpretability and accuracy\.

Layer selection logic\.Rather than selecting the globally best post\-insertion layer, we choose the best layer within the middle half of the network \(late layers can be less informative\): layers 4–12 for Llama\-3\.2\-1B\-Instruct and 7–22 for Qwen2\.5\-1\.5B\-Instruct\. We report both the*selected*and*best*layers, and provide the full sweep in the Appendix[A\.1](https://arxiv.org/html/2605.12809#A1.SS1)\.

### 5\.2Latent Space Orthogonality Analysis

We evaluate representation disentanglement by comparing feature orthogonality across three spaces: input text embeddings \(text\), dense pre\-latent activations \(immediately before SAE insertion,Pre\-Latent\), and thekk\-sparse SAE latent space \(SAE Latent\)\. Following prior work, we summarize orthogonality using the off\-diagonal Gram statistics \(mean absolute value, mean squared value, and Frobenius norm\), the fraction of near orthogonal feature pairs with correlation\|ρ\|<0\.1\|\\rho\|<0\.1, and stable rank‖A‖F2/‖A‖22\|\|A\|\|\_\{F\}^\{2\}/\|\|A\|\|\_\{2\}^\{2\}\(Fel and others,[2025](https://arxiv.org/html/2605.12809#bib.bib1095)\)\. Table[3](https://arxiv.org/html/2605.12809#S5.T3)shows a clear separation: pre\-SAE latents are highly entangled with stable rank 1\.17, 2\.0% near\-orthogonal pairs, which says that most of the features are highly entangled with each other\. Whereas SAE latents with high latent dimensions are substantially more disentangled with stable rank 25\.02 and 98\.67% near orthogonal pairs everywhere, which proves that latents are naturally highly entangled even without proper guidance or constraints\. Then, compared with the main objective text embeddings that we claim to try to disentangle in this paper, the text embeddings on average have a stable rank of 5\.35, 64\.9% near\-orthogonal pairs, the SAE recovers more orthogonal directions while reducing pairwise correlation magnitude, supporting the use of sparse latents for influence estimation\. The specification of the SAE latent size and an ablation of the latent space orthogonality are included in Appendix[A\.2](https://arxiv.org/html/2605.12809#A1.SS2)\.

### 5\.3Gradient Pre\-Filtering for Scalable Influence Computation

Exact influence computation over the full training set is expensive\. We therefore pre\-filter training examples using gradient similarity and retain only the top 1%–10% candidates per test example\. Concretely, for a test exampleztestz\_\{\\mathrm\{test\}\}and each training exampleziz\_\{i\}, we score

si=⟨∇θℒ​\(zi;θ\),∇θℒ​\(ztest;θ\)⟩,s\_\{i\}\\;=\\;\\left\\langle\\nabla\_\{\\theta\}\\mathcal\{L\}\\\!\\left\(z\_\{i\};\\theta\\right\),\\,\\nabla\_\{\\theta\}\\mathcal\{L\}\\\!\\left\(z\_\{\\mathrm\{test\}\};\\theta\\right\)\\right\\rangle,\(21\)and keep𝒞K​\(ztest\)=TopKi​\(si\)\\mathcal\{C\}\_\{K\}\(z\_\{\\mathrm\{test\}\}\)=\\mathrm\{TopK\}\_\{i\}\(s\_\{i\}\)\. We defer additional experiments on filtering thresholds and stability to Appendix[B\.3](https://arxiv.org/html/2605.12809#A2.SS3.SSS0.Px1)\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots_commonsense/necessity_Llama-3.2-1B-Instruct.png)\(a\)Llama\-3\.2\-1B\-Instruct \(necessity\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots_commonsense/sufficiency_Llama-3.2-1B-Instruct.png)\(b\)Llama\-3\.2\-1B\-Instruct \(sufficiency\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots_commonsense/necessity_Qwen2.5-1.5B-Instruct.png)\(c\)Qwen2\.5\-1\.5B\-Instruct \(necessity\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/plots_commonsense/sufficiency_Qwen2.5-1.5B-Instruct.png)\(d\)Qwen2\.5\-1\.5B\-Instruct \(sufficiency\)\.

Figure 4:CommonsenseQA necessity \(remove top\-kkfeatures\) and sufficiency \(keep top\-kkfeatures\) tests comparing influence\-selected features to baselines\.![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test_146.png)\(a\)Original test input \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test_146_train1655.png)\(b\)Most influential training sample \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test146_train1655_feature2.png)\(c\)Individual feature 1338\.

Figure 5:Token\-level influence visualizations for a representative OpenBookQA test question \(top\), its most influential training example \(middle\), and its top influential token heatmaps \(bottom\)\. Colors denote the dominant influential latent feature and intensities denote per\-token influence\.
### 5\.4Quantitative Evaluation: Necessity and Sufficiency Tests

To evaluate the faithfulness of sparse feature influence, we run two deletion/insertion style tests\. For each evaluation sequence, we record the baseline prediction and correct\-token statistics, then apply a binary mask at the SAE layer using a chosen ranking over latent features\.

Our core signal is the representation level influence matrixIFR∈ℝN×H\\mathrm\{IFR\}\\in\\mathbb\{R\}^\{N\\times H\}for a test sample, whereNNis the number of retained training examples after gradient pre filtering andHHis the SAE latent dimension;IFR​\[i,k\]\\mathrm\{IFR\}\[i,k\]measures the influence of training exampleiion latent featurekk\. We aggregate over training examples \(mean overii\) to obtain a test conditioned importance vectors∈ℝHs\\in\\mathbb\{R\}^\{H\}withsk=1N​∑i=1NIFR​\[i,k\]s\_\{k\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathrm\{IFR\}\[i,k\], and take the top\-kkfeatures underssas the*most influential*for that test sample\.

We compare our influence score against three simple baselines: \(i\) ranking by activation magnitude, \(ii\) ranking by feature frequency, and \(iii\) a random set\. Concretely, for each method we induce an ordering over latent dimensions and construct:

- •Necessity \(Remove Top\-k\):A mask that zeroes out the top\-kfeatures, leaving other features intact\.
- •Sufficiency \(Keep Top\-k\):A mask that preserves only the top\-kranked features, zeroing all others\.

We evaluate both settings across multiple values ofk= \{25,50,100,125,150,175,200\}\. For necessity, we report the average change in the correct token’s logit \(Δ\\Deltalogit\), the average change in negative log likelihood \(Δ\\DeltaNLL\), and prediction flip rate relative to the baseline\. For sufficiency, we report the same answer rate and the retained mean correct logit under masking\.

Figures[3](https://arxiv.org/html/2605.12809#S5.F3)and[4](https://arxiv.org/html/2605.12809#S5.F4)demonstrate a consistent and quantitative separation between influence\-based ranking and all baselines across both Llama\-3\.2\-1B and Qwen2\.5\-1\.5B\.

Necessity\.Under the remove top\-kkintervention, influence produces the steepest and most systematic degradation\. On both models and both datasets, removing as few as 50 \- 100 influence\-ranked features induces a large negative shift in the correct token logit, a substantial increase in NLL, and a rapid rise in flip rate, often approaching saturation at moderatekk\. In contrast, random masking leads to gradual degradation, while activation magnitude and frequency exhibit intermediate behavior but consistently weaker impact\. The approximately monotonic dependence ofΔ\\Deltalogit,Δ\\DeltaNLL, and flip rate onkksuggests that influence induces a meaningful global ordering over latent features in terms of their causal contribution\.

Sufficiency\.The keep top\-kkexperiment exhibits the complementary pattern\. Influence retains substantially higher same answer rates and larger correct logits at everykk\. Notably, a relatively small subset of influence\-ranked features suffices to recover a large fraction of the original predictive confidence, whereas random and heuristic rankings require many more features and still fail to match the retained performance\. This indicates that predictive information is concentrated in a compact set of influence identified latent directions\.

Interpretation\.Taken together, the dual behavior: sharp degradation under removal and strong recovery under retention, provides evidence that influence ranking captures features that are not merely highly active or frequent, but structurally implicated in the model’s decision\. The consistency of this pattern across two distinct architectures further suggests that the effect reflects a representation level property rather than model specific artifacts\.

### 5\.5Qualitative Attribution: Linking Predictions to Training Evidence

We now present case studies showing how our method links a test\-time prediction to specific training evidence\. For each case, we report the model prediction, the most influential training example, and token\-level heatmaps obtained by projecting representation\-level influence back to the input space\.

#### Case study protocol\.

We focus on test instances that are answered incorrectly by the pretrained model but correctly after finetuning, so that the resulting attributions reflect task\-specific learning rather than generic priors\. We run the same procedure for both Llama\-3\.2\-1B\-Instruct and Qwen2\.5\-1\.5B\-Instruct; for readability, the main text shows one representative example, while additional cases, including Qwen\-based results, are deferred to Appendix[A\.3](https://arxiv.org/html/2605.12809#A1.SS3)\. We additionally provide a heatmap visualization under reasoning augmented training in Case Study D \(Figure[A\.4](https://arxiv.org/html/2605.12809#A1.F4)\), illustrating potential model behavior change when additional reasoning is provided\. The model setup used in this subsection is included in Appendix[B](https://arxiv.org/html/2605.12809#A2)\.

#### Case Study A: Representation\-level influence highlights shared astronomical reasoning cues\.

Figure[5](https://arxiv.org/html/2605.12809#S5.F5)shows an OpenBookQA question asking why Earth’s rotation on its axis makes Orion appear to change position in the sky\. RepInfLLM retrieves a highly influential training example with summed influence 44\.27: “the Earth rotating on its axis causes stars to appear to move across the sky when?”, whose wording differs from the test question but encodes the same causal relation\. The dominant latent features concentrate on the phrase group “Earth rotating,” “on its axis,” “causes stars,” “appear,” “move,” and “sky,” rather than on isolated lexical overlaps\. This pattern suggests that the prediction is supported by a shared representation of the concept “Earth’s rotation causes apparent celestial motion,” rather than by memorizing an answer string or matching option labels\. More broadly, the example illustrates the value of representation\-level influence: it surfaces semantically aligned training evidence and highlights the concept\-bearing tokens that mediate the model’s prediction\.

We further inspect individual high influence latent features for the retrieved training sample\. Latent 1338 primarily activates the concept bearing phrase “Earth rotating on its axis” together with “across the sky,” matching the causal astronomy relation required by the test question\. Latent 1835 captures a complementary pattern involving “Earth,” “causes,” “appear,” and temporal answer choices such as “day” and “daytime\.” Together, these latents suggest that the retrieved training example supports the prediction through multiple semantically coherent components: the physical mechanism of Earth’s rotation and the question\-specific temporal framing\. This provides more fine\-grained evidence than a single aggregate influence score\.

## 6Discussion

In this work, we introduce a novel interpretability framework based on IF that is applicable to any prediction task\. Our proposed methods scale to large models with reasonable runtimes\. Methodologically, our key contribution is the integration of SAEs into the LLM during fine\-tuning, enabling the computation of influence scores over approximately orthogonal latent representations\. By projecting these latent attributions back to the input space, our framework yields human\-interpretable insights while preserving the theoretical soundness of influence estimation\. Ablation studies demonstrate that our method achieves high necessity and sufficiency, confirming its ability to isolate the most influential latents driving a model’s prediction on a given test sample\. Beyond the current scope, this approach holds promise for multimodal settings, where it could serve as a diagnostic tool to interpret model behavior, identify failure modes, and assess performance bottlenecks across heterogeneous data modalities—offering a principled pathway toward more transparent and trustworthy multimodal systems\.

However, several limitations remain\. First, our method inherits the local nature of influence functions\. The influence score approximates the effect of infinitesimal perturbations around a fixed trained model, and therefore should not be interpreted as exact leave\-one\-out retraining\. It may miss nonlinear training effects, feature formation, or global representation changes that would occur under finite data removal or additional finetuning\. Our method improves where the influence is measured, but it does not eliminate this fundamental approximation\.

A second limitation is that SAE latents are only approximately disentangled\. Our orthogonality analysis shows that SAE representations are much less entangled than dense pre\-latent activations, but orthogonality does not guarantee semantic independence or causal modularity\. Some latent features may still mix multiple concepts, and some concepts may be distributed across several features\. Thus, the influence scores should be interpreted as attribution over a more structured latent basis, not as proof of fully independent causal concepts\. Furthermore, there is a trade\-off between the latent dimension size \(reflected by the orthogonality score\) and the interpretability of the final heatmap, which we plan to investigate in future work\.

Third, token\-level visualizations are derived projections from latent influence, not the primary causal quantity\. The strongest attribution produced by RepInfLLM is at the level of training examples and SAE features\. Mapping these features back to input tokens makes the explanation more readable, but this projection depends on activation patterns and can blur cases where one feature is activated by multiple correlated tokens\. Furthermore, the context in our datasets is relatively short, which may make it difficult for humans to achieve full interpretability of the latents\. Future work using longer contexts could help address this limitation\.

Finally, we do not report direct comparisons with standard data attribution methods because most prior approaches operate in input space and evaluate instance\-level rankings\. In contrast, our framework estimates representation\-level influence in a sparse latent space and only subsequently projects attributions back to text; conventional ground\-truth protocols therefore do not transfer directly\.

## Acknowledgments

We thank AWS for providing compute resources that supported prototyping of the framework\. We also thank the NVIDIA Academic Grant Program Award for providing 32K A100 GPU\-hours on Brev, which enabled us to scale up the experimental results\.

## References

- A\. Abdulaal, H\. Fry, N\. Montaña\-Brown, A\. Ijishakin, J\. Gao, S\. Hyland, D\. C\. Alexander, and D\. C\. Castro \(2024\)An x\-ray is worth 15 features: sparse autoencoders for interpretable radiology report generation\.arXiv preprint arXiv:2410\.03334\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- Refusal in language models is mediated by a single direction\.CoRR\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2406.11717),[Link](https://arxiv.org/abs/2406.11717)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- S\. Bach, A\. Binder, G\. Montavon, F\. Klauschen, K\. Müller, and W\. Samek \(2015\)On pixel\-wise explanations for non\-linear classifier decisions by layer\-wise relevance propagation\.PLOS ONE10\(7\),pp\. e0130140\(en\)\.External Links:ISSN 1932\-6203,[Document](https://dx.doi.org/10.1371/journal.pone.0130140),[Link](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- Y\. Bansal, P\. Nakkiran, and B\. Barak \(2021\)Revisiting model stitching to compare neural representations\.CoRR\.External Links:2106\.07682,[Document](https://dx.doi.org/10.48550/arXiv.2106.07682),[Link](http://arxiv.org/abs/2106.07682)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- F\. Barez, T\. Wu, I\. Arcuschin, M\. Lan, V\. Wang, N\. Siegel, N\. Collignon, C\. Neo, I\. Lee, A\. Paren, A\. Bibi, R\. Trager, D\. Fornasiere, J\. Yan, Y\. Elazar, and Y\. Bengio \(2025\)Chain\-of\-thought is not explainability\.alphaXiv\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1)\.
- P\. Basu and F\. Echenique \(2020\)On the falsifiability and learnability of decision theories\.Theoretical Economics15\(4\),pp\. 1279–1305\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p3.1)\.
- A\. G\. Baydin, B\. A\. Pearlmutter, A\. A\. Radul, and J\. M\. Siskind \(2018\)Automatic differentiation in machine learning: a survey\.Journal of Machine Learning Research18\(153\),pp\. 1–43\.External Links:[Link](http://jmlr.org/papers/v18/17-468.html)Cited by:[§4\.4](https://arxiv.org/html/2605.12809#S4.SS4.p1.4)\.
- Y\. Belinkov \(2022\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00422),[Link](https://aclanthology.org/2022.cl-1.7)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- B\. Bilodeau, N\. Jaques, P\. W\. Koh, and B\. Kim \(2024\)Impossibility theorems for feature attribution\.Proc\. Natl\. Acad\. Sci\. U\.S\.A\.121\(2\),pp\. e2304406120\.External Links:2212\.11870,ISSN 0027\-8424, 1091\-6490,[Document](https://dx.doi.org/10.1073/pnas.2304406120),[Link](http://arxiv.org/abs/2212.11870)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. Black, L\. Sharkey, L\. Grinsztajn, E\. Winsor, D\. Braun, J\. Merizian, K\. Parker, C\. R\. Guevara, B\. Millidge, G\. Alfour, and C\. Leahy \(2022\)Interpreting neural networks through the polytope lens\.CoRR\.External Links:[Link](https://arxiv.org/abs/2211.12312)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.htmlCited by:[§1](https://arxiv.org/html/2605.12809#S1.p4.1),[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1),[§4\.1](https://arxiv.org/html/2605.12809#S4.SS1.p1.3),[§4\.1](https://arxiv.org/html/2605.12809#S4.SS1.p1.8),[§4\.4](https://arxiv.org/html/2605.12809#S4.SS4.p2.1)\.
- C\. Burns, H\. Ye, D\. Klein, and J\. Steinhardt \(2023\)Discovering latent knowledge in language models without supervision\.ICLR\.External Links:2212\.03827,[Link](http://arxiv.org/abs/2212.03827)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- B\. Bussmann, P\. Leask, and N\. Nanda \(2024\)Batchtopk sparse autoencoders\.arXiv preprint arXiv:2412\.06410\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- N\. Cammarata, G\. Goh, S\. Carter, L\. Schubert, M\. Petrov, and C\. Olah \(2020\)Curve detectors\.Distill\.External Links:[Link](https://distill.pub/2020/circuits/curve-detectors)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- N\. Cammarata, G\. Goh, S\. Carter, C\. Voss, L\. Schubert, and C\. Olah \(2021\)Curve circuits\.Distill\.External Links:[Link](https://distill.pub/2020/circuits/curve-circuits/)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- G\. Casalicchio, C\. Molnar, and B\. Bischl \(2018\)Visualizing the feature importance for black box models\.ECML PKDD11051,pp\. 655–670\.External Links:1804\.06620,[Document](https://dx.doi.org/10.1007/978-3-030-10925-7%5F40),[Link](http://arxiv.org/abs/1804.06620)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- L\. Chan, L\. Lang, and E\. Jenner \(2023\)Natural abstractions: key claims, theorems, and critiques\.AI Alignment Forum\(en\)\.External Links:[Link](https://www.alignmentforum.org/posts/gvzW46Z3BsaZsLc25/natural-abstractions-key-claims-theorems-and-critiques-1)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- D\. Chanin, A\. Hunter, and O\. Camburu \(2023\)Identifying linear relational concepts in large language models\.CoRR\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2311.08968),[Link](https://arxiv.org/abs/2311.08968)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- L\. W\. Cong, G\. Feng, J\. He, and J\. Li \(2023\)Sparse modeling under grouped heterogeneity with an application to asset pricing\.Technical reportNational Bureau of Economic Research\.Cited by:[item 2](https://arxiv.org/html/2605.12809#S1.I1.i2.p1.1)\.
- I\. C\. Covert, S\. Lundberg, and S\. Lee \(2021\)Explaining by removing: a unified framework for model explanation\.J\. Mach\. Learn\. Res\.22\(1\),pp\. 209:9477–209:9566\.External Links:ISSN 1532\-4435,[Link](https://arxiv.org/abs/2011.14878)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2023\)Sparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[item 2](https://arxiv.org/html/2605.12809#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- M\. Deng, L\. Tao, and J\. Benton \(2023\)Measuring feature sparsity in language models\.CoRR\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2310.07837),[Link](https://arxiv.org/abs/2310.07837)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- J\. Dunefsky, P\. Chlenski, and N\. Nanda \(2024\)Transcoders find interpretable LLM feature circuits\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=J6zHcScAo0)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1),[§4\.2](https://arxiv.org/html/2605.12809#S4.SS2.p1.4)\.
- N\. Elhage, T\. Hume, O\. Catherine, N\. Neel, T\. Henighan, S\. Johnston, S\. ElShowk, N\. Joseph, N\. DasSarma, B\. Mann, D\. Hernandez, A\. Askell, K\. Ndousse, D\. Drain, A\. Chen, Y\. Bai, D\. Ganguli, L\. Lovitt, Z\. Hatfield\-Dodds, J\. Kernion, T\. Conerly, S\. Kravec, S\. Fort, S\. Kadavath, J\. Jacobson, E\. Tran\-Johnson, J\. Kaplan, J\. Clark, T\. Brown, S\. McCandlish, D\. Amodei, and C\. Olah \(2022a\)Softmax linear units\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/solu/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen,et al\.\(2022b\)Toy models of superposition\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/toy_model/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- N\. Elhage, R\. Lasenby, and C\. Olah \(2023\)Privileged bases in the transformer residual stream\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2023/privileged-basis/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- J\. Engels, I\. Liao, E\. J\. Michaud, W\. Gurnee, and M\. Tegmark \(2024\)Not all language model features are linear\.CoRR\.External Links:2405\.14860,[Document](https://dx.doi.org/10.48550/arXiv.2405.14860),[Link](http://arxiv.org/abs/2405.14860)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- T\. Felet al\.\(2025\)Archetypal sae: adaptive and stable dictionary learning for concept extraction in large vision models\.arXiv preprint\.Cited by:[§5\.2](https://arxiv.org/html/2605.12809#S5.SS2.p1.3)\.
- V\. Feldman and C\. Zhang \(2020\)What neural networks memorize and why: discovering the long tail via influence estimation\.Advances in Neural Information Processing Systems33,pp\. 2881–2891\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1)\.
- J\. Futoma, M\. Simons, T\. Panch, F\. Doshi\-Velez, and L\. A\. Celi \(2020\)The myth of generalisability in clinical research and machine learning in health care\.The Lancet Digital Health2\(9\),pp\. e489–e492\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu \(2024\)Scaling and evaluating sparse autoencoders\.arXiv preprint arXiv:2406\.04093\.Cited by:[item 2](https://arxiv.org/html/2605.12809#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§4\.1](https://arxiv.org/html/2605.12809#S4.SS1.p1.3),[§4\.1](https://arxiv.org/html/2605.12809#S4.SS1.p1.8)\.
- A\. Geiger, H\. Lu, T\. Icard, and C\. Potts \(2021\)Causal abstractions of neural networks\.NeurIPS34,pp\. 9574–9586\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/4f5c422f4d49a5a807eda27434231040-Abstract.html)Cited by:[§4\.3](https://arxiv.org/html/2605.12809#S4.SS3.p7.2)\.
- M\. Ghassemi, L\. Oakden\-Rayner, and A\. L\. Beam \(2021\)The false hope of current approaches to explainable artificial intelligence in health care\.The lancet digital health3\(11\),pp\. e745–e750\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- R\. Grosse, J\. Bae, C\. Anil, N\. Elhage, A\. Tamkin, A\. Tajdini, B\. Steiner, D\. Li, E\. Durmus, E\. Perez,et al\.\(2023\)Studying large language model generalization with influence functions\.arXiv preprint arXiv:2308\.03296\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p3.1),[§3](https://arxiv.org/html/2605.12809#S3.SS0.SSS0.Px1.p2.4)\.
- W\. Gurnee, N\. Nanda, M\. Pauly, K\. Harvey, D\. Troitskii, and D\. Bertsimas \(2023\)Finding neurons in a haystack: case studies with sparse probing\.TMLR\.External Links:[Link](https://arxiv.org/abs/2305.01610)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- F\. R\. Hampel \(1974\)The influence curve and its role in robust estimation\.Journal of the american statistical association69\(346\),pp\. 383–393\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p3.1)\.
- T\. Han, L\. C\. Adams, J\. Papaioannou, P\. Grundmann, T\. Oberhauser, A\. Löser, D\. Truhn, and K\. K\. Bressem \(2023\)MedAlpaca–an open\-source collection of medical conversational ai models and training data\.arXiv preprint arXiv:2304\.08247\.Cited by:[§5](https://arxiv.org/html/2605.12809#S5.p1.1)\.
- R\. Hendel, M\. Geva, and A\. Globerson \(2023\)In\-context learning creates task vectors\.EMNLP\.External Links:2310\.15916,[Document](https://dx.doi.org/10.48550/arXiv.2310.15916),[Link](http://arxiv.org/abs/2310.15916)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- T\. Henighan, S\. Carter, T\. Hume, N\. Elhage, R\. Lasenby, S\. Fort, N\. Schiefer, and C\. Olah \(2023\)Superposition, memorization, and double descent\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2023/toy-double-descent/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- E\. Hernandez, A\. S\. Sharma, T\. Haklay, K\. Meng, M\. Wattenberg, J\. Andreas, Y\. Belinkov, and D\. Bau \(2023\)Linearity of relation decoding in transformer language models\.CoRR\.External Links:2308\.09124,[Document](https://dx.doi.org/10.48550/arXiv.2308.09124),[Link](http://arxiv.org/abs/2308.09124)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- A\. Hyvärinen \(2013\)Independent component analysis: recent advances\.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences371\(1984\),pp\. 20110534\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- M\. Ivanitskiy, A\. F\. Spies, T\. Rauker, G\. Corlouer, C\. Mathwin, L\. Quirke, C\. Rager, R\. Shah, D\. Valentine, C\. D\. Behn, K\. Inoue, and S\. W\. Fung \(2023\)Structured world representations in maze\-solving transformers\.CoRR\.External Links:[Link](https://arxiv.org/abs/2312.02566)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. Jain and B\. C\. Wallace \(2019\)Attention is not explanation\.arXiv preprint arXiv:1902\.10186\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1)\.
- janus \(2022\)Simulators\.LessWrong\(en\)\.External Links:[Link](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- I\. T\. Jolliffe and J\. Cadima \(2016\)Principal component analysis: a review and recent developments\.Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences374\(2065\),pp\. 20150202\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- A\. Karvonen \(2024\)Emergent world models and latent variable estimation in chess\-playing language models\.COLM\.External Links:2403\.15498,[Document](https://dx.doi.org/10.48550/arXiv.2403.15498),[Link](http://arxiv.org/abs/2403.15498)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- D\. P\. Kingma and M\. Welling \(2014\)Auto\-encoding variational bayes\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- P\. W\. Koh and P\. Liang \(2017\)Understanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p3.1),[§4\.3](https://arxiv.org/html/2605.12809#S4.SS3.p7.2)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of neural network representations revisited\.ICML\.External Links:1905\.00414,[Document](https://dx.doi.org/10.48550/arXiv.1905.00414),[Link](http://arxiv.org/abs/1905.00414)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. G\. Krantz and H\. R\. Parks \(2002\)The implicit function theorem: history, theory, and applications\.Springer Science & Business Media\.Cited by:[§3](https://arxiv.org/html/2605.12809#S3.SS0.SSS0.Px1.p1.11)\.
- K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023\)Emergent world representations: exploring a sequence model trained on a synthetic task\.ICLR\.External Links:[Link](https://arxiv.org/abs/2210.13382)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems 30,pp\. 4765–4774\.Cited by:[§4\.3](https://arxiv.org/html/2605.12809#S4.SS3.p2.7)\.
- A\. Makhzani and B\. Frey \(2013\)K\-sparse autoencoders\.arXiv preprint arXiv:1312\.5663\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- G\. L\. Marchetti, C\. Hillar, D\. Kragic, and S\. Sanborn \(2023\)Harmonics of learning: universal fourier features emerge in invariant networks\.CoRR\.External Links:2312\.08550,[Document](https://dx.doi.org/10.48550/arXiv.2312.08550),[Link](http://arxiv.org/abs/2312.08550)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. Mueller \(2024\)Sparse feature circuits: discovering and editing interpretable causal graphs in language models\.arXiv preprint arXiv:2403\.19647\.Cited by:[item 2](https://arxiv.org/html/2605.12809#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- C\. McDougall, A\. Conmy, C\. Rushing, T\. McGrath, and N\. Nanda \(2023\)Copy suppression: comprehensively understanding an attention head\.CoRR\.External Links:2310\.04625,[Document](https://dx.doi.org/10.48550/arXiv.2310.04625),[Link](http://arxiv.org/abs/2310.04625)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.NeurIPS\.External Links:2202\.05262,[Document](https://dx.doi.org/10.48550/arXiv.2202.05262),[Link](http://arxiv.org/abs/2202.05262)Cited by:[§4\.2](https://arxiv.org/html/2605.12809#S4.SS2.p1.4)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2381–2391\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§5](https://arxiv.org/html/2605.12809#S5.p1.1)\.
- N\. Nanda \(2022\)200 cop in mi: interpreting algorithmic problems\.Neel Nanda’s Blog\(en\)\.External Links:[Link](https://www.lesswrong.com/posts/ejtFsvyhRkMofKAFy/200-cop-in-mi-interpreting-algorithmic-problems)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- N\. Nanda \(2023\)Actually, othello\-gpt has a linear emergent world representation\.Neel Nanda’s Blog\.External Links:[Link](https://neelnanda.io/mechanistic-interpretability/othello)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- L\. O’Mahony, V\. Andrearczyk, H\. Muller, and M\. Graziani \(2023\)Disentangling neuron representations with concept vectors\.CVPR Workshops\.External Links:2304\.09707,[Document](https://dx.doi.org/10.48550/arXiv.2304.09707),[Link](http://arxiv.org/abs/2304.09707)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- M\. Oberst and D\. Sontag \(2019\)Counterfactual off\-policy evaluation with gumbel\-max structural causal models\.InInternational Conference on Machine Learning,pp\. 4881–4890\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- C\. Olah, A\. Satyanarayan, I\. Johnson, S\. Carter, L\. Schubert, K\. Ye, and A\. Mordvintsev \(2018\)The building blocks of interpretability\.Distill\.External Links:[Link](https://distill.pub/2018/building-blocks)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, S\. Johnston, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2022\)In\-context learning and induction heads\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- K\. Park, Y\. J\. Choe, Y\. Jiang, and V\. Veitch \(2024\)The geometry of categorical and hierarchical concepts in large language models\.ICML MI Workshop \(Oral\)\(en\)\.External Links:[Link](https://openreview.net/forum?id=KXuYjuBzKo)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- S\. Rajamanoharan, A\. Conmy, L\. Smith, T\. Lieberum, V\. Varma, J\. Kramár, R\. Shah, and N\. Nanda \(2024\)Improving dictionary learning with gated sparse autoencoders\.CoRR\.External Links:2404\.16014,[Document](https://dx.doi.org/10.48550/arXiv.2404.16014),[Link](http://arxiv.org/abs/2404.16014)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- M\. T\. Ribeiro, S\. Singh, and C\. Guestrin \(2016\)"Why should i trust you?": explaining the predictions of any classifier\.NAACL\.External Links:1602\.04938,[Document](https://dx.doi.org/10.48550/arXiv.1602.04938),[Link](http://arxiv.org/abs/1602.04938)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- A\. Scherlis, K\. Sachan, A\. S\. Jermyn, J\. Benton, and B\. Shlegeris \(2023\)Polysemanticity and capacity in neural networks\.CoRR\.External Links:2210\.01892,[Document](https://dx.doi.org/10.48550/arXiv.2210.01892),[Link](http://arxiv.org/abs/2210.01892)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- R\. R\. Selvaraju, A\. Das, R\. Vedantam, M\. Cogswell, D\. Parikh, and D\. Batra \(2016\)Grad\-cam: why did you say that? visual explanations from deep networks via gradient\-based localization\.ICCV\.External Links:[Link](https://arxiv.org/abs/1610.02391)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- M\. Shanahan, K\. McDonell, and L\. Reynolds \(2023\)Role play with large language models\.Nature623\(7987\),pp\. 493–498\(en\)\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-023-06647-8),[Link](https://www.nature.com/articles/s41586-023-06647-8)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- L\. Sharkey \(2022\)Circumventing interpretability: how to defeat mind\-readers\.CoRR\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2212.11415),[Link](https://arxiv.org/abs/2212.11415)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- A\. Shrikumar, P\. Greenside, and A\. Kundaje \(2017\)Learning important features through propagating activation differences\.ICML\.External Links:1704\.02685,[Document](https://dx.doi.org/10.48550/arXiv.1704.02685),[Link](http://arxiv.org/abs/1704.02685)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- D\. Smilkov, N\. Thorat, B\. Kim, F\. Viégas, and M\. Wattenberg \(2017\)SmoothGrad: removing noise by adding noise\.CoRR\.External Links:1706\.03825,[Document](https://dx.doi.org/10.48550/arXiv.1706.03825),[Link](http://arxiv.org/abs/1706.03825)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017a\)Axiomatic attribution for deep networks\.ICML\.External Links:1703\.01365,[Document](https://dx.doi.org/10.48550/arXiv.1703.01365),[Link](http://arxiv.org/abs/1703.01365)Cited by:[§4\.3](https://arxiv.org/html/2605.12809#S4.SS3.p2.7),[§4\.3](https://arxiv.org/html/2605.12809#S4.SS3.p3.5)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017b\)Axiomatic attribution for deep networks\.InInternational conference on machine learning,pp\. 3319–3328\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- V\. Surkov, C\. Wendler, M\. Terekhov, J\. Deschenaux, R\. West, and C\. Gulcehre \(2025\)Unpacking sdxl turbo: interpreting text\-to\-image models with sparse autoencoders\.InMechanistic Interpretability for Vision at CVPR 2025 \(Non\-proceedings Track\),Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 4149–4158\.Cited by:[§5](https://arxiv.org/html/2605.12809#S5.p1.1)\.
- A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones, H\. Cunningham, N\. L\. Turner, C\. McDougall, M\. MacDiarmid, C\. D\. Freeman, T\. R\. Sumers, E\. Rees, J\. Batson, A\. Jermyn, S\. Carter, C\. Olah, and T\. Henighan \(2024a\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1),[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, and B\. Chen \(2024b\)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p4.1)\.
- C\. Tigges, O\. J\. Hollinsworth, A\. Geiger, and N\. Nanda \(2024\)Language models linearly represent sentiment\.ICML MI Workshop\(en\)\.External Links:[Link](https://openreview.net/forum?id=Xsf6dOOMMc)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- E\. Todd, M\. L\. Li, A\. S\. Sharma, A\. Mueller, B\. C\. Wallace, and D\. Bau \(2023\)Function vectors in large language models\.CoRR\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2310.15213),[Link](https://arxiv.org/abs/2310.15213)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p3.1)\.
- E\. J\. Topol \(2019\)High\-performance medicine: the convergence of human and artificial intelligence\.Nature medicine25\(1\),pp\. 44–56\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p1.1)\.
- M\. Tsimpoukelli, J\. L\. Menick, S\. Cabi, S\. Eslami, O\. Vinyals, and F\. Hill \(2021\)Multimodal few\-shot learning with frozen language models\.Advances in Neural Information Processing Systems34,pp\. 200–212\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p3.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.Advances in Neural Information Processing Systems36,pp\. 74952–74965\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1)\.
- K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt \(2023\)Interpretability in the wild: a circuit for indirect object identification in gpt\-2 small\.ICLR\.External Links:2211\.00593,[Document](https://dx.doi.org/10.48550/arXiv.2211.00593),[Link](http://arxiv.org/abs/2211.00593)Cited by:[§4\.2](https://arxiv.org/html/2605.12809#S4.SS2.p1.4)\.
- X\. Wang, H\. Guo, S\. Jha, and R\. Gao \(2024\)Disentangled representation learning\.arXiv preprint arXiv:2211\.11695\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p4.1)\.
- A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. Bowman \(2020\)BLiMP: the benchmark of linguistic minimal pairs for english\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321),[Link](https://aclanthology.org/2020.tacl-1.25)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.
- S\. Wiegreffe and Y\. Pinter \(2019\)Attention is not not explanation\.arXiv preprint arXiv:1908\.04626\.Cited by:[§1](https://arxiv.org/html/2605.12809#S1.p2.1)\.
- Q\. Yin, C\. T\. Leong, H\. Zhang, M\. Zhu, H\. Yan, Q\. Zhang, Y\. He, W\. Li, J\. Wang, Y\. Zhang,et al\.\(2024\)Direct preference optimization using sparse feature\-level constraints\.arXiv preprint arXiv:2411\.07618\.Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p2.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023\)Representation engineering: a top\-down approach to ai transparency\.CoRR\.External Links:2310\.01405,[Document](https://dx.doi.org/10.48550/arXiv.2310.01405),[Link](http://arxiv.org/abs/2310.01405)Cited by:[§2](https://arxiv.org/html/2605.12809#S2.p1.1)\.

## Appendix

## Appendix AFull Experimental Results

This appendix contains the full results that the main experimental results haven’t shown fully\.

### A\.1SAE Layer Sweep and Training Results

We sweep SAE insertion layers and SAE configurations across benchmarks\. Accuracy is typically best preserved in intermediate layers, and can degrade when inserting too early \(low\-level representations\) or too late \(near task\-head computations\)\. In all plots, the*shaded region*denotes the*middle\-half*of the network layers considered in the main experiments \(Section[5\.1](https://arxiv.org/html/2605.12809#S5.SS1)\)\. Concretely, for a model withLLtransformer blocks, the middle\-half range is

ℒmid=\{⌈L/4⌉,…,⌊3​L/4⌋\}\.\\mathcal\{L\}\_\{\\mathrm\{mid\}\}\\;=\\;\\\{\\lceil L/4\\rceil,\\ldots,\\lfloor 3L/4\\rfloor\\\}\.The*red dot*marks the best recovered accuracy \(smallest drop\) over the full sweep\. The*selected layer*is the best layer*restricted*toℒmid\\mathcal\{L\}\_\{\\mathrm\{mid\}\}\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/sae_results/results_medqa_combined.png)\(a\)MedQA\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/sae_results/results_openbookqa_combined.png)\(b\)OpenBookQA\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/sae_results/results_commonsense_qa_combined.png)\(c\)CommonsenseQA\.

Figure A\.1:Accuracy changeΔ\\Deltaversus SAE insertion layer across datasets and SAE configurations\.#### Takeaways\.

Across all three datasets: \(i\) there typically exists a contiguous band of intermediate layers where the post\-insertion accuracy drop is small; \(ii\) the optimal insertion layer can shift by several blocks across model families and datasets; and \(iii\) larger SAEs \(higher latent dimensionHH\) tend to be more forgiving to insertion, while more aggressive sparsity \(smallerkk\) increases layer sensitivity\. These trends motivate the layer\-selection rule in Section[5\.1](https://arxiv.org/html/2605.12809#S5.SS1): we search withinℒmid\\mathcal\{L\}\_\{\\mathrm\{mid\}\}to avoid both input\-adjacent and head\-adjacent regimes, and we report both the selected layer \(restricted toℒmid\\mathcal\{L\}\_\{\\mathrm\{mid\}\}\) and the best layer over the full sweep\.

#### Depth trade\-off: downstream preservation vs\. feature utility\.

To quantify the trade\-off between preserving downstream behavior and obtaining useful/interpretable features, we compare necessity test scores at two insertion depths \(L10 and L13\)\. For Llama\-3\.2\-1B\-Instruct on OpenBookQA, Layer 10 is the selected layer under our middle\-half selection policy, while Layer 13 is the best performance layer; both points are marked in Figure[1\(b\)](https://arxiv.org/html/2605.12809#A1.F1.sf2)\. Table[A\.1](https://arxiv.org/html/2605.12809#A1.T1)shows that the deeper layer \(L13\) consistently yields smaller NLL increase for mostkk, indicating worse task behavior preservation, while also exhibiting substantially lower flip rates, indicating weaker controllable feature effects\. This supports the practical choice of intermediate layers for auditing: deeper insertion can preserve accuracy better, but the resulting features tend to be less interpretable/usable for intervention\-based analysis\.

Table A\.1:Trade\-off across insertion depth: intermediate insertion layer better preserves downstream behavior and yields stronger intervention effects with lowerΔ\\DeltaNLL and higher flip rate \(FR\), suggesting higher feature usefulness for interpretability\.

### A\.2Orthogonality Ablations Across SAE/AE Variants

#### Orthogonality across SAE/AE variants\.

To study how architecture and regularization affect disentanglement, Table[A\.2](https://arxiv.org/html/2605.12809#A1.T2)compares dense AE \(D\) and sparse SAE \(S\) variants across orthogonality metrics and task accuracy \(Acc\.\)\. For the orthogonal\-AE baseline, we add a penalty on the off\-diagonal mass of the feature Gram matrix of the intermediate dense latent\. LetZ^∈ℝN×d\\hat\{Z\}\\in\\mathbb\{R\}^\{N\\times d\}denote minibatch latent activations after centering each feature and normalizing columns to unitℓ2\\ell\_\{2\}norm, and letG=Z^⊤​Z^G=\\hat\{Z\}^\{\\top\}\\hat\{Z\}\. We use

ℒortho=1d​\(d−1\)​∑i≠jGi​j2=1d​\(d−1\)​‖G−Id‖F,off2,\\mathcal\{L\}\_\{\\mathrm\{ortho\}\}=\\frac\{1\}\{d\(d\-1\)\}\\sum\_\{i\\neq j\}G\_\{ij\}^\{2\}=\\frac\{1\}\{d\(d\-1\)\}\\left\\\|G\-I\_\{d\}\\right\\\|\_\{F,\\mathrm\{off\}\}^\{2\},and train withℒ=ℒtask\+λrec​ℒrec\+λortho​ℒortho\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\lambda\_\{\\mathrm\{rec\}\}\\mathcal\{L\}\_\{\\mathrm\{rec\}\}\+\\lambda\_\{\\mathrm\{ortho\}\}\\mathcal\{L\}\_\{\\mathrm\{ortho\}\}\. In the table,OW\\mathrm\{OW\}denotesλortho\\lambda\_\{\\mathrm\{ortho\}\}, andOW=–\\mathrm\{OW\}=\\text\{\-\-\}indicates no orthogonality penalty\.

Overall, sparse SAEs yield the strongest near\-orthogonality \(up to99\.10%99\.10\\%atk=16384/256k\{=\}16384/256\) and lowest off\-diagonal correlation magnitudes, while dense AEs improve substantially as OW increases \(e\.g\., better stable rank and lower mean\-squared correlation atOW=1\.0\\mathrm\{OW\}=1\.0\)\. Across settings, this indicates an orthogonality–accuracy tradeoff: improving feature orthogonality is generally accompanied by some drop in task accuracy \(within roughly70\.870\.8–73\.473\.4in our runs\)\.

Table A\.2:Orthogonality comparison across sparse SAE and dense AE variants\. Lower is better for mean/squared/Frobenius off\-diagonal metrics; higher is better for stable rank and near\-orthogonality\. G\-Abs and G\-Sq stand for the mean absolute/squared value of the gram off\-diagnoal entries\.

### A\.3Qualitative Attribution: Linking Predictions to Training Evidence

We now present case studies illustrating how our method links a test prediction to specific training evidence\. For each case, we report the model prediction, the most influential training examples, and token\-level heatmaps obtained by projecting representation\-level influence back to the input space\.

#### Case study protocol\.

We select test instances that are predicted incorrectly by the pretrained model but become correct after finetuning, so that the attributions reflect task\-specific learning rather than generic priors\. We run this procedure for both Llama\-3\.2\-1B\-Instruct and Qwen2\.5\-1\.5B\-Instruct; for readability, the main text shows one representative example and additional cases \(including Qwen\-based results\) are provided in Appendix[A\.3](https://arxiv.org/html/2605.12809#A1.SS3)\.

For each selected test example, we compute representation\-level influenceIFR\\mathrm\{IFR\}and use it to rank influential latent features and influential training examples\. We then visualize token\-level influence by projecting the influential latent signal onto SAE activations and color tokens by their dominant influential latent, with intensity given by activation×\\timesinfluence \(normalized per sentence\)\. We also interactively support hovering over each latents to see its corresponding tokens\.

#### Case Study B: Openbook Influence retrieval identifies shared physical mechanisms\.

This example illustrates that RepInfLLM retrieves evidence at the level of physical mechanisms rather than exact answer strings\. The test question asks which object can serve as an electrical conductor, with the correct answer “a penny\.” The most influential training sample instead asks what flows when one electrical conductor contacts another electrical conductor, but it activates the same underlying concept: conductors permit the flow of electricity or power\. The highlighted latent features concentrate on “one electrical conductor,” “another electrical conductor,” “flow,” and “power,” showing that the influence signal is localized to the mechanism\-bearing phrase rather than to superficial option overlap\. This suggests that representation\-level influence can recover training evidence that supports the test prediction through shared scientific structure\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test57.png)\(a\)Test input \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test57_train4594.png)\(b\)Most influential training example \(token\-level influence\)\.

Figure A\.2:Case Study B \(Llama\-3\.2\-1B\-Instruct\): Electrical conductor\.
#### Case Study C: MedQA influence highlight Asthma exacerbation and small\-airway obstruction

The test example in Figure[A\.3](https://arxiv.org/html/2605.12809#A1.F3)describes an 8\-year\-old with shortness of breath and dry cough after environmental exposure, diffuse end\-expiratory wheezing, and a decreased inspiratory\-to\-expiratory ratio\. The correct choice is*terminal bronchioles*, consistent with inflammation and narrowing of small airways in an obstructive process\.

The retrieved training example concerns a severe asthma presentation and asks for a classic physiologic finding \(pulsus paradoxus\)\. Although the questions differ, influential latents activate on shared features related to bronchospasm and respiratory distress \(e\.g\., wheeze, reduced airflow, difficulty breathing\)\. This illustrates that representation\-level influence can connect test predictions to training evidence through a shared obstruction motif, without requiring exact surface\-form matching\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test17.png)\(a\)Test input \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test17_train1949.png)\(b\)Most influential training example \(token\-level influence\)\.

Figure A\.3:Case Study C \(Qwen2\.5\-1\.5B\-Instruct\): asthma exacerbation and small\-airway obstruction\.
#### Case Study D: Reasoning augmentation sharpens deep\-sea animal influence\.

For this case study, we did something different with the setup\. We construct a reasoning\-augmented training sample by keeping the original question answer pair fixed and appending a short rationale that makes the relevant semantic relation explicit\. We then recompute representation level influence and compare it with the baseline heatmap to test whether the added reasoning sharpens attribution from surface overlap toward concept level evidence\.

Figure[A\.4](https://arxiv.org/html/2605.12809#A1.F4)shows how adding an explicit reasoning cue changes the retrieved evidence\. The test question asks why frilled sharks and angler fish are classified as deep\-sea animals, requiring the model to connect these species to organisms living far below the ocean surface\. In the baseline heatmap, the retrieved training example is relevant mainly because it contains “angler fish” among the answer choices, but the attribution remains diffuse across the multiple\-choice sequence and does not clearly isolate the required reasoning\. After adding the reasoning chain, the influence becomes sharper: highlighted regions concentrate on the phrase “outside of the ocean” and the animal tokens, especially “angler fish” and “shark\.” This suggests that reasoning augmentation helps the latent features align animal identity with ocean depth context, yielding a clearer link between the retrieved training sample and the test answer\.

![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/test0.png)\(a\)Test input \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/baseline_test0.png)\(b\)Most influential training example \(token\-level influence\)\.
![Refer to caption](https://arxiv.org/html/2605.12809v1/Images/reasoning_test0.png)\(c\)Most influential training example with reasoning augmentation\.

Figure A\.4:Case Study D\. Reasoning augmented heatmap comparison

## Appendix BExperimental Details

This appendix records the settings needed to reproduce the experiments in Section[5](https://arxiv.org/html/2605.12809#S5)\.

### B\.1Models and fine\-tuning

- •Base models\.Qwen2\.5\-1\.5B\-Instruct and Llama\-3\.2\-1B\-Instruct\.
- •Fine\-tuning\.We experiment with both full fine\-tuning and LoRA\. LoRA is faster and more memory\-efficient; in our setting it also yields better task performance, consistent with reduced catastrophic forgetting under constrained compute\.
- •Optimization\.AdamW with learning rate2×10−42\\times 10^\{\-4\}, batch size 32, 4 epochs, maximum sequence length 512\.\.
- •Evaluation\.Accuracy over multiple\-choice options\{A,B,C,D,E\}\\\{A,B,C,D,E\\\}, extracted from the model generation using a deterministic option parser\.

### B\.2SAE training

- •Insertion layers\.Llama: layers 4–12; Qwen: layers 7\-22\. The reported “selected layer” is the best\-performing layer withinℒmid\\mathcal\{L\}\_\{\\mathrm\{mid\}\}\.
- •Objective\.Reconstruction MSE on the instrumented hidden state, optionally trained jointly with the downstream task loss when enabled \(see Table[B\.1](https://arxiv.org/html/2605.12809#A2.T1)\)\.
- •SAE size and sparsity\.Latent dimensionH∈\{16384,2048,512\}H\\in\\\{16384,2048,512\\\}and Top\-kksparsityk∈\{256,128,64\}k\\in\\\{256,128,64\\\}\.
- •Training data and steps\.2 epochs\.

#### Latent size selection\.

The alternative visualization uses the task\-sized SAE with latent dimensionH=2048H=2048and Top\-k=256k=256, whereas some exploratory visualizations use larger SAEs to inspect finer\-grained concepts\. We choose the compact SAE for the 1B\-scale OpenbookQA experiments because the finetuning set contains only∼\\sim5k examples and exact influence computation scales with the number of latent coordinates\. The smaller latent space produces a denser heatmap with densor activation, making full influence sweeps and heatmap rendering readable\.

### B\.3Influence computation and filtering

#### Gradient pre\-filtering\.

We rank training examples by cosine similarity between their gradients and the test gradient:

sim​\(zi,ztest\)\\displaystyle\\mathrm\{sim\}\(z\_\{i\},z\_\{\\text\{test\}\}\)=⟨gi,gtest⟩‖gi‖​‖gtest‖,\\displaystyle=\\frac\{\\langle g\_\{i\},g\_\{\\text\{test\}\}\\rangle\}\{\\\|g\_\{i\}\\\|\\,\\\|g\_\{\\text\{test\}\}\\\|\},\(22\)gi\\displaystyle g\_\{i\}=∇θℓ​\(hθ​\(xi\),yi\),\\displaystyle=\\nabla\_\{\\theta\}\\ell\(h\_\{\\theta\}\(x\_\{i\}\),y\_\{i\}\),gtest\\displaystyle g\_\{\\text\{test\}\}=∇θℓ​\(hθ​\(xtest\),ytest\)\.\\displaystyle=\\nabla\_\{\\theta\}\\ell\(h\_\{\\theta\}\(x\_\{\\text\{test\}\}\),y\_\{\\text\{test\}\}\)\.We retain the top \{1% / 5% / 10%\} candidates for exact influence computation\.

#### iHVP approximation\.

We computeHθ2−1​gtestH\_\{\\theta\_\{2\}\}^\{\-1\}g\_\{\\text\{test\}\}using conjugate gradient \(CG\), with damping1e\-3and iteration budget20\. Unless otherwise stated, the empirical Hessian is formed using a batched loss with batch size 8 to improve curvature stability\.

#### Influence target\.

Unless otherwise stated, influence is computed on the test loss\.

Table B\.1:Key experimental hyperparameters used throughout the appendix\.

## Appendix CScaling to 1B\-Parameter LLMs

Scaling representation influence to 1B\-parameter LLMs is challenging due to: \(i\) memory blow\-ups from naïvely materializing second\-order objects or large JVP tensors; and \(ii\) numerical instability when the empirical Hessian exhibits negative curvature, which can break iterative solvers\.

### C\.1Memory scaling via contraction order

We compute the representation influence for latent dimensionjjas

ℐjr​\(ztrainr,ztest\)\\displaystyle\\mathcal\{I\}\_\{j\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=−gtest⊤​Hθ2−1⏟iHVP term​JVP​\(G,rtrain,ej\)⏟per\-latent JVP,\\displaystyle=\-\\underbrace\{g\_\{\\text\{test\}\}^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}\}\_\{\\text\{iHVP term\}\}\\;\\underbrace\{\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\)\}\_\{\\text\{per\-latent JVP\}\},\(23\)whereHθ2H\_\{\\theta\_\{2\}\}is the Hessian of the training objective restricted to downstream parametersθ2\\theta\_\{2\}andgtestg\_\{\\text\{test\}\}is the test gradient\.

#### Avoiding Hessian materialization\.

We never explicitly constructHθ2H\_\{\\theta\_\{2\}\}\. Instead, we rely on Hessian\-vector products \(HVPs\) computed via automatic differentiation, enabling CG solves without storing the Hessian\.

#### Avoiding Jacobian materialization\.

A naive implementation computes all\{JVP​\(G,rtrain,ej\)\}j=1H\\\{\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\)\\\}\_\{j=1\}^\{H\}in one autograd call by concatenating basis vectors; this leads to catastrophic memory usage\. Profiling shows the per\-feature JVP stage can allocate∼\\sim10–15 GB on GPT\-2, and aggregation can add another∼\\sim10–15 GB, pushing peak memory toward∼\\sim40 GB\. Dense offloading is worse: storing the full Jacobian scales as𝒪​\(H⋅\|θ2\|\)\\mathcal\{O\}\(H\\cdot\|\\theta\_\{2\}\|\)and can reach hundreds of GB at 1B scale\.

We therefore compute influence in a*streamed contraction*manner:

1. 1\.Precompute iHVP once per test input\.Computestest⊤≜gtest⊤​Hθ2−1s\_\{\\text\{test\}\}^\{\\top\}\\triangleq g\_\{\\text\{test\}\}^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}once and reuse it across all latent dimensions\.
2. 2\.Fuse JVP with contraction\.For eachjj, computerj≜JVP​\(G,rtrain,ej\)r\_\{j\}\\triangleq\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\)and immediately contractℐjr=−stest⊤​rj\\mathcal\{I\}\_\{j\}^\{r\}=\-s\_\{\\text\{test\}\}^\{\\top\}r\_\{j\}, then discardrjr\_\{j\}\.

This reduces storage from𝒪​\(H⋅\|θ2\|\)\\mathcal\{O\}\(H\\cdot\|\\theta\_\{2\}\|\)to𝒪​\(\|θ2\|\)\\mathcal\{O\}\(\|\\theta\_\{2\}\|\)and avoids Jacobian storage\. In our Llama\-3\.2\-1B profiling, this reduces the peak from an estimated dense\-materialization regime \(∼\\sim400 GB\) to a practical tens\-of\-GB regime\.

### C\.2Stabilizing iHVP under noisy curvature

We obtainstest⊤s\_\{\\text\{test\}\}^\{\\top\}by solvingHθ2​v=gtestH\_\{\\theta\_\{2\}\}v=g\_\{\\text\{test\}\}using CG, whereH​vHvis computed via autograd HVPs\. Although CG assumes symmetric positive definite curvature, too\-small batches yield noisy empirical Hessians with occasional negative curvature, manifested asp⊤​H​p<0p^\{\\top\}Hp<0and early termination or degenerate solutions\. Empirically, batch size≤4\\leq 4can cause a large fraction of influence entries to collapse to near\-zero due to solver instability\. We therefore defineHθ2H\_\{\\theta\_\{2\}\}using a batched empirical loss with batch size 8 for stable curvature in all 1B runs\.

### C\.3Compute\-time scaling and accelerations

Compute is dominated by the latent loop if we explicitly sweep per\-feature JVPs\. Without further reductions, on Llama\-3\.2\-1B the per\-\(ztrain,ztest\)\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)cost decomposes into roughly∼\\sim1 s \(forward/backward\),∼\\sim5 s \(CG iHVP\), and∼\\sim30–60 s \(full JVP sweep, depending on model and SAE size\), which is infeasible when repeated over many training candidates\.

Table C\.1:Runtime and memory breakdown for 1B\-scale influence computation per\(ztrain,ztest\)\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)pair, based on our profiling\. “JVP sweep” corresponds to streamed contraction \(no Jacobian storage\)\.We apply two practical reductions:

1. 1\.Task\-sized SAE\.Since MedQA fine\-tuning has∼\\sim9k training samples, we train effective SAEs with latent sizeH=512H=512and Top\-k=64k=64, reducing the JVP sweep to∼\\sim30 s and total to∼\\sim40 s per candidate training example\.
2. 2\.Gradient\-similarity pre\-filtering\.We compute exact influence only on the top1%1\\%–10%10\\%training candidates ranked by gradient cosine similarity\.

With these reductions, end\-to\-end influence computation is∼\\sim1 hour per test sample in our setting\.

### C\.4Further acceleration via the gradient\-derivative formulation \(Sec\. 3\.4\)

Beyond streamed JVP contraction, we exploit the gradient\-derivative reformulation from Sec\. 3\.4, which eliminates the explicit per\-feature JVP sweep\. Recall that representation influence can be written as

ℐr​\(ztrainr,ztest\)=−stest⊤​G​\(rtrain\),\\displaystyle\\mathcal\{I\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-s\_\{\\text\{test\}\}^\{\\top\}G\(r\_\{\\text\{train\}\}\),\(24\)wherestest⊤≜gtest⊤​Hθ2−1s\_\{\\text\{test\}\}^\{\\top\}\\triangleq g\_\{\\text\{test\}\}^\{\\top\}H\_\{\\theta\_\{2\}\}^\{\-1\}andG​\(rtrain\)=∇θ2ℓ​\(hθ2​\(rtrain\),ytrain\)G\(r\_\{\\text\{train\}\}\)=\\nabla\_\{\\theta\_\{2\}\}\\ell\(h\_\{\\theta\_\{2\}\}\(r\_\{\\text\{train\}\}\),y\_\{\\text\{train\}\}\)\. Instead of iterating overjjand computingJVP​\(G,rtrain,ej\)\\mathrm\{JVP\}\(G,r\_\{\\text\{train\}\},e\_\{j\}\), we directly differentiate the scalar projection with respect tortrainr\_\{\\text\{train\}\}:

∇rtrain\(stest⊤​G​\(rtrain\)\)\.\\displaystyle\\nabla\_\{r\_\{\\text\{train\}\}\}\\big\(s\_\{\\text\{test\}\}^\{\\top\}G\(r\_\{\\text\{train\}\}\)\\big\)\.\(25\)This single reverse\-mode pass produces the full latent\-level influence vector\{ℐjr\}j=1H\\\{\\mathcal\{I\}\_\{j\}^\{r\}\\\}\_\{j=1\}^\{H\}, replacing an𝒪​\(H\)\\mathcal\{O\}\(H\)JVP loop with one sensitivity backpropagation\.

Table C\.2:Runtime breakdown \(seconds\) per\(ztrain,ztest\)\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)pair using the gradient\-derivative formulation on Qwen\-2\.5\-1\.5B\.Table C\.3:Runtime breakdown \(seconds\) per\(ztrain,ztest\)\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)pair using the gradient\-derivative formulation on Llama\-3\.2\-1B\.#### Overall compute comparison\.

Table[C\.4](https://arxiv.org/html/2605.12809#A3.T4)compares the streamed\-JVP pipeline with the gradient\-derivative formulation\. The latter removes the∼\\sim30–60 s JVP sweep \(Table[C\.1](https://arxiv.org/html/2605.12809#A3.T1)\) and reduces total per\-pair runtime to≈2\\approx 2–44seconds, corresponding to a∼10×\\sim 10\\times–20×20\\timespractical speedup\.

Table C\.4:End\-to\-end per\-\(ztrain,ztest\)\(z\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)runtime comparison \(seconds\)\.Combined with batched iHVP solves and gradient\-similarity pre\-filtering, this reduction makes representation\-level influence computation practical for 1B–1\.5B parameter LLMs under commodity multi\-GPU constraints\.

## Appendix DDerivations

### D\.1Exact Path\-Integral Decomposition

LetG:ℝdl→ℝ\|θ2\|G:\\mathbb\{R\}^\{d\_\{l\}\}\\rightarrow\\mathbb\{R\}^\{\|\\theta\_\{2\}\|\}be continuously differentiable\. Fixr0,rtrain∈ℝdlr\_\{0\},r\_\{\\text\{train\}\}\\in\\mathbb\{R\}^\{d\_\{l\}\}and define the straight\-line path

r​\(α\)=r0\+α​\(rtrain−r0\),α∈\[0,1\]\.r\(\\alpha\)=r\_\{0\}\+\\alpha\(r\_\{\\text\{train\}\}\-r\_\{0\}\),\\qquad\\alpha\\in\[0,1\]\.\(26\)
#### Path integral identity\.

By the chain rule,

dd​α​G​\(r​\(α\)\)=JG​\(r​\(α\)\)​d​r​\(α\)d​α=JG​\(r​\(α\)\)​\(rtrain−r0\),\\frac\{d\}\{d\\alpha\}G\(r\(\\alpha\)\)=J\_\{G\}\(r\(\\alpha\)\)\\,\\frac\{dr\(\\alpha\)\}\{d\\alpha\}=J\_\{G\}\(r\(\\alpha\)\)\(r\_\{\\text\{train\}\}\-r\_\{0\}\),\(27\)whereJG​\(r\)J\_\{G\}\(r\)denotes the Jacobian ofGG\. Applying the fundamental theorem of calculus,

G​\(rtrain\)−G​\(r0\)\\displaystyle G\(r\_\{\\text\{train\}\}\)\-G\(r\_\{0\}\)=∫01dd​α​G​\(r​\(α\)\)\\displaystyle=\\int\_\{0\}^\{1\}\\frac\{d\}\{d\\alpha\}G\(r\(\\alpha\)\)\(28\)=∫01JG​\(r​\(α\)\)​\(rtrain−r0\)​𝑑α\.\\displaystyle=\\int\_\{0\}^\{1\}J\_\{G\}\(r\(\\alpha\)\)\(r\_\{\\text\{train\}\}\-r\_\{0\}\)\\,d\\alpha\.

#### Coordinate decomposition\.

Using the canonical basis expansion

rtrain−r0=∑j=1dl\(rtrain\(j\)−r0\(j\)\)​ej,r\_\{\\text\{train\}\}\-r\_\{0\}=\\sum\_\{j=1\}^\{d\_\{l\}\}\(r\_\{\\text\{train\}\}^\{\(j\)\}\-r\_\{0\}^\{\(j\)\}\)e\_\{j\},\(29\)and linearity of integration,

G​\(rtrain\)\\displaystyle G\(r\_\{\\text\{train\}\}\)=G​\(r0\)\+∑j=1dl\(rtrain\(j\)−r0\(j\)\)​\(∫01JG​\(r​\(α\)\)​ej​𝑑α\)\.\\displaystyle=G\(r\_\{0\}\)\+\\sum\_\{j=1\}^\{d\_\{l\}\}\(r\_\{\\text\{train\}\}^\{\(j\)\}\-r\_\{0\}^\{\(j\)\}\)\\left\(\\int\_\{0\}^\{1\}J\_\{G\}\(r\(\\alpha\)\)e\_\{j\}\\,d\\alpha\\right\)\.\(30\)
Equations Eq\. \([28](https://arxiv.org/html/2605.12809#A4.E28)\)–Eq\. \([30](https://arxiv.org/html/2605.12809#A4.E30)\) hold exactly under the sole assumption thatGGis continuously differentiable\.

### D\.2Derivative Swapping with activation

While the Jacobian\-vector product \(JVP\) formulation in Eq\. \([18](https://arxiv.org/html/2605.12809#S4.E18)\) provides a rigorous attribution mechanism, computing it naively is computationally prohibitive\. Evaluating the JVP for each latent dimensionj∈\{1,…,dl\}j\\in\\\{1,\\dots,d\_\{l\}\\\}requiresdld\_\{l\}separate forward\-mode passes\. For a sparse autoencoder with thousands of features, this𝒪​\(dl\)\\mathcal\{O\}\(d\_\{l\}\)complexity leads to severe computational bottlenecks and memory allocation limits\.

We can achieve a massive optimization by exploiting the structure of the computation graph and the symmetry of mixed partial derivatives \(Clairaut’s Theorem\)\. We transition from evaluatingdld\_\{l\}directional derivatives to performing a single reverse\-mode gradient pass\.

The Independence of Upstream Activations\.Recall that the model is split at layerll, such that the latent representationrtrain=hθ1​\(xtrain\)r\_\{\\text\{train\}\}=h\_\{\\theta\_\{1\}\}\(x\_\{\\text\{train\}\}\)is produced entirely by upstream parametersθ1\\theta\_\{1\}, while we differentiate with respect to the downstream parametersθ2=\{θ:θ\>l\}\\theta\_\{2\}=\\\{\\theta:\\theta\_\{\>l\}\\\}\. Becausertrainr\_\{\\text\{train\}\}serves as an input to the downstream computation and does not depend onθ2\\theta\_\{2\}, it is treated as a constant with respect toθ2\\theta\_\{2\}\. Therefore, the parameter gradient of the activation is strictly zero:

∂rtrain\(j\)∂θ2=𝟎\.\\frac\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}=\\mathbf\{0\}\.\(31\)
Equivalence of Inside and Outside Weighting\.LetΔ\(j\)=∂ℓ∂rtrain\(j\)\\Delta^\{\(j\)\}=\\frac\{\\partial\\ell\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}denote the sensitivity of the training loss to thejj\-th latent feature\. In our operationalized code, we define a feature’s loss contribution ass\(j\)=rtrain\(j\)​Δ\(j\)s^\{\(j\)\}=r\_\{\\text\{train\}\}^\{\(j\)\}\\Delta^\{\(j\)\}\. Taking the gradient of this contribution with respect to the downstream parametersθ2\\theta\_\{2\}, we apply the product rule:

∂s\(j\)∂θ2=∂∂θ2​\(rtrain\(j\)​Δ\(j\)\)=∂rtrain\(j\)∂θ2⏟=𝟎​Δ\(j\)\+rtrain\(j\)​∂Δ\(j\)∂θ2\.\\frac\{\\partial s^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}\\;=\\;\\frac\{\\partial\}\{\\partial\\theta\_\{2\}\}\\left\(r\_\{\\text\{train\}\}^\{\(j\)\}\\Delta^\{\(j\)\}\\right\)\\;=\\;\\underbrace\{\\frac\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}\}\_\{=\\mathbf\{0\}\}\\Delta^\{\(j\)\}\+r\_\{\\text\{train\}\}^\{\(j\)\}\\frac\{\\partial\\Delta^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}\.\(32\)Because the first term vanishes due to the computation graph cut Eq\. \([31](https://arxiv.org/html/2605.12809#A4.E31)\), we are left with:

∂s\(j\)∂θ2=rtrain\(j\)​∂∂θ2​\(∂ℓ∂rtrain\(j\)\)=rtrain\(j\)​∂G​\(rtrain\)∂rtrain\(j\),\\frac\{\\partial s^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}\\;=\\;r\_\{\\text\{train\}\}^\{\(j\)\}\\frac\{\\partial\}\{\\partial\\theta\_\{2\}\}\\left\(\\frac\{\\partial\\ell\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\\right\)\\;=\\;r\_\{\\text\{train\}\}^\{\(j\)\}\\frac\{\\partial G\(r\_\{\\text\{train\}\}\)\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\},\(33\)whereG​\(rtrain\)=∇θ2ℓG\(r\_\{\\text\{train\}\}\)=\\nabla\_\{\\theta\_\{2\}\}\\ell\. This proves that multiplying the activation “inside” the gradient tracker is mathematically identical to multiplying by the activation “outside” the mixed\-partial term, perfectly aligning the empirical implementation with the theoretical Taylor decomposition from Definition[4\.2](https://arxiv.org/html/2605.12809#S4.Thmtheorem2)\.

Order Swapping for Constant\-Time Evaluation\.We now apply the influence contraction with the inverse\-HVP test signalv=Hθ2−1​gtestv=H\_\{\\theta\_\{2\}\}^\{\-1\}g\_\{\\text\{test\}\}\. The influence for neuronjjbecomes:

ℐjr​\(ztrainr,ztest\)=−\(∂s\(j\)∂θ2\)⊤​v=−rtrain\(j\)​\(∂∂θ2​∂ℓ∂rtrain\(j\)\)⊤​v\.\\mathcal\{I\}\_\{j\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\left\(\\frac\{\\partial s^\{\(j\)\}\}\{\\partial\\theta\_\{2\}\}\\right\)^\{\\top\}v\\\\ =\-\\,r\_\{\\text\{train\}\}^\{\(j\)\}\\left\(\\frac\{\\partial\}\{\\partial\\theta\_\{2\}\}\\frac\{\\partial\\ell\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\\right\)^\{\\top\}v\.\(34\)
By the symmetry of mixed partial derivatives \(assuming standard regularity conditions on the loss surface\), we can swap the order of differentiation:

\(∂∂θ2​∂ℓ∂rtrain\(j\)\)⊤​v=∂∂rtrain\(j\)​\(\(∂ℓ∂θ2\)⊤​v\)=∂∂rtrain\(j\)​\(G​\(rtrain\)⊤​v\)\.\\left\(\\frac\{\\partial\}\{\\partial\\theta\_\{2\}\}\\frac\{\\partial\\ell\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\\right\)^\{\\top\}v\\;=\\;\\frac\{\\partial\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\\left\(\\left\(\\frac\{\\partial\\ell\}\{\\partial\\theta\_\{2\}\}\\right\)^\{\\top\}v\\right\)\\\\ \\;=\\;\\frac\{\\partial\}\{\\partial r\_\{\\text\{train\}\}^\{\(j\)\}\}\\Big\(G\(r\_\{\\text\{train\}\}\)^\{\\top\}v\\Big\)\.\(35\)
Define the scalar projectionP=G​\(rtrain\)⊤​vP=G\(r\_\{\\text\{train\}\}\)^\{\\top\}v\. This scalar represents the alignment between the training parameter gradients and the test signal\. Eq\. \([35](https://arxiv.org/html/2605.12809#A4.E35)\) reveals that the sensitivity for neuronjjis exactly the partial derivative ofPPwith respect tortrain\(j\)r\_\{\\text\{train\}\}^\{\(j\)\}\.

Consequently, the entire vector of sensitivities for alldld\_\{l\}neurons can be computed simultaneously by taking a single gradient of the scalarPPwith respect to the latent representationrtrainr\_\{\\text\{train\}\}:

Sensitivity Vector=∇rtrainP=∇rtrain\(G​\(rtrain\)⊤​v\)\.\\text\{Sensitivity Vector\}=\\nabla\_\{r\_\{\\text\{train\}\}\}P=\\nabla\_\{r\_\{\\text\{train\}\}\}\\Big\(G\(r\_\{\\text\{train\}\}\)^\{\\top\}v\\Big\)\.\(36\)
Combining Eq\. \([34](https://arxiv.org/html/2605.12809#A4.E34)\) and Eq\. \([36](https://arxiv.org/html/2605.12809#A4.E36)\), the final influence vector for all latent features is obtained via an element\-wise product \(Hadamard product, denoted by⊙\\odot\) with the realized activations:

ℐ→r​\(ztrainr,ztest\)=−\(∇rtrainP\)⊙rtrain\.\\vec\{\\mathcal\{I\}\}^\{r\}\(z^\{r\}\_\{\\text\{train\}\},z\_\{\\text\{test\}\}\)=\-\\Big\(\\nabla\_\{r\_\{\\text\{train\}\}\}P\\Big\)\\odot r\_\{\\text\{train\}\}\.\(37\)
This formulation requires only*two*backward passes total: one to construct the computation graph forG​\(rtrain\)G\(r\_\{\\text\{train\}\}\), and a second to compute the gradient of the scalar projectionPPwith respect tortrainr\_\{\\text\{train\}\}\. The time complexity with respect to the feature dimension drops from𝒪​\(dl\)\\mathcal\{O\}\(d\_\{l\}\)to𝒪​\(1\)\\mathcal\{O\}\(1\)\. In practice, this JVP approach enables the simultaneous computation of influences across all latent features and batch samples, allowing our method to gracefully scale to 1B\-parameter models and SAEs with tens of thousands of features without materializing explosive Jacobians\.