Agentic interpretability is becoming a research direction of its own.

Instead of one-shot labeling, AI agents can: form hypotheses design experiments inspect activations and circuits refine explanations from feedback

Our work, SAGE, pioneered this paradigm for SAE feature autointerpretation. This direction is now expanding with Agentic-iModels for interpretable model design and HYVE for autonomous circuit explanation.

Agentic-iModels asks: can agents design models that are easier for agents to interpret?

Its autoresearch loop evolves tabular regressors for accuracy and LLM simulatability, improving downstream data-science agents by up to 73%.

HYVE brings agentic reasoning to circuit explanation.

Given a localized circuit, it observes components, forms hypotheses, tests them with causal interventions, and produces component- and task-level explanations across 84 circuits.

The field is moving from one-shot labeling toward autonomous, hypothesis-driven interpretability research.

SAGE (p1): https://arxiv.org/abs/2511.20820 Agentic-iModels (p2): https://arxiv.org/abs/2605.03808 HYVE (P3, p4): https://arxiv.org/abs/2606.24026

#AI #LLM #Interpretability #MechanisticInterpretability #AIResearch #Agents #SparseAutoencoders #SAE #AutoInterp

An Agentic Explainer Framework for Interpreting SAE Features in Language Models

Source: https://arxiv.org/html/2511.20820 Jiaojiao Han1Wujiang Xu2Mingyu Jin2Mengnan Du3† 1New Jersey Institute of Technology2Rutgers University 3The Chinese University of Hong Kong, Shenzhen [email protected]@cuhk.edu.cn †Corresponding author

Abstract

Large language models (LLMs) have achieved remarkable progress, yet their internal mechanisms remain largely opaque, posing a significant challenge to their safe and reliable deployment. Sparse autoencoders (SAEs) have emerged as a promising tool for decomposing LLM representations into more interpretable features, but explaining the features captured by SAEs remains a challenging task. In this work, we proposeSAGE(SAEAGenticExplainer), an agent-based framework that recasts feature interpretation from a passive, single-pass generation task into an active, explanation-driven process. SAGE implements a rigorous methodology by systematically formulating multiple explanations for each feature, designing targeted experiments to test them, and iteratively refining explanations based on empirical activation feedback. Experiments on features from SAEs of diverse language models demonstrate that SAGE produces explanations with significantly higher generative and predictive accuracy compared to state-of-the-art baselines. The code is available athttps://github.com/jiujiubuhejiu/SAGE.

SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models

Jiaojiao Han1Wujiang Xu2Mingyu Jin2Mengnan Du3†1New Jersey Institute of Technology2Rutgers University3The Chinese University of Hong Kong, [email protected]@cuhk.edu.cn†Corresponding author

1Introduction

Large language models (LLMs) have achieved remarkable progress across diverse domains, including natural language understanding, generation, and reasoning. However, despite their impressive capabilities, LLMs remain largely opaque systems, often regarded as black boxes whose internal mechanisms are poorly understoodZhaoet al.(2024). To address this opacity, the research community has increasingly focused on decoding the information encoded in LLM representations, seeking to understand how these models process and store knowledge. Among various interpretability approaches, sparse autoencoders (SAEs) have attracted growing attention due to their ability to decompose dense neural activations into sparse, potentially interpretable featuresShuet al.(2025). Recent work has demonstrated that SAEs can identify meaningful feature dimensions in transformer representations, with applications ranging from circuit discovery to activation steeringFerrandoet al.(2025); Heet al.(2025).

Despite this progress, interpreting SAE features remains a significant challenge. As SAEs are trained using unsupervised learning objectives, the semantic meaning of their learned features must be inferred post-hoc through analysis of their activation patterns. Current approaches, exemplified by NeuronpediaLin (2023), rely on automated interpretation pipelines that generate natural language explanations for each SAE feature using large language models such as GPT-4 and Claude 4.5. While these methods have produced preliminary results, two fundamental problems persist. First, the generated explanations lack consistency and rigor. When different LLMs are used to explain the same feature, they often produce divergent explanations, undermining confidence in the interpretations. Second, although SAEs are explicitly designed to decompose polysemous LLM representations into monosemantic features, where each feature captures a single, coherent concept. In practice, many SAE features still exhibit polysemantic behavior, activating in response to multiple distinct semantic or structural patterns. Existing methods like Neuronpedia provide only a single explanation per feature, failing to account for this multi-faceted activation behavior and potentially missing important aspects of feature functionality.

To address these challenges, we proposeSAGE(SAEAGenticExplainer), an agent-based framework that transforms feature interpretation from passive observation into active, explanation-driven experimentation. Rather than relying on single-pass interpretations from off-the-shelf LLMs, SAGE implements a rigorous scientific methodology that systematically formulates multiple explanations about each feature’s behavior, designs targeted experiments to test these explanations, and iteratively refines its understanding based on empirical evidence. Furthermore, by maintaining multiple parallel explanations throughout the interpretation process, SAGE naturally captures polysemantic features, producing comprehensive multi-faceted explanations when appropriate. The major contributions of this work can be summarized as:

• •We propose SAGE, a novel agent-based framework that reformulates feature interpretation as an active, explanation-driven scientific process rather than a passive, single-pass generation task.
• •SAGE formulates, tests, and iteratively refines multiple parallel explanations for each feature based on empirical activation feedback.
• •We perform experiments on features from diverse LLMs, demonstrating that SAGE produces more accurate, consistent, and actionable feature interpretations compared to existing methods.

2Problem Formulation

Figure 1:Overview of the SAGE framework. The process begins when an explainer LLM generates an initial explanations (HiH_{i}) from high-activation text derived from the target LLM and SAE. A designer LLM then creates test text (TiT_{i}) to validate this explanation, initiating a multi-turn explanation refinement loop. Within this loop, an analyzer LLM observes the activations produced whenTiT_{i}is fed into the target LLM. A reviewer LLM then evaluates this feedback and decides whether to accept, reject, refute, or refine the current explanations. This iterative process continues until an explanations is accepted, culminating in the final explanation synthesis (H∗H^{*}).In this section we first provide the technical background Sparse Autoencoders (SAEs), and then formulate the task of SAE feature explanation.

2.1Sparse Autoencoders

SAEsBrickenet al.(2023b); Cunninghamet al.(2023); Templetonet al.(2024)are designed to address the opacity of large models by decomposing dense neural activationsx∈ℝdmodelx\in\mathbb{R}^{d_{\text{model}}}into sparse, potentially interpretable featuresf∈ℝdsaef\in\mathbb{R}^{d_{\text{sae}}}. This is achieved by projecting the input into a much higher-dimensional feature space, wheredsae≫dmodeld_{\text{sae}}\gg d_{\text{model}}.The architecture consists of an encoder that computes the sparse featuresff, and a decoder that uses these sparse features to reconstruct the original activation,x^\hat{x}:

f=ReLU​(We​(x−bpre)+be),x^=Wd​f+bdec.f=\text{ReLU}(W_{e}(x-b_{\text{pre}})+b_{e}),\,\,\hat{x}=W_{d}f+b_{\text{dec}}.(1)Here,WeW_{e}andWdW_{d}are the encoder and decoder weight matrices, whilebpreb_{\text{pre}},beb_{e}, andbdecb_{\text{dec}}are bias terms. The model is trained to balance two competing objectives: reconstruction fidelity and feature sparsity, achieved with the loss functionℒ\mathcal{L}:

ℒ=‖x−x^‖22⏟Reconstruction Loss+λ​‖f‖1⏟Sparsity Penalty\mathcal{L}=\underbrace{\|x-\hat{x}\|^{2}_{2}}_{\text{Reconstruction Loss}}+\underbrace{\lambda\|f\|_{1}}_{\text{Sparsity Penalty}}(2)The first term ensures the reconstructed vectorx^\hat{x}is close to the original inputxx. The second term, anL1L_{1}penalty on the feature activationsff, encourages most features to be zero. The hyperparameterλ\lambdacontrols the trade-off between these two objectives.

2.2SAE Feature Explanation

Since SAEs are trained on unsupervised objectives, the semantic meaning of their learned features, specific directions in the activation space, must be inferred post-hoc. An SAE model projects activations into a high-dimensional feature spacef∈ℝds​a​ef\in\mathbb{R}^{d_{sae}}, so a trained SAE withds​a​e=16,000d_{sae}=16,000, for example, contains 16K individual features. The ultimate goal of our work is to provide a natural language explanationEjE_{j}for each of thej∈{1,…,ds​a​e}j\in\{1,...,d_{sae}\}features. We formally define the task of SAE feature explanation for a single featurefjf_{j}as finding a natural language explanation,EjE_{j}, that accurately describes the set of semantic or structural input patterns that cause that feature to activate.

As noted in the introduction, current single-pass generation methods often produce explanations that lack this empirical validation and fail to account for polysemantic features that respond to multiple distinct patterns. To address these limitations, we reformulate the task: instead of seeking a single, staticEjE_{j}, our agent-based framework discovers an empirically validated explanationEEthrough an iterative process of testing and refining multiple explanations{H1,…,Hn}\{H_{1},...,H_{n}\}based on multi-turn interactions with the SAE model.

3The Proposed SAGE Framework

In this section, we presentSAGE(SAE Agentic Explainer), a novel agent-based framework designed to address the challenge of SAE feature explanation1. Instead of relying on passive, single-pass generation, SAGE transforms this task into an active, iterative scientific process (see Figure1).

The process begins when an Explainer LLM generates an initial set of explanations,{Hi}\{H_{i}\}, based on high-activation text from the target LLM and SAE. A Designer LLM then creates targeted test text,TiT_{i}, to validate these explanations, which initiates the multi-turn explanation refinement loop. Within this loop, an Analyzer LLM observes the empirical feature activations produced whenTiT_{i}is processed by the target LLM. A Reviewer LLM evaluates this activation feedback and decides the next step: to accept, reject, refute, or refine the current explanations. This iterative, feedback-driven process continues until an explanation is accepted, culminating in the final explanation synthesis,EE.

3.1Initial Explanation Generation

The interpretation process of our SAGE framework for a single target SAE featurefjf_{j}, a learned direction in the model’s activation space, begins with standard feature analysis. We first extract the top-k text segments from a corpus that maximally activate this featurefjf_{j}. These high-activation examples serve as the empirical foundation for explanation generation. The explainer LLM then analyzes these examples using promptPi​n​i​tP_{init}(see Appendix) to formulate an initial set ofnnexplanations,{H1,H2,…,Hn}\{H_{1},H_{2},...,H_{n}\}, about the semantic concept encoded byfjf_{j}. Unlike single-pass methods that commit to a single interpretation, SAGE maintains multiple parallel explanations to capture potentially complex, context-dependent, or polysemantic activation patterns. Each explanationsHiH_{i}represents a distinct, testable theory about what interpretable concept or pattern triggers the feature’s activation.

3.2Multi-Turn Explanation Refinement

The second stage of SAGE is a multi-turn execution loop, where each explanation undergoes iterative refinement through empirical testing. For each active explanationHiH_{i}at turntt, the system executes a structured testing cycle.

First, the explainer LLM generates test textTiT_{i}designed to validate explanationHiH_{i}using promptPtestP_{\text{test}}. This generated text represents a concrete prediction: ifHiH_{i}correctly captures the concept encoded by the SAE feature, thenTiT_{i}should strongly activateFjF_{j}. The text generation process is guided by both the explanation and accumulated evidence from previous iterations, enabling increasingly sophisticated probes of feature boundaries. Next, we obtain empirical feedback by passingTiT_{i}through the target LLM and measuring the SAE feature activation:ai=SAEj​(TargetLLM​(Ti)).a_{i}=\text{SAE}_{j}(\text{TargetLLM}(T_{i})).The activation magnitudeaia_{i}provides direct evidence about explanation validity. Based on activation analysis, the analyzer LLM determines the next state for each explanation using system promptPanalyzeP_{\text{analyze}}. Our framework supports four state transitions that capture different experimental outcomes:

• •Accept: When test textTiT_{i}produces strong activations matching predictions, explanationHiH_{i}is accepted as a valid interpretation.
• •Reject: If repeated tests fail to produce meaningful activations or consistently contradict predictions, explanationHiH_{i}will be rejected.
• •Refine: Partial activation matches suggest the explanation captures some aspect of feature behavior but requires modification. The system generates refined explanationHi′H^{\prime}_{i}and updated test textTi′T^{\prime}_{i}for the next iteration.
• •Refute: When activation patterns directly contradict explanation predictions, the system maintainsHiH_{i}but generates alternative test textTi′T^{\prime}_{i}to explore why the expected behavior didn’t occur.

The state transition logic is formalized as:

(Hi(t+1),Ti(t+1),statusi)=Transition​(Hi(t),Ti(t),ai(t)),(H_{i}^{(t+1)},T_{i}^{(t+1)},\text{status}_{i})=\text{Transition}(H_{i}^{(t)},T_{i}^{(t)},a_{i}^{(t)}),(3)where the transition function is implemented through structured prompting of the analyzer LLM with activation analysis results. The multi-turn execution continues until all explanations reach terminal states (accepted or rejected) or maximum turns are met. Through successive iterations, initial broad and rough explanations evolve into precise descriptions of SAE feature behavior.

This iterative process enables several key capabilities. Complex conditional features emerge through refinement what begins as “technical terms” might evolve into “technical discussions in formal contexts” through testing. Polysemantic features are naturally discovered when multiple non-overlapping explanations are accepted. Edge cases and boundary conditions surface through the refute-retry cycle. Each iteration adds to an accumulating evidence base:

ℰ(t)=ℰ(t−1)∪{(Hi(t),Ti(t),ai(t))}i=1n.\mathcal{E}^{(t)}=\mathcal{E}^{(t-1)}\cup\{(H_{i}^{(t)},T_{i}^{(t)},a_{i}^{(t)})\}_{i=1}^{n}.(4)This evidence history informs subsequent explanation refinement and test generation, creating a feedback loop that drives increasingly sophisticated understanding.

3.3Final Explanation Synthesis

After the iterative process converges, SAGE synthesizes final interpretations from accepted explanations. The reviewer LLM reviews all accepted explanationsℋaccepted\mathcal{H}_{\text{accepted}}and their supporting evidence using promptPsynthesizeP_{\text{synthesize}}to generate comprehensive feature explanationsEE.

For monosemantic features, this typically yields a single refined explanation with extensive empirical validation. For polysemantic features, the synthesis identifies distinct behavioral facets and their activation conditions. The final output includes both natural language explanations and concrete examples that reliably trigger feature activation.

4Experiments

In this section, we conduct experiments to evaluate the proposed SAGE framework.

4.1Experimental Setup

Table 1:This table outlines the experimental setup, detailing the diverse set of open-source LLMs, corresponding SAE models, and the specific transformer layers selected for feature evaluation.LLMsSAE ModelLayersQwen3-4btranscoder-hp3, 7, 11, 23Gemma-2-2bgemmascope-res-16k3, 7, 11, 23GPT-OSS-20bresid-post-aa3, 7, 11, 23

Table 2:Comparison of explanation quality between SAGE and Neuronpedia baseline using generative accuracy and predictive accuracy metrics.MethodGPT-OSS-20bQwen3-4bGemma-2-2bLayerGen. Acc.↑\uparrowPred. Acc.↑\uparrowLayerGen. Acc.↑\uparrowPred. Acc.↑\uparrowLayerGen. Acc.↑\uparrowPred. Acc.↑\uparrowNeuronpedia30.260.6230.220.6830.750.68SAGE30.590.8030.540.7230.970.83Neuronpedia70.570.6070.250.6470.300.65SAGE70.770.7170.540.6670.800.70Neuronpedia110.300.67110.120.65110.360.70SAGE110.520.71110.230.65110.560.74Neuronpedia230.120.52230.090.65230.280.64SAGE230.670.68230.280.67230.560.67

Implementation Details.

We evaluate SAE features from a diverse set of open-source language models using pre-trained SAEs111https://www.neuronpedia.org/. The specific configurations of models, SAEs, and their corresponding layers employed in this study are as given in Table1. We evaluate SAGE across these transformer architectures, focusing on layers 3, 7, 11, and 23 to capture feature behaviors spanning from early semantic processing to high-level abstraction. For each target layer, we randomly sample 10 features to ensure representative evaluation while maintaining computational feasibility. We employ GPT-5222https://platform.openai.com/docs/models/gpt-5as the core language model for all agents within the SAGE framework, including the Explainer, Designer, Analyzer, and Reviewer components. A critical component of our evaluation methodology, and for our baseline comparison against Neuronpedia, our top-kkactivating exemplars are taken from the “dashboard” of Neuronpedia. For the parameters introduced in Section3.1, we set the number of top-k text segmentskkto 10 and the number of initial explanationsnnto 4.

Baseline Comparison.

We conduct systematic comparisons against Neuronpedia, the current state-of-the-art automated interpretation system. To ensure fair comparison with Neuronpedia, we maintain strict experimental controls: (1)Consistent Exemplar Data: All top-kkexemplars are obtained through Neuronpedia’s standardized activation sampling interface; (2)Uniform Explanation Models: Both systems utilize the same LLM (GPT-5) for generating natural language explanations; (3)Standardized Activation Measurement: Ground-truth activation values are retrieved using Neuronpedia’s evaluation APIs; (4)Identical Test Sets: Feature selection and test sentence sampling procedures are identical across methods.

Evaluation Metrics.

We evaluate the quality and utility of the generated feature explanations using two complementary metrics. The first, Generative Accuracy, assesses the causal validity of an explanation by measuring whether it can be used to generate novel text that reliably activates the target feature. The second, Predictive Accuracy, assesses the descriptive power of an explanation by measuring its ability to predict feature activations on held-out data. Full details on the implementation of these metrics are provided in AppendixA.

Table 3:Comparison of explanations of SAGE with Neuronpedia.Blue: first semantics,Red: second semantics.LLMExample featureDescription by NeuronpediaDescription by SAGE (Ours)layer-type/idGemma-2-2b11-gemmascope- res-16k/ 5125mentions of multithreading synchronization and thread-safety mechanisms, especially lock-related constructs and events.Code synchronization/locking constructs (Python lock/RLock/Event idioms; C++ mutex; Java-like synchronized)“.Natural-language uses of ’lock/unlock/Sherlock’ remain at baseline.Qwen3-4b23-transcoder -hp/ 24625English negative contractions using “n’t,” often in auxiliary or modal verb constructions.English “can’t” contraction suffix detector (“’t”/“’t” after “ can“; localized, orthography/punctuation/newline robust; moderate activations with occasional spillover)GPT-OSS-20b3-resid-post- aa/ 121075mentions of terrestrial, land-based contexts such as habitats, ecosystems, animals, or planets.Terrestrial lexeme/morpheme detector: exact ’ terrestrial’ token (strong) and ’…restrial’ fragments (strong-to-moderate), with stem-only fragments moderate.Habitat list co-activation: weak activation on ’ aquatic’ when co-listed with strongly activated ’ terrestrial’.Gemma-2-9b-it20-gemmascope -res-131k/ 1mentions of Java exceptions in code/logs, especially invalid-argument error types and related exception handling.Java IllegalArgumentException lexical detector (surface-form ’ IllegalArgument’ with weak ’Exception’ co-activation; modest sensitivity to ’ Illegal’ prefix).

4.2Explanation Results Comparisons

Table2compares SAGE against the Neuronpedia baseline across three language models using generative and predictive accuracy metrics. SAGE demonstrates substantial generative accuracy improvements across all configurations, with gains ranging from 29% to 458%. The most pronounced improvements occur at deeper layers where Neuronpedia deteriorates significantly. At layer 23, SAGE achieves 0.67 for GPT-OSS-20B versus Neuronpedia’s 0.12, representing a 458% improvement. Predictive accuracy shows more modest but consistent gains, with SAGE scoring 0.65-0.83 compared to Neuronpedia’s 0.52-0.70.

This performance divergence reveals a key distinction between the approaches. While both methods adequately describe existing activation patterns, SAGE’s explanations possess significantly greater causal validity for generating novel feature-activating content. Unlike generative accuracy, predictive performance remains stable across network depths for both methods. The generalizability across model architectures confirms that iterative experimental validation benefits extend across diverse model families and scales.

4.3Qualitative Evaluation

In this section, we provide several case explanations in Table3to qualitatively demonstrate the precision and faithfulness of SAGE’s explanations.

The baseline’s tendency to over-generalize is evident in feature24625from Qwen3-4b, described as detecting “English negative contractions using ’n’t’”. Our empirical validation, however, reveals a far more specific function. SAGE’s process (e.g., Test 2: “won’t” and Test 10: “don’t”) explicitly refutes this broad hypothesis, showing zero activation. Instead, SAGE correctly identifies the feature’s true, narrow scope: an “English ’can’t’ contraction suffix detector,” defining a sharp, accurate boundary.

This rigorous validation is equally critical for polysemous features. For feature5125from Gemma-2-2b, the baseline provides a vague description of “multithreading synchronization” without defining its limits. SAGE’s iterative validation, in contrast, not only confirms activation on multi-language code constructs (Python RLock, C++ mutex, Java synchronized) but also actively tests and refutes activations on natural-language uses of the word “lock” (e.g., Test 3: “He turned the lock on the door.”). SAGE’s final description, “Code synchronization/locking constructs… Natural-language uses… remain at baseline,” provides a far more complete and useful explanation.

This pattern of superior precision is consistent across other examples. For instance, SAGE describes feature121075(GPT-OSS-20b) as a “Terrestrial lexeme/morpheme detector” sensitive to exact tokens, rather than the baseline’s general “terrestrial… contexts”. Similarly, for feature1(Gemma-2-9b-it), SAGE specifies a “lexical detector” for the exact string “IllegalArgument”. In all cases, SAGE provides more specific, empirically-grounded, and faithful explanations of the feature’s true behavior.

4.4Ablation Studies

We ablated the number of initial explanationskkgenerated by the explainer LLM to balance interpretation quality with computational efficiency. Figure2shows results fork∈{5,10,15}k\in\{5,10,15\}. Withk=5k=5, SAGE achieves the lowest token consumption (26,500 tokens per turn) but insufficient explanation diversity, yielding only 0.648 prediction accuracy. The limited hypothesis space prevents comprehensive feature understanding, particularly for polysemantic features requiring multiple interpretations. Atk=15k=15, prediction accuracy peaks at 0.667 but incurs a 19% higher computational cost (31,500 tokens per turn) compared tok=10k=10. The performance gain diminishes as additional explanations often represent redundant hypotheses. The optimal configuration emerges atk=10k=10, achieving 0.664 prediction accuracy statistically equivalent tok=15k=15(difference of 0.003) while maintaining computational efficiency at 26,500 tokens per turn. This provides sufficient explanation diversity to capture complex feature semantics without diminishing returns. We adoptk=10k=10as the default configuration, balancing interpretive thoroughness with computational efficiency.

Figure 2:Ablation study on initial explanation countkk. Prediction accuracy saturates atk=10k=10while token consumption continues increasing, demonstrating optimal efficiency atk=10k=10.

5Conclusions

In this work, we addressed the critical challenge of consistently and comprehensively interpreting features from SAEs in language models. To tackle this, we proposed SAGE, a novel agent-based framework that reformulates feature interpretation as an active, explanation-driven scientific process rather than a passive, single-pass generation task. SAGE employs a multi-turn execution loop where an explainer LLM systematically formulates, tests, and refines multiple explanations for each feature by generating targeted text and analyzing empirical activation feedback. Our comprehensive evaluations demonstrate that SAGE yields explanations with superior generative and predictive accuracy compared to existing state-of-the-art methods. Additionally, by maintaining and validating multiple parallel explanations, SAGE naturally discovers and provides multi-faceted explanations for polysemantic features, addressing a fundamental limitation of current interpretation approaches.

Limitations

Our study has several limitations, primarily stemming from resource constraints. For each LLM and its corresponding SAE, our evaluation was conducted on only four selected layers rather than all available layers. Furthermore, within each of these layers, we randomly sampled 10 features for experimental evaluation instead of assessing the complete set of features.

References

• Refusal in language models is mediated by a single direction.InNeurIPS,External Links:LinkCited by:Appendix B.
• [2]L. Bereska and S. GavvesMechanistic interpretability for ai safety-a review.Transactions on Machine Learning Research.Cited by:Appendix B.
• S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders (2023)Language models can explain neurons in language models.Note:https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.htmlAccessed: YYYY-MM-DDCited by:Appendix B.
• T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah (2023a)Towards monosemanticity: decomposing language models with dictionary learning.Transformer Circuits Thread.Note:https://transformer-circuits.pub/2023/monosemantic-features/index.htmlCited by:Appendix B.
• T. Bricken, A. Templeton, J. Batson,et al.(2023b)Towards monosemanticity: decomposing language models with dictionary learning.Transformer Circuits Thread.Cited by:§2.1.
• H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey (2023)Sparse autoencoders find highly interpretable features in language models.arXiv preprint arXiv:2309.08600.Cited by:2nd item,§2.1.
• M. Faruqui, Y. Tsvetkov, D. Yogatama, C. Dyer, and N. A. Smith (2015)Sparse overcomplete word vector representations.InACL,pp. 1491–1500.Cited by:Appendix B.
• J. Ferrando, O. B. Obeso, S. Rajamanoharan, and N. Nanda (2025)Do i know this entity? knowledge awareness and hallucinations in language models.InICLR,External Links:LinkCited by:§1.
• L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu (2025)Scaling and evaluating sparse autoencoders.InICLR,External Links:LinkCited by:Appendix B,Appendix B.
• Y. Gur-Arieh, R. Mayan, C. Agassy, A. Geiger, and M. Geva (2025)Enhancing automated interpretability with output-centric feature descriptions.InProceedings of ACL,Vienna, Austria.Cited by:Appendix B.
• Z. He, W. Shu, X. Ge, L. Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu,et al.(2024)Llama scope: extracting millions of features from llama-3.1-8b with sparse autoencoders.arXiv preprint arXiv:2410.20526.Cited by:Appendix B.
• Z. He, M. Jin, B. Shen, A. Payani, Y. Zhang, and M. Du (2025)SAE-SSV: supervised steering in sparse representation spaces for reliable control of language models.InEMNLP,Suzhou, China.External Links:LinkCited by:Appendix B,§1.
• R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey (2024)Sparse autoencoders find highly interpretable features in language models.InICLR,Cited by:Appendix B,Appendix B.
• [14]C. Kissane, R. Krzyzanowski, J. I. Bloom, A. Conmy, and N. NandaInterpreting attention layer outputs with sparse autoencoders.InICML 2024 Workshop on Mechanistic Interpretability,Cited by:Appendix B.
• T. Lieberum, S. Rajamanoharan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramár, A. Dragan, R. Shah, and N. Nanda (2024)Gemma scope: open sparse autoencoders everywhere all at once on gemma 2.InACL BlackboxNLP Workshop,pp. 278–300.Cited by:Appendix B.
• J. Lin (2023)Neuronpedia: interactive reference and tooling for analyzing neural networks.Note:Software available from neuronpedia.orgExternal Links:LinkCited by:Appendix B,§1.
• S. Prasai, M. Du, Y. Zhang, and F. Yang (2026)KnowThyself: an agentic assistant for llm interpretability.AAAI Demo Track.Cited by:Appendix B.
• S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, and N. Nanda (2024a)Improving sparse decomposition of language model activations with gated sparse autoencoders.InNeurIPS,pp. 775–818.Cited by:Appendix B.
• S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda (2024b)Jumping ahead: improving reconstruction fidelity with jumprelu sparse autoencoders.arXiv preprint arXiv:2407.14435.Cited by:Appendix B.
• T. R. Shaham, S. Schwettmann, F. Wang, A. Rajaram, E. Hernandez, J. Andreas, and A. Torralba (2024)A multimodal automated interpretability agent.InForty-first International Conference on Machine Learning,Cited by:Appendix B.
• L. Sharkey, D. Braun, and B. Millidge (2022)Taking features out of superposition with sparse autoencoders.External Links:LinkCited by:Appendix B.
• W. Shi, S. Li, T. Liang, M. Wan, G. Ma, X. Wang, and X. He (2025)Route sparse autoencoder to interpret large language models.InEMNLP,Suzhou, China,pp. 6812–6826.External Links:Link,Document,ISBN 979-8-89176-332-6Cited by:Appendix B.
• D. Shu, X. Wu, H. Zhao, D. Rai, Z. Yao, N. Liu, and M. Du (2025)A survey on sparse autoencoders: interpreting the internal mechanisms of large language models.InEMNLP Findings,Suzhou, China.External Links:LinkCited by:Appendix B,§1.
• A. Templeton, T. Conerly, J. Marcus,et al.(2024)Scaling monosemanticity: extracting interpretable features from claude 3 sonnet.Transformer Circuits Thread.Cited by:§2.1.
• M. Wang, X. Chen, Y. Wang, Z. He, J. Xu, T. Liang, Q. Liu, Y. Yao, W. Wang, R. Ma,et al.(2025a)Two experts are all you need for steering thinking: reinforcing cognitive effort in moe reasoning models without additional training.NeurIPS.Cited by:Appendix B.
• M. Wang, Z. Xu, S. Mao, S. Deng, Z. Tu, H. Chen, and N. Zhang (2025b)Beyond prompt engineering: robust behavior control in LLMs via steering target atoms.InACL,Vienna, Austria,pp. 23381–23399.External Links:Link,DocumentCited by:Appendix B.
• L. Wu, M. Wang, Z. Xu, T. Cao, N. Oo, B. Hooi, and S. Deng (2025a)Automating steering for safe multimodal large language models.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp. 792–814.Cited by:Appendix B.
• X. Wu, W. Yu, X. Zhai, and N. Liu (2025b)Self-regularization with sparse autoencoders for controllable llm-based classification.InSIGKDD,pp. 3250–3260.Cited by:Appendix B.
• X. Wu, J. Yuan, W. Yao, X. Zhai, and N. Liu (2025c)Interpreting and steering llms with mutual information-based explanations on sparse autoencoders.arXiv preprint arXiv:2502.15576.Cited by:Appendix B.
• W. J. Yeo, N. Prakash, C. Neo, R. Satapathy, R. K. Lee, and E. Cambria (2025)Understanding refusal in language models with sparse autoencoders.InEMNLP Findings,C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.),Suzhou, China.Cited by:Appendix B.
• H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, S. Wang, D. Yin, and M. Du (2024)Explainability for large language models: a survey.ACM Transactions on Intelligent Systems and Technology15(2),pp. 1–38.Cited by:§1.

Appendix AMore Details of the Evaluation Metrics

We employ two complementary evaluation metrics to assess the quality and utility of feature explanations generated by our SAGE framework.

• •*Generative Accuracy.*This metric assesses the causal validity of an explanation: can it be used togeneratenovel text that reliably triggers the feature? We instruct an LLM to generateNNsentences based solely on the feature’s explanation. We define a success thresholdTactT_{\text{act}}as 50% of the maximum activation observed in the initial top-10 exemplars. The generative accuracy is the success rate: the fraction of generated sentences whose maximal token activationFj​(G​(Hi))F_{j}(G(H_{i}))exceedsTactT_{\text{act}}.
• •*Predictive Accuracy.*This metric assesses the descriptive power of an explanation: can it be used topredictfeature activations on held-out data? We use a held-out set of exemplarsDheld-outD_{\text{held-out}}, distinct from theDj,kD_{j,k}used for explanation generation, sampled from high, medium, and low activation groups. Following past workCunninghamet al.(2023), we employ a simulatorσ\sigma, which is an LLM prompted with the feature explanationEjE_{j}. For each tokenttin a held-out example,σ\sigmapredicts the discretized activation level. Rather than single-point prediction, we compute the expected activation value using the log-probabilitiesσ\sigmaassigns to the output tokens ’0’ through ’10’. The predictive accuracy is the mean Pearson correlation coefficient (ρ\rho) between the predicted activation values and the true, normalized per-token activations acrossDheld-outD_{\text{held-out}}.

Appendix BRelated Work

Sparse autoencoders (SAEs).SAEs were introduced as an unsupervised dictionary-learning approach to address superpositionFaruquiet al.(2015)in LLMShuet al.(2025); Hubenet al.(2024). By mapping model activations into a higher-dimensional sparse space, SAEs isolate a small number of latent features per input, yielding monosemantic features that correspond to single interpretable concepts rather than polysemantic neuronsBrickenet al.(2023a). A number of SAE variants and tools have been developed to improve their efficacy and accessibility. The vanilla SAE typically uses anL1L_{1}sparsity penalty on the latent vector to encourage most neurons to stay inactiveSharkeyet al.(2022)and recent variants like the Top-KKSAE instead enforce a fixed numberKKof active features per inputGaoet al.(2025). Other improvements include gated or JumpReLU SAEs that modify the activation function to better balance feature detection and strength estimationRajamanoharanet al.(2024b,a). Some pre-trained repositories, such as Gemma ScopeLieberumet al.(2024)and Llama ScopeHeet al.(2024), enable broader research.

SAEs Application.SAEs have been used to interpret model representations and understand model capabilitiesWuet al.(2025b,c). Beyond static analysis, researchers have begun leveraging SAE-discovered features to steer model behavior. Such activation steering via SAE features has been used to alter attributes like sentiment, truthfulness, or style without fine-tuning the entire modelHeet al.(2025); Shiet al.(2025); Wanget al.(2025b,a). SAEs have also been applied in the context of model safety and alignment. One study showed that features learned by an SAE from a language model can serve as effective probes for classifying toxic content across languagesBereska and Gavves. By identifying which sparse features correspond to a model’s refusals or safety responses, one can understand and even adjust the model’s safety mechanisms. Intervening on these features has been shown to influence the model’s tendency to refuse or comply with certain promptsArditiet al.(2024); Yeoet al.(2025); Wuet al.(2025a). Overall, SAEs offer a transparent, feature-level handle on model behaviors that is valuable for safety research.

SAEs Feature Explanation.Inspired by the automated interpretability pipeline that uses GPT-4 to explain GPT-2 neurons from their activating examples (MaxAct)Billset al.(2023), a framework that has since become the standard for large-scale interpretation of neurons and SAE-learned features in both language and vision modelsLin (2023); Hubenet al.(2024); Gaoet al.(2025). Neuronpedia combines an activation-based methodKissaneet al.that highlights the tokens most strongly triggering a feature with a logit-projection methodKissaneet al.that infers the feature’s semantic direction by measuring its positive and negative influence on output logits. Recent work proposes an “output-centric” automated feature interpretation that interprets model features not only by considering which inputs activate them, but also by examining the impact of their activation on the model output to generate more accurate and causal interpretationsGur-Ariehet al.(2025).

Agents for Explainability.Recent work has explored using agentic frameworks for explainability. For instance, MAIAShahamet al.(2024)employs a vision-language model equipped with a set of tools to automate the interpretation of computer vision models. MAIA iteratively designs experiments, composes tools for tasks like input synthesis and exemplar generation, and formulates explanations to explain model behaviors, such as identifying feature selectivity or failure modes. Similarly, KnowThyselfPrasaiet al.(2026)provides an agentic assistant specifically for LLM interpretability. It unifies various interpretability tools into a single chat-based interface, allowing users to ask natural language questions. In contrast to these applications, our work proposes an agent framework specifically designed to interpret the features learned by SAEs.

Appendix CExamples of SAE Explanations

Qwen3-4b 3-transcoder-hp 148551

Lexical ’amnesty’ (lowercase common-noun event; not ’Amnesty International’ or derived forms)

Specific -mstr/-msta lexemes: ’Darmstadt’, ’hamstring’ (singular), and ’Armstrong’ (surname); excludes unrelated ’-stadt’ cities, plurals, or orthographic near-misses (e.g., ’Ingolstadt’, ’Amsterdam’, ’hamster’)

Gemma-2-2b 11-gemmascope-res-16k 148551

sudden/suddenly” lexical-morpheme detector (incl. “all of a/the sudden”) with split-morpheme robustness and punctuation spillover

Spillover in “Suddenly, there was …” raising comma and ’was’ when preceded by “Suddenly

Gemma-2-9b-it 20-gemmascope-res-131k 2

Expository-definition scaffolding (endowed-with PPs and predicate coordination in technical/encyclopedic style)

Inert on copular/list coordinations (negative control))

Qwen3-4b 7-transcoder-hp 158076

“Recreat-” Morpheme and “-ational” Suffix Morphological Detector (Activates on words like ’Recreational’ and ’Recreativo’ via strong peaks on ’creat’ and ’ational’ subtokens)

GPT-OSS-20b 3-resid-post-aa 72038

Chinese lexical “ 的 一 “ detector (strong) with weak secondary sensitivity to the character “ 一 “ in non-Chinese CJK contexts

Gemma-2-2b 11-gemmascope-res-16k 13574

m-final subword detector (case-/domain-agnostic) with vowel+m hierarchy (UM≥\geqOM > um >> AM/IM) and occasional internal-’em’ spillover due to tokenization

GPT-OSS-20b 7-resid-post-aa 74421

Apartheid lexical/subword detector with compositional co-occurrence boosts (peak on ’heid’ or ’ apartheid’; moderate ’Apart’/’apart’; boosted policy/state/government/system/regime; contextual ’South/Africa’; negatives low)

Appendix DAgent Prompts

PinitTask: We have executed the maximum activation test on the corpus. Your mission is to systematically analyze and interpret specific SAE features. After analyzing the exemplar data, you MUST explicitly state hypotheses.Real Exemplar Data from Corpus Analysis:{exemplars_summary}Required Output Format:OBSERVATION: - Pattern 1: [specific pattern description based on real data] - Pattern 2: [another pattern description based on real data] - Common elements: [list of common features from real exemplars] [HYPOTHESIS LIST]: Hypothesis_1: [Specific, testable claim based on analysis] Hypothesis_2: [Alternative explanation for the patterns] Hypothesis_3: [Edge case consideration - what might NOT activate this feature] Hypothesis_4: [Additional hypothesis covering different aspects] Analysis & Hypothesis Formation Guidelines:•Analyze the REAL activation values and key tokens from the exemplars•Look for linguistic patterns (suffixes, prefixes, word types)•Identify semantic patterns (topics, domains, concepts)•Note structural patterns (syntax, formatting)•Be specific: “English -tion suffixes” not “English words”•Focus on COMMON patterns across multiple exemplars•Consider which specific tokens have the highest activation values•MANDATORY: After observations, form specific, testable hypotheses about what the feature detects•Be precise: “This feature detects Python import statements” not “This feature detects programming”•Each hypothesis must be testable withmodel.run•Include at least one negative control hypothesisFormat Requirements:•Always start each hypothesis with “Hypothesis_X: [your specific hypothesis]”•Base hypotheses directly on observations, not assumptions•Include positive and negative cases•Cover different aspects of the feature (linguistic, semantic, structural)Rules:•Observe activation patterns, activation values and identify high-activating examples•Do NOT issue[TOOL]commands•Base analysis on the REAL exemplar data provided above•Be scientific and evidence-based•Focus on what the feature actually detects based on the activation patterns

psynthesizeTask: Review all hypotheses and their testing results. Determine if additional testing is needed before drawing final conclusions.All Hypotheses Information:{hypotheses_summary}Required Output Format:REVIEW SUMMARY: [Brief summary of all hypotheses and their current status] ASSESSMENT: [Are all hypotheses adequately tested?] [Are there any gaps in evidence?] [Are there any contradictions between hypotheses?] DECISION: Need more testing: [YES / NO] [If YES: Specify which hypotheses need additional testing and suggested test sentences] [If NO: Explain why current evidence is sufficient for final conclusion] IMPORTANT - If “Need more testing: YES”:When suggesting additional tests, format them EXACTLY like this so they can be automatically executed:- H1: Test negative control: "She left for Paris." - H1: Test another negative: "I bought it for $5." - H2: Test verbal use: "Batteries last for hours." Format Requirements for Suggested Tests:1.Start each line with “- H[number]:“2.Put the test sentence in double quotes: “test sentence here“3.Keep sentences simple (3-10 words)4.One test per lineReview Guidelines:•Check if each hypothesis has sufficient test evidence (at least 2-3 tests)•Verify that CONFIRMED/REFUTED hypotheses have strong supporting evidence•Identify any hypotheses that may need refinement or additional testing•Consider if there are any high-activation corpus tokens that haven’t been tested•Ensure no critical patterns are missing from the analysis•Limit: Suggest a maximum of 2-3 tests per hypothesis (focus on the most critical gaps)Rules:•Be thorough: review ALL hypotheses, not just the confirmed ones•Be honest: if evidence is insufficient, say so•Be specific: if more testing is needed, use the format above for suggested tests•Do NOT issue[TOOL]commands•Base assessment on REAL test data provided above•Safety: This is review iteration {self.sm.review_count if hasattr(self.sm, ’review_count’) else 1}/3. After 3 iterations, proceed to final conclusion regardless.