Generative Interpretability via Scalable Neuro-Symbolic Models

# Generative Interpretability via Scalable Neuro-Symbolic Models
Source: [https://arxiv.org/html/2609.13529](https://arxiv.org/html/2609.13529)
Conference:ACM AI Summit 2026; August 31\-September 02, 2026; Atlanta, GA, USAACM AI Summit 2026 \(AI Summit ’26\), August 31\-September 02, 2026, Atlanta, GA, USADOI:[10\.1145/3806096\.3844850](https://doi.org/10.1145/3806096.3844850)ISBN:979\-8\-4007\-2638\-5/2026/08CCS:Computing methodologies Knowledge representation and reasoningCCS:Computing methodologies Natural language processingCCS:Computing methodologies Artificial intelligenceXiaocong YangAffiliation:AI Interpretability @ Illinois, University of Illinois Urbana\-Champaign,Urbana,IL,USAemail:[xy51@illinois\.edu](mailto:[email protected])

© cc

###### Abstract\.

As the use of Large Language Models moves from chatbots into agentic systems, where outputs become actions with irreversible consequences on reality, the existing paradigm on AI Interpretability research, post\-hoc interpretability, is structurally inadequate for safe and trustworthy model deployment: it explains behavior after the fact but cannot audit or intervene in an inference computation before it commits to an output\. We therefore argue for a shift toward*generative interpretability*, an architectural property under which a model’s inference pass natively exposes semantically meaningful checkpoints that are human\-understandable and amenable to causal intervention\. We show the merits of generative interpretability as comparison to other interpretability research paradigms, and propose Neuro\-Symbolic Models as a concrete instantiation\.

###### Keywords:

AI Interpretability, Neuro\-Symbolic Models, Large Language Models, AI Safety, Generative Interpretability

††cc\-license:by## 1\.Introduction

Recent years witnessed the rapid development of connectionist AI models, especially Large Language Models \(LLMs\) with unprecedented real\-world impacts\. Although powerful, they are generally believed to lack interpretability on their internal working mechanisms and are therefore described as "black\-box" models\([Lipton, 2017](https://arxiv.org/html/2609.13529#bib.bib44);[Rudin, 2019](https://arxiv.org/html/2609.13529#bib.bib24)\)\. A direct consequence is that it is usually infeasible to understand, audit and intervene in undesirable model behaviors such as hallucination, sycophancy and harmful content generation in a scalable and systematic way\. The negative consequences of such undesired model behaviors are dramatically amplified as the trend of deploying LLMs in agentic systems, such as healthcare systems that help with diagnoses\([Wang et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib45)\)and autonomous driving systems that control the actions of a running vehicle\([Wu et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib46)\)\. In those systems, agents interact with the real\-world environments and therefore can have irreversible consequences that are much more severe than in a chatbot: as reported by[Maslej et al\. \(2025\)](https://arxiv.org/html/2609.13529#bib.bib48), documented AI safety incidents surged 56\.4% from 2023 to 2024, with autonomous driving systems involved in fatal crashes\([Responsible AI Collaborative, 2024](https://arxiv.org/html/2609.13529#bib.bib50)\)and agentic AI systems causing financial losses\([Bengio and others, 2026](https://arxiv.org/html/2609.13529#bib.bib49)\)\. Those issues have risen to the urgency at the national security level: the DARPA\-NSF\-CAISI AI Forge program\([, 2026](https://arxiv.org/html/2609.13529#bib.bib47)\)identifies AI Interpretability as one of the three priorities in AI research, highlighting the importance of interpretable AI in agentic settings for auditable autonomy and operational runtime control\.

In this work, we argue that a paradigm transition is necessary for interpretability research to meet this requirement\. Most current work on explaining LLMs falls into the category of post\-hoc interpretability\([Madsen et al\., 2022](https://arxiv.org/html/2609.13529#bib.bib1)\), which relies on reverse\-engineering tools external to the LLM\. Such tools are confined to observation: their interpretations are confounded by the tools’ own approximation errors, and the tools themselves are usually discarded at deployment rather than used to make the deployed model safer\. As a result, post\-hoc interpretability cannot reliably translate into safer behaviors from deployed LLMs\. Instead, we propose the concept ofgenerative interpretabilityas an alternative\. The key claim from this concept is that,interpretability should be a property inherent to the model computations that allows effective audit and intervention at deployment,directly targeting the research thrusts and challenges proposed in the DARPA\-NSF\-CAISI AI Forge report\.

## 2\.Post\-hoc Interpretability and Its Limitations

Post\-hoc interpretability refers to the property where the computations and behaviors of an already\-trained model can be attributed and grounded into human understandable terms\([Madsen et al\., 2022](https://arxiv.org/html/2609.13529#bib.bib1)\)\. Based on the explanandum, they can be summarized into two categories:anatomical interpretabilityandphysiological interpretability\.

### 2\.1\.Anatomical Interpretability

Anatomical interpretability studieslocal structuresin a model that function stably and consistentlyover different inputs, usually accompanied by modularity and functional specialization\. For example, the anatomy of multi\-head self\-attention layers \(MHAs\) and feed\-forward networks \(FFNs\) in a Transformer\([Vaswani et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib40)\)have been thoroughly studied before the emergence of LLMs\. For example, the MHAs in a BERT model\([Devlin et al\., 2018](https://arxiv.org/html/2609.13529#bib.bib3)\)are discovered to specialize in processing certain semantic or syntactic information in input\([Clark et al\., 2019](https://arxiv.org/html/2609.13529#bib.bib2)\)\. The FFNs in a Transformer are usually interpreted as factual knowledge banks\([Geva et al\., 2021](https://arxiv.org/html/2609.13529#bib.bib4);[Dai et al\., 2022](https://arxiv.org/html/2609.13529#bib.bib5)\), which can be edited to influence model outputs on certain facts\([Meng et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib6)\)\.

### 2\.2\.Physiological Interpretability

Physiological interpretability recovers theend\-to\-end information flowof aspecific inference passexecuted by the model\. The theoretical basis can be traced back to causal mediation analysis\([Pearl, 2013](https://arxiv.org/html/2609.13529#bib.bib7)\), where the computation flow on a given input sample is characterized as a Directed Acyclic Graph \(DAG\) and interventions are applied to intermediate nodes to study the causal effect on model output\. Early efforts in this category include probing\([Belinkov, 2021](https://arxiv.org/html/2609.13529#bib.bib8)\)and naive saliency tracing\([Sundararajan et al\., 2017](https://arxiv.org/html/2609.13529#bib.bib10);[Li et al\., 2019](https://arxiv.org/html/2609.13529#bib.bib9);[Sundararajan and Najmi, 2020](https://arxiv.org/html/2609.13529#bib.bib11)\)\. With the development of LLMs, the Transformer Circuit Theory\([Elhage et al\., 2021](https://arxiv.org/html/2609.13529#bib.bib12)\)and its follow\-up work\([Elhage et al\., 2022](https://arxiv.org/html/2609.13529#bib.bib13);[Bricken et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib14);[Lindsey et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib15)\)are proposed and become one of the most cutting\-edge techniques in post\-hoc interpretability\. Instead of performing naive, metric\-based tracing, Transformer Circuit Theory targets reducing an inference pass by LLMs to combinations of simplified operations on semantically meaningful directions calledfeaturesin model activation spaces\. These work systematically study and answer three key questions centered around features: \(1\) how are features distributed in activation spaces \(quasi\-orthogonally\); \(2\) how to identify features and their meanings \(via dictionary learning\); and \(3\) how does a model use these meaningful features to generate its output \(via circuit tracing with attribution graphs\)\.

### 2\.3\.The Shared Fundamental Limitation

The work from post\-hoc interpretability gives us insights to better understand LLMs\. However, all of them have a fundamental limitation:they can explain what a model does, but cannot change what it does, in a systematic and scalable way\.On the one hand, post\-hoc interpretability tools are far from as effective in inference\-time intervention as they are in representation explaining\. For example,[Basu et al\. \(2026\)](https://arxiv.org/html/2609.13529#bib.bib51)documented a large gap between a model’s internal representations and its outputs that steering interventions based on different post\-hoc interpretability tools failed to reliably close\. On the other hand, post\-hoc interpretability tools suffer from severe scalability and/or efficiency problems that block them from being deployed outside of laboratory environment\. Those powerful interpretability tools in Transformer Circuit Theory, such as Sparse Autoencoders and Cross\-layer Transcoders, cannot be efficiently integrated into LLMs to be deployed, due to the extremely large feature dimensions used to encourage sparse and clean feature decomposition\. Other lightweight tools, such as model editing\([Wang et al\., 2024](https://arxiv.org/html/2609.13529#bib.bib16)\)and activation patching\([Zhang and Nanda, 2024](https://arxiv.org/html/2609.13529#bib.bib17)\), are limited to modifying a specific type of model behavior with the dependency on labeled data related to that behavior, and hence far from scalable and systematic\.

## 3\.Generative Interpretability

The empirical failure of intervention with post\-hoc interpretability tools directly motivates the necessity of a new interpretability paradigm that is tailored for effective audit\. In this section, we articulate generative interpretability as the appropriate requirement on models to procure inference\-time audit and controllable intervention\.

### 3\.1\.Definition

We define generative interpretability as follows\.

###### Definition 3\.1 \(Generative Interpretability\)\.

An architectural property of a model under which its inference pass111We use the term “inference pass” to describe a single computation trajectory that maps input to a single meaningful output unit\. For example, in auto\-regressive LLMs, it is the generation of a next token\. We use “inference path" for the static computational pathway defined by the model architecture, along which an inference pass is executed for a given input\.exposes intermediate checkpoints that are \(i\) semantically meaningful to humans natively and \(ii\) causally intervenable by construction\.

Human understandability\.Interpretability in AI is typically framed in terms of human social attribution and cognition\([Biran and Cotton, 2017](https://arxiv.org/html/2609.13529#bib.bib19);[Miller, 2017](https://arxiv.org/html/2609.13529#bib.bib18)\)\. By*semantically meaningful to humans*, we require that a checkpoint to be mapped to a human\-readable concept\. Audit and intervention ultimately reflect human decisions and values, and they can only identify undesired model behaviors when the intermediate states are expressed in terms they understand\. This excludes interpretations that are only transparent in terms of mathematical functionalities\. For instance, an optimization view yields a mathematically interpretable Transformer\([Yu et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib29)\), whose layers have well\-defined roles such as token set compression and sparsification, yet it does not reveal any semantic meanings the model assigns during processing inputs\. By*native*, we require the acquisition of meanings not to rely on external tools to recover\. For example, the framework by[Mueller et al\. \(2025\)](https://arxiv.org/html/2609.13529#bib.bib52)separates the choice of checkpoints from the discovery of their meanings; whether the checkpoints are Attention heads, neurons or activation subspaces, their meanings rely on post\-hoc interpretability tools to search and assign, and therefore, it cannot avoid the key defects of post\-hoc interpretability\.

Causal intervention\.An interpretability framework is considered faithful when reflecting the real*reasoning process*of a model processing inputs\([Jacovi and Goldberg, 2020](https://arxiv.org/html/2609.13529#bib.bib20);[Geiger et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib22)\)\. In particular, the framework should be able to be tested with*“do\-intervention”*\([Pearl, 2009](https://arxiv.org/html/2609.13529#bib.bib21)\)and the counterfactual output should align with the interpretation consistently, as argued in\([Nussbaum\-Hoffer et al\., 2026](https://arxiv.org/html/2609.13529#bib.bib39)\)\. This is important to achieve predictable model behaviors when imposing intervention, although this is usually not the case with post\-hoc interpretability tools\. The requirement on causal intervention directly implies that the checkpoints need to be variables the inference pass functionally depends on, where altering their values necessarily propagates through the downstream computation and so do\-interventions have well\-defined counterfactuals by construction, instead of contingent and approximate causal effects as with post\-hoc interpretability tools\.

### 3\.2\.Two Axes of Interpretability

The definition above commits to two requirements but leaves open how generative interpretability relates to existing interpretability paradigms\. We organize this comparison along two axes capturing distinct dimensions of commitment:*when*interpretation is available relative to inference \(the temporal axis\), and*where*along the inference path human\-readable structures reside \(the spatial axis\)\.

The temporal axis\.We divide interpretability approaches into three categories based on when human\-understandable states are available relative to model inference\. Most traditional machine learning models, such as linear regression models, decision trees, and rule\-based systems, are interpretablebefore inference\. Their structure or static parameters are human\-understandable in themselves and fully determine their behavior on every input, so the inference pass generates no extra interpretable information beyond a naive application of already\-readable patterns\.[Rudin \(2019\)](https://arxiv.org/html/2609.13529#bib.bib24)advocates this property, and their follow\-up work shows the superior performance by a fully interpretable, domain\-specific model\([Hu et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib38)\)\. Those models are usually described aswhite\-box models\. The aforementioned post\-hoc interpretability methods recover interpretable states onlyafter inference: the human\-understandable states are not natively exposed by the model but reconstructed by external tools from completed inference passes\. The third category, where interpretable states emerge during inference passes, has drawn less attention\. A representative work is Concept Bottleneck Models \(CBMs\)\([Koh et al\., 2020](https://arxiv.org/html/2609.13529#bib.bib25)\)and its extension to LLMs\([Sun et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib26)\), where the computations from neural networks are grounded in semantic concepts from a pre\-defined, human\-understandable concept pool as the final component in inference paths\. However, these models are usually limited to simple similarity matching in a specific domain\.

The spatial axis\.Interpretability approaches can also be categorized by the spatial position of interpretable states in an inference path\. As a baseline, the vanilla LLM and interpretations based on it \(such as Chain\-of\-thought\([Wei et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib27)\)\) offer human\-readable representations in form of text tokens onlyat input/output boundaries\. All post\-hoc interpretability methods rely on external tools to construct interpretations that lieexternallyto the model\. The CBM family provide an interpretableinterior checkpoint layerthat allows simple concept matching, and most white\-box models are interpretablethroughout the modelby definition\.

The position of generative interpretability\.Along both axes, generative interpretability occupies an intermediate position\. Temporally, it residesduring inference\. A checkpoint’s schema \(concept space\) is usually pre\-defined, while the values filling it are produced by an opaque sub\-computation \(such as a neural network\) as the input is processed to that checkpoint\. The interpretable states, therefore, can neither be derived from the static model alone, nor relying on the full inference pass to be completed\. Spatially, it requiresmultiple interior checkpointsfor the purpose of effective causal intervention\. Boundary readability exposes only inputs and outputs, and a single checkpoint captures only one cross\-section of a multi\-step computation within a model, which is insufficient to attribute undesired output \(this will be discussed more in Section[3\.3](https://arxiv.org/html/2609.13529#S3.SS3)\)\.

### 3\.3\.Generative Interpretability as the Minimal Sufficient Commitment for Auditable LLMs

We argue that, for LLMs, generative interpretability is the minimal architectural commitment sufficient to make audit and intervention over the inference pass that is scalable in development and computationally tractable at deployment\. The commitment is prescriptive: a property of the models we should build on existing LLMs, not as they are\.

A partial order by auditable content\.The axes induce partial orders on commitment strength\. Spatially, a throughout\-interpretable model is legible at any interior point, subsuming multiple\-checkpoint readability, which subsumes a single checkpoint and then the input\-output boundary\. Temporally, a model legible before inference remains legible during it when given an input\. Post\-hoc interpretability lies outside this chain, as its readable structure is reconstructed externally rather than residing in the model, and therefore implies no commitment on the model architectures\.

Why not weaker\.Auditing content generation before it goes to output rules out after\-inference interpretability immediately, whose readable state is recovered only once the output—and any unsafe content—already exists\. During inference is thus the weakest temporal position that still permits pre\-generation audit\. Spatially, two weaker commitments fall short\. The input\-output boundary \(vanilla LLMs, CoT\) exposes no intermediate checkpoints on the inference path\. Note that any audit on CoT’s intermediate steps is based on tokens that are already generated via inference passes, so it can only reject an undesired generation rather than attribute to the computation that produces it and correct accordingly\. Moreover, the intermediate steps in a CoT are not faithfully reflecting the internal computation\([Arcuschin et al\., 2026](https://arxiv.org/html/2609.13529#bib.bib33);[Scalena et al\., 2026](https://arxiv.org/html/2609.13529#bib.bib34);[Turpin et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib35)\), making intervention on them unreliable\([Valmeekam et al\., 2026](https://arxiv.org/html/2609.13529#bib.bib36)\)\. A single interior checkpoint as in CBM audits one cross\-section, and in the setting of LLMs, it cannot attribute an undesired output to the computation either, as empirically verified in\([Basu et al\., 2026](https://arxiv.org/html/2609.13529#bib.bib51)\)\. From a theoretical perspective, as proposed by[Poggio \(2025\)](https://arxiv.org/html/2609.13529#bib.bib37), any efficiently computable function, characterized by efficient Turing computability, is provably compositionally sparse with a DAG structure\. Therefore, to meet the requirement of causal intervention as in Definition[3\.1](https://arxiv.org/html/2609.13529#S3.Thmtheorem1), we need at least multiple checkpoints to model such structures\.

Why not stronger\.One might demand a stronger, white\-box\-style commitment\. However, such semantic throughout\-readability harms the representational capacity of the model\. A representative is the effort to achieve monosemanticity of neurons or directions in activation spaces, supporting clean mapping between semantic concepts and model structures\. However, this is infeasible for LLMs, which must represent far more meaningful concepts than they have dimensions, packing them into near\-orthogonal directions through superposition\([Elhage et al\., 2022](https://arxiv.org/html/2609.13529#bib.bib13)\)\. Add\-dimensional space admits exponentially many such near\-orthogonal directions\([Johnson and Lindenstrauss, 1984](https://arxiv.org/html/2609.13529#bib.bib41)\), and forcing every direction to be monosemantic would abandon this packing and collapse the number of meaningful concepts from exponential to linear in the dimension\. Generative interpretability, however, only commits interpretability at limited checkpoints, leaving most of the model unconstrained to represent knowledge efficiently\.

## 4\.Neuro\-Symbolic Model as an Instance with Generative Interpretability

In the development of AI history, Neuro\-Symbolic Model\([Bhuyan et al\., 2024](https://arxiv.org/html/2609.13529#bib.bib28)\)is a successful instance that exploits the benefits from both Connectionism and Symbolism AI paradigms\. From the perspective of information compression, neural and symbolic models process raw input data in fundamentally different ways\. Symbolic systems compress by projecting onto a pre\-defined schema vocabulary, and the representations are discrete and hard\-bounded that generalize in an algebraic manner\([Marcus, 2001](https://arxiv.org/html/2609.13529#bib.bib42)\), naturally providing human\-understandable states\. Neural networks, however, compress by projecting onto learned patterns that emerge during training as continuous, distributed representations that generalize based on interpolation\.

Before InferenceDuring InferenceAfter InferenceThroughoutWhite\-box Models\([Rudin, 2019](https://arxiv.org/html/2609.13529#bib.bib24)\)——Multiple Interior Checkpoints—Neuro\-Symbolic Models—Single Interior Checkpoint—CBM\([Koh et al\., 2020](https://arxiv.org/html/2609.13529#bib.bib25)\), CB\-LLM\([Sun et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib26)\)—I/O Boundary Only—Vanilla LLM \(CoT\)\([Wei et al\., 2023](https://arxiv.org/html/2609.13529#bib.bib27)\)—External——Post\-hoc Interp\. \(SAEs\([Templeton et al\., 2024](https://arxiv.org/html/2609.13529#bib.bib23)\), circuits\([Lindsey et al\., 2025](https://arxiv.org/html/2609.13529#bib.bib15)\), probing\([Belinkov, 2021](https://arxiv.org/html/2609.13529#bib.bib8)\), etc\.\)Temporal \(earlier\)Spatial \(more pervasive\)

Figure 1\.Interpretability paradigms organized by two temporal and spatial axes\. Shaded cells admit audit and intervention before undesired content is generated\. Neuro\-symbolic models realize generative interpretability within this region while preserving distributed, high\-capacity representations\.### 4\.1\.Symbolic Components as Checkpoints

Unlike the vague patterns in neural networks, the representations in a symbolic system are human\-understandable by construction\. The causality requirement immediately selects among different neuro\-symbolic architectures that meets Definition[3\.1](https://arxiv.org/html/2609.13529#S3.Thmtheorem1)\. Using the categorization criteria proposed in\([Bhuyan et al\., 2024](https://arxiv.org/html/2609.13529#bib.bib28)\), the Type\-3 Neuro\-Symbolic Models, where the neural and symbolic components work in a cooperative way in a feedback loop, best satisfy this requirement\. A representative Type\-3 model is NLProlog\([Weber et al\., 2019](https://arxiv.org/html/2609.13529#bib.bib30)\), where an inference pass consists of composable proof steps represented by predicate symbols, and similarity scoring produced by text embeddings inside each step stays opaque\. The proof steps thus form multiple interior checkpoints within a single inference pass, each auditable against the expected intermediate representation\.

### 4\.2\.Challenges of Building Neuro\-Symbolic Models with LLMs

Neuro\-Symbolic paradigm is a promising candidate to mitigate the black\-box nature of LLMs given by the intrinsic generative interpretability\. However, most existing Neuro\-Symbolic Models are built on a small\-sized neural network component and confined to a specific task domain, as there are several key challenges that block scaling up\.

Challenge 1: Construction of a universal, general\-purpose symbolic system\.In existing Neuro\-Symbolic Models, it is feasible to build the symbolic components from scratch by human experts for a narrow, closed use case and domain, such as logic rules for math theorem proving\. However, this is intractable for building a universal symbolic system covering as many domains as an LLM does for general purpose, such as the Cyc project\([Yuret, 1996](https://arxiv.org/html/2609.13529#bib.bib31)\)starting from 1980s but paused in recent years of “information explosion”\. A promising alternative approach is to build upon enormous existing symbolic systems from all different domains and use data\-driven approaches to align, merge and extend into a colossal one to match the scope of LLMs, as is being studied by us concurrently\.

Challenge 2: Integration of neural and symbolic components\.Although the Type\-3 Neuro\-Symbolic Model mentioned earlier gives us a sketchy prototype of the fusion of neural and symbolic components, the existing instances from this type usually simplify into a rule\-based, fixed workflow template that only works on a specific task\. It seems difficult to characterize a universal workflow pattern that guides the design of general Neuro\-Symbolic Models\. A compromising approach is to abstract the shared workflow template for a task category and build Neuro\-Symbolic Models for it, such as the Matching Neural Network for Information Retrieval proposed by[Zhai \(2024\)](https://arxiv.org/html/2609.13529#bib.bib32), while the shared symbolic structure of the workflow is usually oversimplified\.

Challenge 3: The boundary between neural and symbolic components for appropriate generalization\.The aforementioned difference of information compression echoes a classical topic of knowledge itself in philosophy\. Some knowledge is governed by a continuous structure of similarity\. For example, whether two unseen words are near\-synonyms is judged by graded similarity instead of by any finite rule\.[Wittgenstein \(1953\)](https://arxiv.org/html/2609.13529#bib.bib43)coined a term “family resemblance” to refer to such knowledge, and neural networks are naturally preferred to be used for their representation\. Other knowledge is governed by discrete, exact rules\. For instance, the correctness of a hash or of formal syntax is rule\-defined, and any minor departure is simply wrong rather than approximately right, whose properties can be well captured by symbolic models\. Characterizing knowledge precisely, therefore, is meaningful for determining the proper boundary of the neural and symbolic components in a Neuro\-Symbolic Model\.

## 5\.Conclusion

We argued for a paradigm shift on AI interpretability research from explaining trained models \(observation\) to building auditable models \(intervention\)\. Generative interpretability formalizes this: an architecture exposes semantically meaningful, human\-understandable, and causally intervenable checkpoints along its inference pass, and Neuro\-Symbolic Models instantiate this paradigm\. We want to emphasize that this shift is not optional in agentic AI settings, especially in high\-stake scenarios such as healthcare, legal system and autonomous driving where the attribution of obligations and responsibility is also valued besides average performance on large samples\. The audits at checkpoints on the inference path, ideally, mark a distinct locus of responsibility, so an undesired outcome traces to the step that bears it rather than to the system as a whole\. Additionally, this mission requires efforts beyond technical work, including designing mechanisms of checks and balances on the power of AI audit, and therefore requires sustained collaborations among universities, frontier AI labs and governments\.

## References

- \[1\]\(2026\)AI Forge: A National Partnership for AI Innovation\.Technical reportDefense Advanced Research Projects Agency \(DARPA\)\.Note:Joint program with NSF and NIST/CAISI\. Distribution Statement A: Approved for Public ReleaseExternal Links:[Link](https://www.darpa.mil/sites/default/files/attachment/2026-06/ai-forge-report.pdf)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- I\. Arcuschin, J\. Janiak, R\. Krzyzanowski, S\. Rajamanoharan, N\. Nanda, and A\. ConmyChain\-of\-thought reasoning in the wild is not always faithful\.External Links:2503\.08679,[Link](https://arxiv.org/abs/2503.08679)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Basuet al\.\(2026\)S\. Basu, S\. Y\. Patel, P\. Sheth, B\. Muralidharan, N\. Elamaran, A\. Kinra, J\. Morgan, and R\. BatnijiInterpretability without actionability: mechanistic methods cannot correct language model errors despite near\-perfect internal representations\.External Links:2603\.18353,[Link](https://arxiv.org/abs/2603.18353)Cited by:[§2\.3](https://arxiv.org/html/2609.13529#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Belinkov \(2021\)Y\. BelinkovProbing classifiers: promises, shortcomings, and advances\.External Links:2102\.12452,[Link](https://arxiv.org/abs/2102.12452)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.23.2.1)\.
- Bengioet al\.\(2026\)Y\. Bengioet al\.International AI Safety Report 2026\.Technical reportInternational AI Safety Report Independent Expert Panel\.External Links:2602\.21012,[Link](https://arxiv.org/abs/2602.21012)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Bhuyanet al\.\(2024\)B\. P\. Bhuyan, A\. Ramdane\-Cherif, R\. Tomar, and T\. P\. SinghNeuro\-symbolic artificial intelligence: a survey\.Neural Computing and Applications36\(21\),pp\. 12809–12844\.External Links:[Document](https://dx.doi.org/10.1007/s00521-024-09960-z)Cited by:[§4\.1](https://arxiv.org/html/2609.13529#S4.SS1.p1.1),[§4](https://arxiv.org/html/2609.13529#S4.p1.1)\.
- Biran and Cotton \(2017\)O\. Biran and C\. V\. CottonExplanation and justification in machine learning : a survey or\.External Links:[Link](https://api.semanticscholar.org/CorpusID:3911355)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p2.1)\.
- Brickenet al\.\(2023\)T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. OlahTowards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.htmlCited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Clarket al\.\(2019\)K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. ManningWhat does BERT look at? an analysis of BERT’s attention\.InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,T\. Linzen, G\. Chrupała, Y\. Belinkov, and D\. Hupkes \(Eds\.\),Florence, Italy,pp\. 276–286\.External Links:[Link](https://aclanthology.org/W19-4828/),[Document](https://dx.doi.org/10.18653/v1/W19-4828)Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Daiet al\.\(2022\)D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. WeiKnowledge neurons in pretrained transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 8493–8502\.External Links:[Link](https://aclanthology.org/2022.acl-long.581/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.581)Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Devlinet al\.\(2018\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.CoRRabs/1810\.04805\.External Links:[Link](http://arxiv.org/abs/1810.04805),1810\.04805Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Elhageet al\.\(2022\)N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen, R\. Grosse, S\. McCandlish, J\. Kaplan, D\. Amodei, M\. Wattenberg, and C\. OlahToy models of superposition\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2022/toy\_model/index\.htmlCited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p4.1)\.
- Elhageet al\.\(2021\)N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. OlahA mathematical framework for transformer circuits\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2021/framework/index\.htmlCited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Geigeret al\.\(2025\)A\. Geiger, D\. Ibeling, A\. Zur, M\. Chaudhary, S\. Chauhan, J\. Huang, A\. Arora, Z\. Wu, N\. Goodman, C\. Potts, and T\. IcardCausal abstraction: a theoretical foundation for mechanistic interpretability\.External Links:2301\.04709,[Link](https://arxiv.org/abs/2301.04709)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p3.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.446/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Huet al\.\(2023\)X\. Hu, C\. Rudin, and M\. SeltzerOptimal sparse decision trees\.External Links:1904\.12847,[Link](https://arxiv.org/abs/1904.12847)Cited by:[§3\.2](https://arxiv.org/html/2609.13529#S3.SS2.p2.1)\.
- Jacovi and Goldberg \(2020\)A\. Jacovi and Y\. GoldbergTowards faithfully interpretable nlp systems: how should we define and evaluate faithfulness?\.External Links:2004\.03685,[Link](https://arxiv.org/abs/2004.03685)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p3.1)\.
- Johnson and Lindenstrauss \(1984\)W\. B\. Johnson and J\. LindenstraussExtensions of lipschitz mappings into hilbert space\.Contemporary mathematics26,pp\. 189–206\.External Links:[Link](https://api.semanticscholar.org/CorpusID:117819162)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p4.1)\.
- Kohet al\.\(2020\)P\. W\. Koh, T\. Nguyen, Y\. S\. Tang, S\. Mussmann, E\. Pierson, B\. Kim, and P\. LiangConcept bottleneck models\.External Links:2007\.04612,[Link](https://arxiv.org/abs/2007.04612)Cited by:[§3\.2](https://arxiv.org/html/2609.13529#S3.SS2.p2.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.14.2.1)\.
- Liet al\.\(2019\)H\. Li, Y\. Tian, K\. Mueller, and X\. ChenBeyond saliency: understanding convolutional neural networks from saliency prediction on layer\-wise relevance propagation\.Image and Vision Computing83\-84,pp\. 70–86\.External Links:ISSN 0262\-8856,[Link](http://dx.doi.org/10.1016/j.imavis.2019.02.005),[Document](https://dx.doi.org/10.1016/j.imavis.2019.02.005)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Lindseyet al\.\(2025\)J\. Lindsey, W\. Gurnee, E\. Ameisen, B\. Chen, A\. Pearce, N\. L\. Turner, C\. Citro, D\. Abrahams, S\. Carter, B\. Hosmer, J\. Marcus, M\. Sklar, A\. Templeton, T\. Bricken, C\. McDougall, H\. Cunningham, T\. Henighan, A\. Jermyn, A\. Jones, A\. Persic, Z\. Qi, T\. B\. Thompson, S\. Zimmerman, K\. Rivoire, T\. Conerly, C\. Olah, and J\. BatsonOn the biology of a large language model\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.23.2.1)\.
- Lipton \(2017\)Z\. C\. LiptonThe mythos of model interpretability\.External Links:1606\.03490,[Link](https://arxiv.org/abs/1606.03490)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Madsenet al\.\(2022\)A\. Madsen, S\. Reddy, and S\. ChandarPost\-hoc interpretability for neural nlp: a survey\.ACM Computing Surveys55\(8\),pp\. 1–42\.External Links:ISSN 1557\-7341,[Link](http://dx.doi.org/10.1145/3546577),[Document](https://dx.doi.org/10.1145/3546577)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p2.1),[§2](https://arxiv.org/html/2609.13529#S2.p1.1)\.
- Marcus \(2001\)G\. F\. MarcusThe algebraic mind: integrating connectionism and cognitive science\.MIT Press,Cambridge, MA\.External Links:[Document](https://dx.doi.org/10.7551/mitpress/1187.001.0001)Cited by:[§4](https://arxiv.org/html/2609.13529#S4.p1.1)\.
- Maslejet al\.\(2025\)N\. Maslej, L\. Fattorini, R\. Perrault,et al\.Artificial Intelligence Index Report 2025\.Technical reportStanford University Human\-Centered Artificial Intelligence \(HAI\)\.External Links:[Link](https://aiindex.stanford.edu/report/)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Menget al\.\(2023\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in gpt\.External Links:2202\.05262,[Link](https://arxiv.org/abs/2202.05262)Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Miller \(2017\)T\. MillerExplanation in artificial intelligence: insights from the social sciences\.CoRRabs/1706\.07269\.External Links:[Link](http://arxiv.org/abs/1706.07269),1706\.07269Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p2.1)\.
- Muelleret al\.\(2025\)A\. Mueller, J\. Brinkmann, M\. Li, S\. Marks, K\. Pal, N\. Prakash, C\. Rager, A\. Sankaranarayanan, A\. S\. Sharma, J\. Sun, E\. Todd, D\. Bau, and Y\. BelinkovThe quest for the right mediator: surveying mechanistic interpretability through the lens of causal mediation analysis\.External Links:2408\.01416,[Link](https://arxiv.org/abs/2408.01416)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p2.1)\.
- Nussbaum\-Hofferet al\.\(2026\)N\. Nussbaum\-Hoffer, N\. Calderon, L\. Ein\-Dor, and R\. ReichartLLM explainability with counterfactual chains and causal graphs\.External Links:2606\.05972,[Link](https://arxiv.org/abs/2606.05972)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p3.1)\.
- Pearl \(2009\)J\. PearlCausal inference in statistics: an overview\.Statistics Surveys3,pp\. 96–146\.External Links:[Document](https://dx.doi.org/10.1214/09-SS057)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p3.1)\.
- Pearl \(2013\)J\. PearlDirect and indirect effects\.External Links:1301\.2300,[Link](https://arxiv.org/abs/1301.2300)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Poggio \(2025\)T\. PoggioOn efficiently computable functions, deep networks and sparse compositionality\.External Links:2510\.11942,[Link](https://arxiv.org/abs/2510.11942)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Responsible AI Collaborative \(2024\)Responsible AI CollaborativeAI Incident Database\.Note:Accessed 2026External Links:[Link](https://incidentdatabase.ai/)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Rudin \(2019\)C\. RudinStop explaining black box machine learning models for high stakes decisions and use interpretable models instead\.External Links:1811\.10154,[Link](https://arxiv.org/abs/1811.10154)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.13529#S3.SS2.p2.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.5.2.1)\.
- Scalenaet al\.\(2026\)D\. Scalena, S\. Candussio, L\. Bortolussi, E\. Fersini, M\. Nissim, and G\. SartiBeyond the commitment boundary: probing epiphenomenal chain\-of\-thought in large reasoning models\.External Links:2606\.13603,[Link](https://arxiv.org/abs/2606.13603)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Sunet al\.\(2025\)C\. Sun, T\. Oikarinen, B\. Ustun, and T\. WengConcept bottleneck large language models\.External Links:2412\.07992,[Link](https://arxiv.org/abs/2412.07992)Cited by:[§3\.2](https://arxiv.org/html/2609.13529#S3.SS2.p2.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.14.2.1)\.
- Sundararajan and Najmi \(2020\)M\. Sundararajan and A\. NajmiThe many shapley values for model explanation\.External Links:1908\.08474,[Link](https://arxiv.org/abs/1908.08474)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Sundararajanet al\.\(2017\)M\. Sundararajan, A\. Taly, and Q\. YanAxiomatic attribution for deep networks\.External Links:1703\.01365,[Link](https://arxiv.org/abs/1703.01365)Cited by:[§2\.2](https://arxiv.org/html/2609.13529#S2.SS2.p1.1)\.
- Templetonet al\.\(2024\)A\. Templeton, T\. Conerly, J\. Marcus, J\. Lindsey, T\. Bricken, B\. Chen, A\. Pearce, C\. Citro, E\. Ameisen, A\. Jones, H\. Cunningham, N\. L\. Turner, C\. McDougall, M\. MacDiarmid, C\. D\. Freeman, T\. R\. Sumers, E\. Rees, J\. Batson, A\. Jermyn, S\. Carter, C\. Olah, and T\. HenighanScaling monosemanticity: extracting interpretable features from claude 3 sonnet\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html)Cited by:[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.23.2.1)\.
- Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.External Links:2305\.04388,[Link](https://arxiv.org/abs/2305.04388)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Valmeekamet al\.\(2026\)K\. Valmeekam, V\. Palod, K\. Stechly, A\. Gundawar, and S\. KambhampatiBeyond semantics: the unreasonable effectiveness of reasonless intermediate tokens\.Transactions on Machine Learning Research\.Note:J2C CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=gDE7YcRC3F)Cited by:[§3\.3](https://arxiv.org/html/2609.13529#S3.SS3.p3.1)\.
- Vaswaniet al\.\(2023\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.External Links:1706\.03762,[Link](https://arxiv.org/abs/1706.03762)Cited by:[§2\.1](https://arxiv.org/html/2609.13529#S2.SS1.p1.1)\.
- Wanget al\.\(2024\)S\. Wang, Y\. Zhu, H\. Liu, Z\. Zheng, C\. Chen, and J\. LiKnowledge editing for large language models: a survey\.External Links:2310\.16218,[Link](https://arxiv.org/abs/2310.16218)Cited by:[§2\.3](https://arxiv.org/html/2609.13529#S2.SS3.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, N\. X\. Zhang, H\. He, T\. Nguyen, K\. Yu, H\. Deng, C\. Brandt, D\. S\. Bitterman, L\. Pan, C\. Cheng, J\. Zou, and D\. LiuSafety challenges of ai in medicine in the era of large language models\.External Links:2409\.18968,[Link](https://arxiv.org/abs/2409.18968)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Weberet al\.\(2019\)L\. Weber, P\. Minervini, J\. Münchmeyer, U\. Leser, and T\. RocktäschelNLProlog: reasoning with weak unification for question answering in natural language\.External Links:1906\.06187,[Link](https://arxiv.org/abs/1906.06187)Cited by:[§4\.1](https://arxiv.org/html/2609.13529#S4.SS1.p1.1)\.
- Weiet al\.\(2023\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§3\.2](https://arxiv.org/html/2609.13529#S3.SS2.p3.1),[Figure 1](https://arxiv.org/html/2609.13529#S4.F1.pic1.18.2.1)\.
- Wittgenstein \(1953\)L\. WittgensteinG\. E\. M\. Anscombe \(Ed\.\)Philosophical investigations\.Wiley\-Blackwell,New York, NY, USA\.Cited by:[§4\.2](https://arxiv.org/html/2609.13529#S4.SS2.p4.1)\.
- Wuet al\.\(2025\)X\. Wu, S\. Chakraborty, R\. Xian, J\. Liang, T\. Guan, F\. Liu, B\. M\. Sadler, D\. Manocha, and A\. S\. BediOn the vulnerability of llm/vlm\-controlled robotics\.External Links:2402\.10340,[Link](https://arxiv.org/abs/2402.10340)Cited by:[§1](https://arxiv.org/html/2609.13529#S1.p1.1)\.
- Yuet al\.\(2023\)Y\. Yu, S\. Buchanan, D\. Pai, T\. Chu, Z\. Wu, S\. Tong, B\. D\. Haeffele, and Y\. MaWhite\-box transformers via sparse rate reduction\.External Links:2306\.01129,[Link](https://arxiv.org/abs/2306.01129)Cited by:[§3\.1](https://arxiv.org/html/2609.13529#S3.SS1.p2.1)\.
- Yuret \(1996\)D\. YuretThe binding roots of symbolic ai: a brief review of the cyc project\.pp\.\.Cited by:[§4\.2](https://arxiv.org/html/2609.13529#S4.SS2.p2.1)\.
- Zhai \(2024\)C\. ZhaiLarge language models and future of information retrieval: opportunities and challenges\.pp\. 481–490\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657848)Cited by:[§4\.2](https://arxiv.org/html/2609.13529#S4.SS2.p3.1)\.
- Zhang and Nanda \(2024\)F\. Zhang and N\. NandaTowards best practices of activation patching in language models: metrics and methods\.External Links:2309\.16042,[Link](https://arxiv.org/abs/2309.16042)Cited by:[§2\.3](https://arxiv.org/html/2609.13529#S2.SS3.p1.1)\.