Long-form Popular Science: A Comprehensive Overview of AI Interpretability

Large models are becoming increasingly powerful, but we still know very little about what happens inside them. When a network with hundreds of billions of parameters gives an answer, we can see the input and output, but it’s hard to describe exactly what computations occurred in between.

Interpretability research focuses precisely on this black box. It cares not only about what the model says, but also about why the model says it. This might sound like an academic interest, but it has actually become an issue of engineering, compliance, and safety.

I care more about a practical question: If we truly understand the model, how real is that understanding, and what can we do with it? Can we debug models, monitor dangerous behaviors, directly modify models, or even guide training in reverse?

With this question in mind, methods like LIME, SHAP, and SAE are just the entry points. Interpretability is more like a chain, ranging from diagnosis, monitoring, and intervention, all the way to guiding training.

I. Why Do We Need Interpretability

1.1 Three Needs: Debugging, Compliance, and Safety

People want to understand models for different reasons, and the starting point determines the method. This is the root of the fragmentation in the entire field.

The most straightforward motivation is engineering debugging. When a model gives wrong answers, shows bias, or is led astray by a strange input, engineers need to know where the problem lies to have a chance to fix it. Feature importance, LIME, SHAP, and saliency maps in traditional machine learning often serve this need.

The second motivation is trust and compliance. When a bank uses a model to deny a loan, or a hospital uses a model to assist in diagnosis, there must at least be an auditable and appealable basis. In discussions around the EU AI Act and GDPR, automated decisions with high risk or significant impact face stronger transparency requirements. The “right to explanation” here has applicable boundaries, but the direction is clear: high-risk models cannot just output a score without further justification.

The third motivation is AI safety. The more powerful models become, the more people worry they might learn to deceive, ingratiate, or hide their true goals. Looking only at outputs is easily deceived, because models can produce plausible statements. Safety researchers therefore want to look inside the model to see if certain dangerous features are present or if certain computational paths have been activated.

These three needs have inconsistent standards for what constitutes a “good explanation.” Engineers need to pinpoint bugs; regulators need explanations that humans can understand and hold accountable; safety researchers need explanations that are faithful enough to support risk judgments. Many debates that appear to be about methods are actually fundamentally about these underlying standards.

1.2 Model Capabilities Outpace Understanding

Anthropic’s CEO Dario Amodei wrote an article in April 2025 titled “The Urgency of Interpretability.” His metaphor is direct: in the future, we should ideally be able to perform a “brain scan” on frontier models, like a doctor using an MRI to examine a patient, to check for dangerous signs internally before deployment.

The timeline may not be precise, but the problem is real. Model capabilities are growing rapidly, but our understanding has not kept pace. We are increasingly using a system we don’t fully understand—that’s why interpretability has suddenly become so important.

1.3 Clarify the Boundaries First

The first step of interpretability is to understand the model, not to directly constrain it. The order cannot be reversed. Only after understanding what is actually being computed inside the model can we talk about monitoring, intervention, and control.

For small models and isolated tasks, we can already achieve fairly solid understanding. Examples like induction heads and modular addition circuits: researchers not only provide an account but also verify it with causal experiments. When it comes to frontier large models, the situation is completely different. We can extract some features and trace a few circuits, but relative to the entire model, this is still just opening a few small windows into the black box.

A more dangerous scenario is pretending to understand. An explanation can be smooth and the diagrams beautiful, but if it does not correspond to the model’s actual computational process, it can lull people into complacency. Therefore, when discussing any method later, we must first ask about faithfulness: an explanation must first be true before it can be considered understandable.

The difficulty of this field lies here as well. The internal representations of models are not designed for human reading, explanations are hard to definitively falsify, and the research object is evolving rapidly. So interpretability is not a problem that can be solved simply by hiring more people or buying more GPUs. Its difficulty stems from the nature of the problem itself.

II. What Is Interpretability Actually Good For?

2.1 True Understanding vs. False Understanding

Interpretability is easily oversold because the word “understanding” is so convenient. When an explanation reads smoothly and the diagrams look nice, readers easily assume the model is well understood. But the real question is: does this explanation touch upon the model’s actual computational process?

For small models and isolated tasks, the answer is yes. The induction head mentioned earlier is a fully reverse-engineered example; researchers can explain which attention heads combine to form it and what it computes. The modular addition circuit is another example. In 2023, Neel Nanda and colleagues fully decomposed the algorithm by which a small transformer learned modular addition, even discovering it secretly used Fourier transforms. This is true understanding, verifiable through causal experiments.

For frontier large models as a whole, the answer is far from achieved. We can extract millions of interpretable features from a model like Claude, and trace a few circuits, but relative to the entire model’s operation, this is still opening small windows on a giant wall. Saying “we understand large models” is an exaggeration.

2.2 From Diagnosis to Safety Assurance

Understanding must also be layered. The lowest layer is diagnosis: understanding why the model is wrong, where biases are, but not directly modifying the model. For example, determining whether a credit model is secretly relying on a proxy variable like zip code is something industry has been doing for a while.

The next layer is monitoring: watching internal features during model operation to see if signals for “deception,” “sycophancy,” or “malice” are lighting up. This layer is equivalent to installing a dashboard for the model. There has been research progress and some early system exploration, but it’s far from a production standard.

The third layer is editing and intervention. Model editing methods like ROME and MEMIT can locate and rewrite specific facts within a model. Anthropic’s Golden Gate Bridge experiment is a more extreme demonstration: artificially amplifying the “Golden Gate Bridge” feature in Claude 3 Sonnet caused the model to steer any conversation back to the Golden Gate Bridge.

A higher layer is guiding training. Ideally, after discovering a harmful mechanism, researchers could modify data, training objectives, or even include “this feature should not exist internally” in the training process. Today, training mainstream large models still relies heavily on data, compute, scaling laws, and engineering experience; interpretability has not yet become a steering wheel for training.

The highest layer is safety assurance. The goal is to detect high-risk tendencies from within the model before deployment and significantly reduce its probability of engaging in dangerous behaviors. As for mathematically proving “the model will not do a certain class of dangerous things,” there is currently no clear path.

III. What Does “Explanation” Mean, and the Terminology Mess

3.1 Why Several Terms Are Always Confused

This field has a problem that bewilders newcomers: several core terms lack agreed-upon definitions and are often used interchangeably. Interpretability, explainability, XAI, transparency—different papers can assign wildly different meanings.

Zachary Lipton’s 2016 paper “The Mythos of Model Interpretability” specifically argued that the term “interpretability” has a sliding meaning, with different papers having different motivations and often using it as an axiom without definition. If reading survey papers and feeling they contradict each other, it’s likely because they are not even talking about the same thing.

In this article, I adopt a relatively narrow focus: understanding the internal mechanisms and decision-making basis of models. Related topics like fairness, privacy, and robustness are not covered in detail.

3.2 Post-hoc, Intrinsic, Local, Global

To clarify methods, we can use two axes. The first axis is post-hoc vs. intrinsic explanation. Post-hoc explanation means first training a black box, then trying to explain it—LIME, SHAP, and saliency maps fall here. Intrinsic explanation means the model structure itself is transparent—looking at the model gives the explanation; linear regression, logistic regression, and decision trees are examples.

The second axis is local vs. global explanation. Local explanation targets a single prediction: “Why did the model classify this image as a cat?” Global explanation targets the entire model: “Which features does the model generally rely on for decisions?”

Crossing these two axes gives a classification map for nearly all current methods. Whether a method is “post-hoc, local” or “intrinsic, global” clarifies half its nature once placed.

Another crucial concept is the “unit of explanation.” Early methods explained input features, focusing on “which pixels are important in this image” or “which fields are important in this record.” For large models, the object of explanation shifts to internal representations and circuits—“which internal feature or circuit is playing a role.”

3.3 Faithfulness Matters More Than Plausibility

Faithfulness means whether the explanation truly reflects the model’s actual computational process. Plausibility (or readability) means whether the explanation seems credible and convincing to humans. Jacovi and Goldberg’s 2020 ACL paper clearly distinguished this pair of concepts, and it is now a foundational framework for evaluating NLP interpretability.

The key point: these two can completely diverge. An explanation can be very plausible and read convincingly yet have nothing to do with the model’s actual operation.

An explanation that seems reasonable makes you stop questioning. If it happens to be unfaithful, you are deceived without realizing it. A conspicuously rough explanation, on the other hand, keeps you alert.

The strongest counter-evidence comes from Adebayo et al.’s 2018 paper “Sanity Checks for Saliency Maps.” They performed a brutal experiment: randomizing the weights of a trained model layer by layer. Theoretically, the model was useless, and its explanations should also break down. Yet, some saliency methods produced heatmaps that barely changed. This shows those heatmaps were not reflecting what the model learned; they were more like edge detection—anyone seeing them thinks “that makes sense,” but they are unrelated to the model.

This experiment provides a very practical criterion: if the model itself has changed, but the explanation hardly changes, that explanation likely does not capture what the model actually learned. When examining any method later, we must first ask this question.

IV. Classic Methods: Transparent Models, LIME, SHAP, and Saliency

4.1 The Cleanest Explanation: Directly Use Transparent Models

Mechanistic interpretability is exciting, but the classic school remains the most commonly used tool in industry. These methods are more mature and closer to daily model debugging and compliance auditing.

The cleanest interpretability is to start with a structurally transparent model. Linear regression and logistic regression: each feature has a coefficient—its magnitude and sign are clear. Decision trees: each decision is a readable rule.

Generalized Additive Models (GAMs) are a stronger version in this category. They allow each feature to have a nonlinear effect on the outcome, but the effects of each feature are still separate and can be plotted as curves, making them readable. Microsoft InterpretML’s EBM (Explainable Boosting Machine) is a modern representative of this line—essentially a tree-based GAM with automatic interaction detection, tracing back to Lou et al.’s 2013 GA2M work. On many structured data tasks, such models are not inferior in accuracy to black boxes, yet they are inherently readable.

4.2 Post-hoc Attribution for Black Boxes

But in reality, many models are black boxes, leading to a set of post-hoc attribution methods specifically designed to explain “already trained black boxes.”

Permutation importance is the simplest: shuffle the values of a feature and see how much model accuracy drops; the larger the drop, the more important the feature. It originated in Breiman’s 2001 random forest paper, bound with out-of-bag samples, later generalized to a model-agnostic version.

LIME was proposed by Ribeiro et al. at KDD 2016. The idea is to sample points around the instance of interest and use a simple linear model to locally approximate the black box, giving you a local explanation of “which features pushed the prediction up or down in this neighborhood.”

SHAP was proposed by Lundberg and Lee at NeurIPS 2017. Its foundation is the Shapley value from game theory, originally designed by Lloyd Shapley in 1953 for cooperative games. The idea is to compute the average marginal contribution of each “player” across all possible coalitions. SHAP applies this to feature attribution, assigning each feature a “contribution share.” A common misconception: exact Shapley value computation is exponential; SHAP (e.g., KernelSHAP, TreeSHAP) is an efficient approximation, not the exact Shapley value.

Saliency methods are for neural networks, especially image models, using gradients to indicate which parts of the input have the greatest impact on the output. The earliest work is Simonyan et al. 2013, followed by Grad-CAM (ICCV 2017) and Integrated Gradients (Sundararajan et al., ICML 2017, titled “Axiomatic Attribution for Deep Networks”), which integrates gradients along a path from a baseline to the input, satisfying several desirable properties.

4.3 The Anti-Black Box Stance in High-Risk Scenarios

Cynthia Rudin published a strong opinion piece in Nature Machine Intelligence in 2019 with the title “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.”

Her core argument: in high-risk scenarios like healthcare and criminal justice, putting a post-hoc explanation on a black box is dangerous because post-hoc explanations are often unfaithful to the black box and can wrap bad practices in a nice package. Moreover, in many scenarios, structurally transparent models are not inferior in accuracy to black boxes. Instead of struggling to explain a black box, it’s better not to use one in the first place.

This stance is almost opposite to the orientation of mechanistic interpretability and is worth remembering. It reminds us that “interpretability” is sometimes both a technology and a modeling choice. In high-risk scenarios, if a transparent model is already good enough, using it directly is often more reliable than post-hoc explaining a black box.

V. After Large Models, the Unit of Explanation Has Changed

5.1 Input Feature Explanation Is No Longer Sufficient

The classic school works well on structured data and traditional image tasks. But for large language models, it starts to fail. High dimensionality and strong nonlinearity are one aspect; the more fundamental issue is that the unit of explanation has changed.

Imagine you ask a model a question and it generates an answer. A saliency map tells you “the 47th token in the input is most important.” That’s essentially useless. The meaning of language doesn’t reside in individual tokens; the real computations in the model occur across dozens of layers inside. The information in the input layer is far from sufficient to carry an explanation.

Thus, attention shifted from “looking outside” to “looking inside.” For large models, the objects worthy of explanation are internal representations and circuits—where the model stores a concept in which direction, and which attention heads move information around.

This shift was forced by the problem. After input-level explanation units failed, the only way to understand large models was to go inside them. This is the logical starting point for mechanistic interpretability.

5.2 Why Neurons Cannot Be Read

The most natural idea is to look at neurons one by one, figuring out what each represents. This doesn’t work because neurons are polysemantic. The same neuron can activate for completely unrelated inputs like “cat,” “legal document,” “Japanese,” and “some code pattern.” Staring at it, you can’t read a clean meaning.

Why polysemantic? Anthropic’s 2022 paper “Toy Models of Superposition” provided a beautiful explanation called superposition.

Analogy: Suppose you only have ten light bulbs but want to represent hundreds of states. You wouldn’t assign one state per bulb (that gives only ten). Instead, you’d use combinations of bulbs: lighting 1,3,5 means one thing; lighting 2,4 means another. Similarly, a model has many more concepts (features) to represent than it has neurons, so it compresses multiple features into the same set of directions, using combinations to distinguish them. The cost is that individual neurons become polysemantic and unreadable alone.

The key implication of superposition: polysemanticity doesn’t mean the model is lazy; it means the model is using an efficient strategy to pack as much information as possible into limited dimensions. This also foreshadows a troublesome conclusion: superposition may be inherent to the model’s representation and may not be easily eliminated.

VI. Mechanistic Interpretability: Features, Circuits, and Causal Intervention

6.1 From Linear Representations to SAE

Since individual neurons are unreadable, where do concepts actually reside? An influential conjecture is the linear representation hypothesis: concepts are directions in activation space, not individual neurons. Moving along that direction amplifies the “signal” for that concept. Early intuition came from the famous word2vec example: “king” - “man” + “woman” ≈ “queen.”

Emphasize: this is a hypothesis, not a theorem. There are already counterexamples, such as Engels et al. 2024 finding that concepts like “day of week” or “month” form a circular, nonlinear structure in models. So a safe statement is that linear representations are empirically common but have known exceptions.

If concepts are superimposed directions, can we use tools to flatten them into clean, monosemantic features? Sparse Autoencoders (SAEs) do exactly this.

The SAE idea: train a wide, sparse network to reconstruct the activation of a model layer, forcing it to activate only a very small number of units at a time. Ideally, each learned unit corresponds to a human-readable concept. Anthropic’s 2023 paper “Towards Monosemanticity” used this on a single-layer model, extracting about four thousand interpretable features. In May 2024, “Scaling Monosemanticity” scaled it up to the middle layers of a production-grade model, Claude 3 Sonnet, extracting up to about 34 million features, covering concrete objects, abstract concepts, multiple languages, and modalities.

Beyond SAE, there are variants like transcoders. SAE takes a snapshot of a layer, showing what features are present there. A transcoder goes further, using a wide, sparse structure to approximate a dense MLP layer, thereby revealing which input features lead to which output features—exposing the transformation itself. Cross-layer versions (CLT) can read from one layer and write to multiple later layers, serving as a building block for attribution graphs.

6.2 From Features to Circuits

Once clean features are in hand, the next step is to see how they connect to compute things. That’s a circuit.

The induction head is the most classic circuit. Anthropic’s 2021 paper “A Mathematical Framework for Transformer Circuits” first identified it in a small two-layer model, composed of two attention heads from different layers: a “previous token head” and an “induction head.” Its job is simple: seeing that a token [A] was previously followed by [B] in the sequence, it predicts [B] again after [A] reappears, completing the pattern [A][B]…[A] -> [B]. A follow-up paper in 2022 further argued that such circuits might be an important mechanism for in-context learning and that they form abruptly during training at a certain stage. The evidence is stronger in small models; for frontier large models, it remains a strong hypothesis.

The modular addition circuit is another fully reverse-engineered example. As mentioned earlier, in 2023 Nanda et al. discovered that when a small model learns modular addition, it internally uses Fourier transforms and trigonometric identities, turning addition into rotation on a circle. Note: this is “modular addition,” which is different from the ordinary two-digit addition that large models do. For the latter, Claude was found to use a parallel heuristic of “estimate roughly, then compute the last digit precisely” in another study.

In 2025, Anthropic’s two companion papers on circuit tracing and “The Biology of a Large Language Model” used cross-layer transcoders to construct attribution graphs, mapping how information flows from input through intermediate features to output as a readable computational graph, with the study subject being Claude 3.5 Haiku.

6.3 High-Level Structure and Causal Intervention

Mechanistic interpretability has also uncovered some surprising high-level structures.

Planning ahead when writing poetry is one example. In the “Biology” paper, researchers found that when the model writes rhyming poetry, it activates candidate rhyming words at the beginning of the line before writing the entire line, then works backward to compose the line so that it naturally lands on that rhyme. When researchers artificially replaced the “planned rhyme word,” the model reorganized the entire line to accommodate the change. This refutes the intuition that “the model just improvises token by token with no foresight.”

Persona vectors are another example. Anthropic’s 2025 work identified directions that control certain personality traits, such as sycophancy, malice, or a tendency to fabricate, which could be used for monitoring or even preventing these bad traits during training.

There is also introspection. In 2025, Anthropic performed an interesting experiment: directly injecting a direction representing a concept into the model’s internals, then asking the model if it noticed “a thought was planted in its mind.” Under specific experimental settings, Claude Opus 4 and 4.1 identified the injected conceptual direction in about 20% of cases. This must be stated cautiously: the success rate is low, the scope narrow, and it cannot be generalized to “models have reliable introspection.” Anthropic also explicitly states this does not imply consciousness.

All these findings share a common risk: are they merely correlations, not causal? The distinguishing method is intervention.

Activation patching is the core method: replace certain internal activations with corresponding activations from another run and observe how the output changes, thereby determining whether that component has a causal role. A more specific variant is causal tracing, popularized by the ROME paper.

The Golden Gate Bridge experiment is the most famous causal demonstration. In conjunction with the May 2024 “Scaling Monosemanticity,” Anthropic found a “Golden Gate Bridge” feature in Claude 3 Sonnet, artificially amplified its strength, and created a demo model called “Golden Gate Bridge Claude,” briefly open to the public for about a day. After amplification, the model would steer any conversation topic back to the Golden Gate Bridge. This is an example of faithfulness because it proves that feature causally controls model behavior—it’s not a post-hoc story. Note: the Golden Gate Bridge is a feature direction extracted by SAE, not a single “Golden Gate Bridge neuron.”

Putting these results back into the chain from Chapter 2, mechanistic interpretability’s strongest areas are still diagnosis and monitoring. It helps us see what’s happening inside the model and watch for dangerous features. Editing interventions have demonstrations like the Golden Gate Bridge, but are far from reliable production tools. As for guiding training and safety assurance, they are more directional visions than mature engineering capabilities.

VII. Intermediate Methods, Evaluation, and Deployment

7.1 Probes and Attention

Between the classic attribution school and mechanistic interpretability lies a middle ground. These methods may not fully reverse-engineer the internal mechanisms, but they help researchers determine whether certain information exists or whether a certain component might be involved in computation.

Probing is a common tool: freeze the model, take activations from a certain layer, train a simple linear classifier on top, and see if you can read a concept (e.g., part-of-speech, syntax, sentiment) from those activations. If you can, it means the layer “encodes” that information. The source is often cited as Alain and Bengio 2016.

Probing has an unavoidable controversy: being able to decode does not mean the model actually uses it. The information is present, but that doesn’t mean the model causally uses it in reasoning. Moreover, the stronger the probe, the more likely it’s the probe doing the computation, not the model having that structure. Hewitt and Liang 2019 specifically hammered this point using “control tasks.” This is why the field has increasingly relied on causal methods.

There is also a classic debate about whether attention weights can serve as explanations. A natural thought: attention tells you “where the model is looking,” so isn’t that an explanation?

Jain and Wallace’s 2019 “Attention is Not Explanation” (NAACL) threw cold water: they showed they could construct very different attention distributions that produce the same predictions, meaning attention weights cannot be reliably used as explanations. The same year, Wiegreffe and Pinter’s “Attention is Not Not Explanation” (EMNLP) countered: it depends on how you define explanation. Attention provides one kind of explanation, not the only one—presence does not imply exclusivity. This back-and-forth well illustrates the field’s nature: an indicator that seems self-evident turns out to be full of pitfalls. There is also a lesson in faithfulness here: weight does not equal importance.

7.2 How to Evaluate an Explanation

Evaluating an explanation requires measuring several dimensions. Faithfulness: does it truly reflect the model? This is the hardest, and sanity checks are for this. Stability: does the explanation jump around with small input perturbations? Readability: do humans find it useful? This typically requires human evaluation.

The difficulty is that these dimensions often conflict. The most faithful explanation is not necessarily the most readable, and the most readable explanation (recall Chapter 4) can be the most unfaithful.

Honestly, there is currently no widely accepted unified benchmark like ImageNet. Evaluation is highly fragmented, with different methods using their own metrics. There are some causal evaluation benchmarks and task-specific benchmarks (like shared tasks for circuit localization), but far from a field-wide consensus. This “no consensus on what counts as success” state itself is a symptom of the difficulty.

Automated interpretability is a tempting direction. In May 2023, OpenAI used GPT-4 to automatically write explanations for each neuron in GPT-2, then used GPT-4 to predict when the neuron would activate based on the explanation, scoring the explanation by the match between prediction and actual activation.

OpenAI was clear about the limitations. GPT-4’s explanations are far inferior to humans; most neurons cannot be concisely explained (due to polysemanticity again); the proportion of high-scoring explanations is very low. A deeper paradox: using a model you don’t fully understand to explain another model—who explains the explainer? This cycle currently has no exit.

7.3 Where Has It Actually Been Deployed?

Diagnosis and data bias detection is the most solid deployment area, corresponding to the first rung of the ladder. Checking whether a model relies on an unintended proxy variable or if training data has leakage—industry does this continuously.

Safety monitoring corresponds to the second rung, with real research and early products beginning to emerge. Monitoring internal features related to sycophancy, hallucination, and deception during model operation—installing a dashboard for the model—this line is progressing with work like persona vectors.

Model editing corresponds to the third rung. Methods like ROME and MEMIT can directly locate and rewrite specific facts in a model without retraining. This is feasible in research but has not become a production standard—whether editing one thing harms another is still an unresolved problem.

Alignment auditing and compliance correspond to higher rungs. Using interpretability to audit a new model and decide whether to deploy it remains largely a vision with sporadic attempts.

Stringing these deployment scenarios together reveals an order that is easily misunderstood. Interpretability first solves the understanding problem; only then can it serve control. Directly equating interpretability with control reverses the causal relationship. Understanding is the cause; control is the effect.

VIII. Why This Is Difficult, and Who Is Doing It

8.1 Difficulties in Verification and the Model Itself

The difficulty of interpretability begins with verification. You can almost never prove an explanation true; at best, you find self-consistent circumstantial evidence, and it’s hard to truly falsify. An explanation can sound smooth, match various examples, yet you still cannot tell if it truly captures the model’s mechanism or is just another nice story. Great effort, no certainty—this is the most tormenting aspect.

This problem also leads to another trouble: success criteria are hard to unify. Does “I understand a circuit” count as progress? How much progress? If an explanation works for a mechanism in a small model but cannot generalize to frontier large models, how should its value be assessed? There are no unified answers, making it difficult for the field to advance with a recognized benchmark like image recognition.

The second difficulty comes from the model itself. The way models organize information is not designed for human reading. Superposition, polysemanticity, compression—all are ways to pack more capability into limited parameters. Tools like SAE can flatten some superposition, but as long as models pursue capability and efficiency, this problem is unlikely to disappear completely.

8.2 Difficulties in Iteration, Scale, and Compute

The third difficulty is that the research object is constantly changing. By the time researchers understand a mechanism in one model, the next generation of models may have different architectures, scales, and training methods. Interpretability research inherently faces a moving target, and it moves fast.

The fourth difficulty is scale and compute. The most worthy research targets are frontier large models, but fine-grained mechanistic analysis on them is very expensive and usually requires access to model weights. Academia can more easily study small models and open-source models, but findings on small models are then questioned about their generalizability to frontier models.

There is also a less glossy reality: this type of research is very labor-intensive. Much work relies on researchers manually looking at features, labeling, and repeatedly designing intervention experiments. The pretty diagrams in front-end results often hide a mountain of labor with uncertain outcomes. Automated explanation is certainly a direction, but automation itself is stuck on faithfulness and plausibility.

8.3 Who Is Doing It, and Why It’s Still Worth Doing

These difficulties explain the unique division of labor in this field. The classic attribution school is mainly in academia and industrial application frontlines, serving solid needs like debugging and compliance—low-profile but high in volume. On the mechanistic interpretability side, Anthropic has the loudest voice; many works cited earlier come from its interpretability team. But it’s far from alone: Google DeepMind has Neel Nanda’s mechanistic interpretability team, OpenAI has done automated interpretability, and academia and the independent open-source community have formed an active circle around SAE and circuit analysis.

Compute determines the division. Doing mechanistic analysis on frontier large models requires weights, infrastructure, and substantial experimental resources—industry labs naturally hold the advantage. Academia more often innovates on methods and validates principles on small models and open-source models. This division isn’t mandated by anyone; it’s naturally pushed by resource distribution.

This is also why interpretability is still worth doing. Today, its tools are crude, findings scattered, and consensus lacking, but the research object is fundamental enough. If the chain of diagnosis, monitoring, intervention, and training guidance can be linked in the future, it will become a basic inspection capability before deploying strong models.

8.4 Several Unsolved Problems

The most immediate problem is scalability to frontier large models. Today, many impressive results come from small-to-medium models, single mechanisms, or controlled tasks. Whole-model understanding still lacks a clear path.

Whether superposition can truly be addressed is also unanswered. SAE can flatten part of it, but if superposition is inherent to the model’s efficient representation capability, it may only be mitigated, never completely eliminated.

Automated interpretability is stuck here too. As long as interpretation relies mainly on human inspection, scale cannot increase. But using a model we don’t fully understand to explain another model circles back to “who explains the explainer.”

There is an even more fundamental problem: can faithfulness be proven? Today, we mainly gather circumstantial evidence—intervention, ablation, controlled experiments—to increase confidence. A general method to rigorously prove an explanation true is currently not in sight.

Interpretability and neuroscience share a similarity: both reverse-engineer a complex system without a manual. Decades of neuroscience experience—what can be borrowed and what cannot—is also an open question. This analogy cannot be overused, but it reminds us that understanding complex systems is often a long, slow, tool-driven process.

IX. Conclusion

Interpretability is most easily over-hyped. It is not about putting a nice layer of explanation on a model, nor about writing a satisfactory report for regulators. The real problem it addresses is simple: we are increasingly handing decisions to a black box, and we have not truly seen how that black box works inside.

Therefore, the first step is always understanding. Only after understanding can we talk about diagnosing errors, monitoring risks, intervening in behavior, and even influencing training in reverse. The order cannot be reversed. If you haven’t touched the model’s actual computational process, all subsequent “explanation,” “audit,” and “safety assurance” are just packaging.

This is why faithfulness is so important. An explanation can be smooth and pretty, but it must first be true. The Adebayo-style sanity checks remind us not to be fooled by “seems reasonable.” Interpretability’s worst enemy is a false answer—a false answer makes people think they already understand.

Today, the most stable part of this field is still diagnosis and monitoring. Interventions have demonstrations like the Golden Gate Bridge Claude, but are not yet reliable production tools. Guiding training and safety assurance are even farther, often still at the research vision stage. The value of interpretability depends on whether this chain can be linked step by step.

I remain optimistic about this direction, but this optimism cannot be divorced from reality. It will not automatically make models safe, nor automatically give regulators answers. Its true meaning is to push “blind use of black boxes” back a little, and push understanding, verification, and intervention forward a little. As long as model capabilities continue to outpace our understanding, this issue will remain important.

Other Articles by the Author (Selection)

• AI for Science Detailed Introduction (Part 2): Capital and Landscape
• What Is Skill: For Beginners
• The Rise and Fall of the Metaverse: Hype, Bankruptcy, Who Made Money?
• AI Influencer Biographies (Part 3)
• How the Blockchain Narrative Collapsed
• Comprehensive Engineering Panorama of AI (Part 3): Agent Explained
• A Brief History of US Venture Capital
• What Is MCP · Long-form Popular Science
• AI Influencer Biographies (Part 1)
• AI for Science Detailed Introduction (Part 1): Paradigm and Landscape
• Generalized Grandparent Integral Calculus
• What Is “Emergence”? A History of Emergence Research
• A Brief History of the Internet Bubble
• AI Influencer List
• My Personal AI System: Philosophical Foundations
• How the NFT Narrative Collapsed
• What Is Dissipative Structure Theory? Does It Relate to AI?
• What Is Embodied Intelligence? Its Relationship with AI
• Long Analysis: SpaceX Future Outlook
• One Article to Clearly Explain the US Immigration System
• One Article to Clearly Explain the US Healthcare System
• Detailed Look at Old Money Chinese Families in the US
• What Resources Have US Jews and Chinese Taken?
• One Article to Understand the Full US Education Ecosystem
• What Is Cybernetics? Is Cybernetics the Predecessor of AI?
• Introduction to Grandparent Integral Calculus
• Pope Leo XIV on Artificial Intelligence (Essentials)
• Vibe Reading: A Systematic Method for Reading in the AI Era
• Complete Guide to the US Tax System