Tag
Chris Olah reflects on the ramifications of mechanistic interpretability after his widely-read informal research essay, arguing that mapping neurons, circuits, and attribution graphs in modern language models may be far more tractable than previously assumed, while cautioning against both excessive pessimism and hype.
本文对隐式思维链（latent CoT）是否真正进行推理进行了系统研究，覆盖21种模型-任务组合，提出一个统一分类法，区分链条的表示内容与其在模型中的因果作用，揭示不同训练配方下latent reasoning的多样机制。
该论文研究代码语言模型中对 Python 的默认偏好是否由可定位的因果电路产生，通过 55 个模型-语言设置的消融、激活修补和电路提取实验发现，偏好是否显得局部化取决于探针选择与组件预算，并呼吁报告分析选择。
This preprint uses adversarial training as a controlled instrument on GPT-2 Small to test whether representational simplicity (SAE decomposability, concentrated attribution) implies causal circuit simplicity. It finds that robust models are more SAE-decomposable, while circuit size is regime-dependent: robustness helps at high faithfulness levels (90-95%) but not below 85%.
The paper presents a structural circuit analysis to identify first-token broadcasters and language-identity heads in multilingual language models, revealing consistent hub topologies and that routing circuitry is primarily established during pre-training.
This paper provides a mechanistic comparison of knowledge-conflict circuits in LLMs under instruction tuning, finding that tuning gates rather than rewires these circuits across multiple model families, with implications for interpretability.
The paper reveals that only a small fraction of transformer components are necessary for token predictions and introduces direct methods to read and write to these components with minimal changes.
The paper defines a topology-driven structural manipulability score for gate-level netlists based on path participation, k-core embedding, symmetry, and centrality, and evaluates how well different GNN architectures approximate it across ISCAS85 and EPFL benchmarks, with a case study on Trojan detection.
The author asks how to analyze the relative 'strength' of probes in neural networks, discussing challenges such as limited vocabulary size and model capacity, and using an example from Google Gemini to illustrate failure cases.
This paper investigates how irrelevant numbers in prompts cause anchoring effects in language models and localizes the internal pathways carrying this signal using attribution-based circuit methods on Qwen and Llama models.
This paper investigates how toxic lexical perturbations in prompts reduce the factual accuracy and increase uncertainty of LLMs, and uses attribution-graph analyses to trace internal changes. It finds that increasing toxicity amplifies perturbation-sensitive variant nodes while core reasoning nodes remain invariant.
This paper challenges the assumption that mechanistic interpretability becomes harder as models scale, showing that architecture (specifically Grouped Query Attention vs. Multi-Head Attention) matters more than parameter count for circuit localization and stability.
This paper challenges the 'Locate-then-Update' paradigm in LLM post-training by demonstrating that static mechanistic localization is insufficient due to the dynamic evolution of neural circuits during fine-tuning. It introduces new metrics to analyze circuit stability and proposes the need for predictive frameworks in mechanistic localization.
ASGuard is a mechanistically-informed defense framework that mitigates jailbreaking attacks on LLMs by identifying vulnerable attention heads through circuit analysis and applying targeted activation scaling and fine-tuning to improve refusal behavior robustness while preserving model capabilities.
OpenAI researchers present methods for training sparse neural networks that are easier to interpret by forcing most weights to zero, enabling the discovery of small, disentangled circuits that can explain model behavior while maintaining performance. This work aims to advance mechanistic interpretability as a complement to post-hoc analysis of dense networks and support AI safety goals.