Individual Parameters in Weight-Sparse Transformers Appear Interpretable

arXiv cs.LG Papers
mechanistic-interpretability weight-sparse-transformers interpretability llm-pipeline sparse-models

Summary

This paper introduces an automated LLM pipeline to generate and verify human-readable descriptions of when individual weights in transformers matter, finding that 12-31% of weights in weight-sparse transformers are interpretable globally, outperforming dense transformers.

arXiv:2607.02964v1 Announce Type: new Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.
Original Article
View Cached Full Text

Cached at: 07/07/26, 04:40 AM 

# Individual Parameters in Weight-Sparse Transformers Appear Interpretable
Source: [https://arxiv.org/abs/2607.02964](https://arxiv.org/abs/2607.02964)
[View PDF](https://arxiv.org/pdf/2607.02964)

> Abstract:A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does\. Dominant circuit\-finding approaches focus on a specific behavior and reverse\-engineer the role of components on the associated sub\-distribution\. However, past work has shown that components can have different functions that are active on different subsets of the input distribution\. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters \(the inputs on which ablating it changes the model's predictions\)\. We introduce an automated LLM pipeline that writes a short, human\-readable description of when a weight matters and verifies it on held\-out text, crediting a weight only if its description generalizes\. Across two sparse and two dense transformers, the fraction of weights that are interpretable \(in this sense\) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded\. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for\.

## Submission history

From: Arnau Marin\-Llobet \[[view email](https://arxiv.org/show-email/bd7bb0a3/2607.02964)\] **\[v1\]**Fri, 3 Jul 2026 05:15:55 UTC \(4,865 KB\)

Similar Articles

Targeted Recovery of Weight-Space Mechanisms From Neural Networks

arXiv cs.LG

This paper introduces Targeted Parameter Decomposition (tPD), a method that selectively recovers interpretable weight-space mechanisms from neural networks for specific inputs, reducing compute requirements compared to full decomposition. It validates tPD on toy models and transformer language models, demonstrating faithful circuit recovery and surgical ablation with minimal side effects.