Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Summary
This paper applies Marchenko-Pastur random matrix theory to pre-trained attention weights, separating each projection matrix into a random-like bulk and spectral outliers. Causal experiments show zeroing these outliers in Mistral-7B drives performance near random chance, revealing that spectral outliers encode dominant learned structure across 11 transformers.
View Cached Full Text
Cached at: 08/11/26, 08:08 AM
# Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention Source: [https://arxiv.org/abs/2608.07921](https://arxiv.org/abs/2608.07921) [View PDF](https://arxiv.org/pdf/2608.07921)[HTML \(experimental\)](https://arxiv.org/html/2608.07921v1) > Abstract:We apply Marchenko\-Pastur \(MP\) random matrix theory to pre\-trained attention weights in order to separate each projection matrix into a random\-like bulk and a set of spectral outliers\. We validate this decomposition causally: zeroing the MP\-identified outliers \(signal\) in Mistral\-7B drives HellaSwag, MMLU, and PIQA close to random\-chance performance, whereas zeroing a count\-matched subset of bulk singular values causes smaller but non\-negligible degradation\. Across 11 pre\-trained transformers we identify five recurring patterns: spectral outliers encode a dominant component of the learned structure; Q projections carry the most outliers; V projections under grouped\-query attention lack a clean signal/noise separation; entry\-level outliers form structured row\-bands in Q and column\-bands in O; and specific residual\-stream dimensions persist as band outliers across layers in K and O\. We close by outlining how these observations could inform parameter\-efficient fine\-tuning and structured pruning\. ## Submission history From: Kasun Dewage Dewage \[[view email](https://arxiv.org/show-email/963cdfa0/2608.07921)\] **\[v1\]**Sat, 8 Aug 2026 04:56:11 UTC \(9,211 KB\)
Similar Articles
I Found a Hidden Ratio in Transformers That Predicts Geometric Stability [R]
The article presents a discovered spectral ratio between MLP and attention norms that predicts geometric stability in transformer models, with an optimal range of 0.5–2 to prevent rank collapse.
Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention
This paper applies Wilsonian renormalization group theory to analyze Transformer attention as a perturbation of the MLP residual-stack fixed point, determining whether attention is relevant or irrelevant based on data correlation length. Experiments on synthetic Markov chains confirm that attention's relevance depends on the spectral structure of the data-generating process, with the first-layer head dominating the transition.
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
Introduces a three-step recipe for identifying attention-head circuits in pretrained transformers using a spectral signal and task-pattern screen without requiring labels, validated across 51M to 1B parameter models and multiple architectures.
@antoniolupetti: "Understanding Transformers and Attention Mechanisms" is a very interesting paper that presents the Transformer archite…
A tweet highlights an arxiv paper by Michel Fabrice Serret that introduces Transformers and attention mechanisms from an applied mathematics perspective, covering vectorization, multi-head attention, and methods to reduce attention costs like KV caching and latent attention.
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
This paper introduces the Entropic Bound, a spectral measure of task-intrinsic capacity for transformers, proving that the intrinsic rank of the token-mixing operator provides a tight lower bound on required model capacity. It shows that while a naive transfer from linear attention fails for real attention, an attention-native intrinsic rank restores the full theoretical structure.