Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Summary
This paper presents the first systematic study of massive activations in hybrid linear-attention LLMs, uncovering pre-attention spikes and inter-spike plateaus governed by cancellation timing, and showing how their morphology recovers at full-attention limits.
View Cached Full Text
Cached at: 08/14/26, 03:26 AM
Paper page - Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Source: https://huggingface.co/papers/2608.12149
Abstract
Massive activations in hybrid linear-attention LLMs exhibit pre-attention spikes and inter-spike plateaus governed by cancellation timing, with morphology recovering at full-attention limits.
We present the first systematic study ofMassive activations(MAs) inlayer-interleaved HLA LLMsand uncover two architecture-aligned morphologies: MAs consistently spike immediately beforefull attentionlayers, formingpre-attention spikes(PAS), and can persist through interveninglinear attentionlayers, giving rise tointer-spike plateaus(ISP). Asfull attentionbecomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology offull attentionLLMs. We establish the recurrence of this organization across fivelinear attentionarchitectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining ofGDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically tooutput gating:full attentionoutput gatingstrongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removingGDNgates yields comparatively modest amplification. Mechanistically, oursystematic-outlier analysissupports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At thefull attentionlimit, this account recovers the stable MA morphology characteristic offull attentionLLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.
View arXiv pageView PDFGitHub4Add to collection
Get this paper in your agent:
hf papers read 2608\.12149
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.12149 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.12149 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.12149 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models
This paper identifies the 'Massive Emergence Layer' where extreme activations in LLMs originate and propagate, proposing a method to mitigate their rigidity and improve model performance on tasks like math reasoning and instruction following.
Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
This paper investigates short-term attention degradation in LLMs, finding a universal exponential-then-plateau pattern and that function token anchoring is architecture-dependent. Causal tests show that increasing attention mass on function tokens does not improve retrieval, suggesting attention degradation is descriptive rather than prescriptive.
Retrofitting Linear Attention into Diffusion Language Models
This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.
Measuring Maximum Activations in Open Large Language Models
This paper measures maximum activation magnitudes across 27 checkpoints from 8 open LLM families, finding significant variance across families, architectures, and training stages, with implications for low-bit quantization and deployment.
Rethinking the Role of Efficient Attention in Hybrid Architectures
This paper systematically analyzes the role of efficient attention modules in hybrid language model architectures, finding that different designs converge in long-context performance under sufficient training, and that long-range retrieval is primarily carried by full attention while efficient attention shapes the optimization trajectory, revealing a 'Large-Window Laziness' phenomenon.