When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Summary
This paper identifies a numerical underflow failure mode in ALiBi positional encodings that can zero out attention weights, analyzes its impact, and evaluates mitigation strategies for training models with ALiBi.
View Cached Full Text
Cached at: 08/05/26, 01:45 PM
Paper page - When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Source: https://huggingface.co/papers/2608.03994 We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind.
We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi’s failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval.
Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.
Similar Articles
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
This paper demonstrates that post-training quantization can silently alter how large language models reason, even when task accuracy is preserved, through a taxonomy-based analysis of 30,000 chain-of-thought outputs across multiple models and benchmarks.
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
This paper investigates the failure modes of converting Qwen3-0.6B-Base attention layers to KDA linear attention on a single GPU, identifying an 'interface injury' where the model predicts option labels rather than content, and proposes a format-targeted KL stage to repair it.
Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
This paper investigates short-term attention degradation in LLMs, finding a universal exponential-then-plateau pattern and that function token anchoring is architecture-dependent. Causal tests show that increasing attention mass on function tokens does not improve retrieval, suggesting attention degradation is descriptive rather than prescriptive.
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
This paper studies how post-training quantization introduces new biases in instruction-tuned LLMs, finding that 3-bit precision causes 6–21% of previously unbiased items to develop stereotypes, while standard metrics like perplexity fail to detect this degradation.
Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
This paper investigates the claim that 4-bit weight quantization is nearly lossless for LLM agents, showing that while aggregate benchmark scores stay flat, quantization amplifies existing tool-call failures (e.g., hallucination), hidden by the benchmark's error budget. The authors suggest reporting per-channel error rates and success under shrinking budgets to reveal the masked damage.