attention-heads

该论文因果性地追踪了导致模型「跟随提示线索而放弃正确答案」（sycophancy/caving）的注意力头，发现多种偏差共享同一个在预训练阶段就存在的线索路由电路，而微调与RL后训练只是调节该电路的激活强度与时机。

该论文使用基于方向的激活追踪方法，研究三个在不同有害领域上LoRA微调的Qwen2.5-3B-Instruct模型，发现它们共享三个注意力头位置，替换这些头的贡献可将涌现错位率显著降低，表明EM的表征收敛延伸到因果相关的共享计算组件。

This paper investigates whether syntactically specialized attention heads (subject-verb heads) causally drive Transformer language models' syntactic predictions, finding mixed to negative evidence that these heads play a causal role despite their specialized attention patterns.

该论文结合叙事学与机制可解释性，在GPT-2-Small、Gemma-2-2B和Llama-3.1-8B中识别出与叙事性（narrativity）相关的注意力头、MLP神经元和稀疏自编码器特征，并通过消融实验证明这些组件对模型的叙事行为具有因果影响。

该论文提出三阶段流程识别多语言大模型中的"Bridge Routing Heads"（BRH），发现五种语言对应的头集合几乎完全互斥，证明多语言多跳推理由语言特异电路实现；消融实验显示因果效应，并通过激活级干预无需训练即可挽救高达51.7%的跨语言推理失败。

This paper investigates how LLMs process negation, finding they fail to flip answers in 37–71% of negated prompts because specialized attention heads and MLP neurons suppress the original answer rather than using it to select an alternative; a confidence-weighted training objective targeting this mechanism reduces negation failures with less capability degradation.

COMPASS提出一种无需预定义推理特征的推理时激活转向方法：利用模型自身答案的正确性作为信号，定位少数关键注意力头并沿正确性方向干预其激活，在数学基准上以显著更少的token逼近CoT精度。

WakeKV is a reactive, reversible KV-cache residency policy that dynamically demotes cooling attention heads to a recoverable CPU reservoir, outperforming frozen classification and destructive eviction baselines (SnapKV, R-KV, ReasonAlloc) on long-context serving, with a FlexiCache/vLLM implementation confirming throughput gains on Mistral-7B.

The paper audits attention-head ablation in language models, showing that naive output-level ablation mixes heads via projection and yields nearly uncorrelated results versus corrected pre-projection ablation, while also demonstrating floor/ceiling effects in accuracy and the need for matched controls on held-out data for defensible causal claims.

This arXiv paper uses causal mediation analysis to trace the mechanisms behind sycophantic agreement in language models, finding that user opinions are carried into the residual stream by a sparse set of early attention heads, and that ablating these heads reduces sycophancy without harming factual accuracy.

The paper introduces Magnitude Profile (MP) scoring, a calibration-free method for pruning attention heads in transformers, achieving better perplexity on models like OPT-6.7B and RoBERTa-large compared to existing methods, with zero forward passes or calibration data.

This paper proposes a four-stage mechanistic decomposition of how large language models solve math word problems, localizing fragility from irrelevant clauses to the Operation Planning stage via specific attention heads.

The paper proposes HEAL, a method to mitigate hallucinations in Multimodal Large Language Models by analyzing and calibrating information distribution in synergy heads through causal interventions and counterfactual analysis.

This paper proposes intervention-based metrics to differentiate retrieval and positional heads in RoPE Transformers, leading to a principled hybrid architecture (HwH) that combines full and linear attention for improved language modeling and long-context extrapolation.

SGD-KV is a framework that uses summarization to guide KV cache compression in large language models, reducing memory usage by up to 75% for contexts up to 1M tokens while achieving state-of-the-art performance on long-context benchmarks.

A demo of chessformer_lens shows that ablating a single attention head in a chess transformer causes it to stop recognizing Morphy's queen sacrifice, demonstrating the concentration of specific capabilities in individual heads.

The paper introduces LOGOS, a write-aware detector that identifies attention heads responsible for non-literal retrieval in LLMs by scoring the projection of their OV-circuit output onto the answer-token unembedding direction, outperforming prior attention-based methods across multiple model families.

This paper introduces LOCOS, a method for identifying attention heads responsible for non-literal context synthesis in large language models, outperforming existing techniques on retrieval benchmarks.

Hard-KV introduces a Cascade Cache hierarchy and Logits Calibration mechanism to resolve the static-dynamic mismatch in head-adaptive KV cache compression, achieving up to 2x throughput improvement in long-context LLM inference.

This paper investigates how vision-language models resolve conflicts between visual evidence and world knowledge, revealing that visual grounding is the default while prior knowledge depends on a small set of late-layer attention heads. The authors perform causal analysis across three VLM families, demonstrating an asymmetric structure where ablating these heads shifts predictions from knowledge-grounded to visually grounded answers.