attention-heads

The paper introduces LOGOS, a write-aware detector that identifies attention heads responsible for non-literal retrieval in LLMs by scoring the projection of their OV-circuit output onto the answer-token unembedding direction, outperforming prior attention-based methods across multiple model families.

This paper introduces LOCOS, a method for identifying attention heads responsible for non-literal context synthesis in large language models, outperforming existing techniques on retrieval benchmarks.

Hard-KV introduces a Cascade Cache hierarchy and Logits Calibration mechanism to resolve the static-dynamic mismatch in head-adaptive KV cache compression, achieving up to 2x throughput improvement in long-context LLM inference.

This paper investigates how vision-language models resolve conflicts between visual evidence and world knowledge, revealing that visual grounding is the default while prior knowledge depends on a small set of late-layer attention heads. The authors perform causal analysis across three VLM families, demonstrating an asymmetric structure where ablating these heads shifts predictions from knowledge-grounded to visually grounded answers.

This paper extends contextual entrainment from token-level to sentence-level, showing that even counterfactual sentences in prompts increase their probability during inference. The effect decreases with model size and is driven by 2-4% of attention heads, which can be ablated without performance loss.

HeRA aligns individual attention heads in Multimodal Large Language Models (MLLMs) to preserve local neighborhood relationships across modalities, improving vision-centric task performance and reducing visual hallucinations.

This paper shows that attention heads meeting common criteria for mechanistic role claims (necessity, linear decodability, ablation reversibility) routinely fail to transfer computations across prompts, and introduces the KID (Knowing/Intent/Doing) framework and a three-stage pipeline for more rigorous role assignment.

This paper identifies a specialized subset of attention heads called CoRe heads in multimodal LLMs that exhibit functional sparsity in cross-modal retrieval. Causal interventions show these heads are crucial for multimodal reasoning, and leveraging this sparsity can accelerate inference.

Proposes MechRL, a reinforcement learning approach to automate circuit discovery in transformer language models. A PPO agent trained on multiple tasks discovers attention head circuits that match known canonical circuits and generalizes to a held-out task.

Introduces a three-step recipe for identifying attention-head circuits in pretrained transformers using a spectral signal and task-pattern screen without requiring labels, validated across 51M to 1B parameter models and multiple architectures.

P2D is a unified framework that leverages task-sensitive attention heads for both data selection and structural pruning, achieving an 8.3 pp performance gain and 7.0× speedup by updating only 10% of heads on 10% of data.

This paper investigates how weight decay acts as a control parameter for transitioning between memorization and generalization in transformers trained on modular arithmetic, and introduces two cheap online diagnostic metrics from attention activations that track these dynamics.

Introduces counterfactual localization to identify when language models become committed to deception during reasoning, using five environments and a corpus of 1.46M sentences across four reasoning models. Shows that attention-based transition features generalize across environments for detecting deceptive commitment.

This paper identifies a circuit underlying a language-switching backdoor in an 8B-parameter language model, where a three-word Latin trigger redirects English output to French via attention heads and orthogonal latent subspaces, with the final layer MLP converting the latent signal to French logits.

This paper investigates prompt-induced hallucinations in vision-language models through mechanistic analysis, identifying specific attention heads responsible for the models' tendency to favor textual prompts over visual evidence. The authors demonstrate that ablating these PIH-heads reduces hallucinations by at least 40% without additional training, revealing model-specific mechanisms underlying this failure mode.