Tag
This paper introduces a pattern distance measuring how close a matrix is to a geometric pattern (band, block-diagonal, arrowhead) after optimal row/column permutations, an alternating Linear Assignment algorithm to compute it, and a pattern-plus-low-rank (PatLR) decomposition evaluated on synthetic, real correlation/graph matrices, and transformer attention heads.
The paper investigates why Graph Transformers rarely form attention sinks compared to LLMs, showing that the attention scoring function — not graph structure — is the main cause: replacing GRIT's gated logit with dot-product attention raises its sink rate from 0.09 to 0.68.
The paper presents a mechanistic analysis of over-refusal in large language models and proposes Semantic Routing Calibration (SRC), a lightweight, training-free inference framework to dynamically suppress hypersensitive safety heads and mitigate over-refusal while preserving intrinsic safety.
This paper introduces Stiefel Attention, which constrains transformer query and key projection matrices to the Stiefel manifold using Riemannian optimization, demonstrating improved performance on modular arithmetic grokking and CIFAR-10 patches.