transformer-attention

The paper presents a mechanistic analysis of over-refusal in large language models and proposes Semantic Routing Calibration (SRC), a lightweight, training-free inference framework to dynamically suppress hypersensitive safety heads and mitigate over-refusal while preserving intrinsic safety.

This paper introduces Stiefel Attention, which constrains transformer query and key projection matrices to the Stiefel manifold using Riemannian optimization, demonstrating improved performance on modular arithmetic grokking and CIFAR-10 patches.