transformer-attention

Tag

Cards List
#transformer-attention

Pattern-Based Matrix Analysis Under Permutations

arXiv cs.LG ↗ · 6d ago Cached

This paper introduces a pattern distance measuring how close a matrix is to a geometric pattern (band, block-diagonal, arrowhead) after optimal row/column permutations, an alternating Linear Assignment algorithm to compute it, and a pattern-plus-low-rank (PatLR) decomposition evaluated on synthetic, real correlation/graph matrices, and transformer attention heads.

0 favorites 0 likes
#transformer-attention

Why don't Graph Transformers attend to the same token?

arXiv cs.LG ↗ · 2026-10-15 Cached

The paper investigates why Graph Transformers rarely form attention sinks compared to LLMs, showing that the attention scoring function — not graph structure — is the main cause: replacing GRIT's gated logit with dot-product attention raises its sink rate from 0.09 to 0.68.

0 favorites 0 likes
#transformer-attention

Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibratione

arXiv cs.CL ↗ · 2026-09-23 Cached

The paper presents a mechanistic analysis of over-refusal in large language models and proposes Semantic Routing Calibration (SRC), a lightweight, training-free inference framework to dynamically suppress hypersensitive safety heads and mitigate over-refusal while preserving intrinsic safety.

0 favorites 0 likes
#transformer-attention

Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not

arXiv cs.LG ↗ · 2026-09-18 Cached

This paper introduces Stiefel Attention, which constrains transformer query and key projection matrices to the Stiefel manifold using Riemannian optimization, demonstrating improved performance on modular arithmetic grokking and CIFAR-10 patches.

0 favorites 0 likes
← Back to home

Submit Feedback