low-rank-attention

FLARE++ is a low-rank attention architecture that replaces static learned queries with input-conditioned dynamic routing, improving on FLARE across PDE surrogate benchmarks and Long Range Arena while preserving linear complexity.

This paper introduces Low-Rank Attention Residuals (LR-AttnRes) for LLMs, which decouple routing from representation by using low-dimensional keys for depth-wise attention, improving performance while reducing FLOPs.

Proposes a hierarchical attention mechanism using overlapping Schwarz domain decomposition to replace dense global low-rank attention with a two-level additive structure of local and coarse blocks, showing faster training and better accuracy with fewer parameters.