low-rank-attention

This paper introduces Low-Rank Attention Residuals (LR-AttnRes) for LLMs, which decouple routing from representation by using low-dimensional keys for depth-wise attention, improving performance while reducing FLOPs.

Proposes a hierarchical attention mechanism using overlapping Schwarz domain decomposition to replace dense global low-rank attention with a two-level additive structure of local and coarse blocks, showing faster training and better accuracy with fewer parameters.