Tag
FreeSolo introduces a native sparse attention mechanism combining short-range sliding window and long-range keypoint-based branches, achieving linear-time complexity and improved zero-shot length extrapolation to 256k+ tokens without costly long-context fine-tuning.
This paper analyzes the mechanics of long-context hybrid LLMs combining full attention with sliding-window (SWA) or gated linear attention (GLA/GDN), identifying a Seesaw Effect in context extension and proposing Sliding-Window Linear Attention that achieves 16x training-free length extrapolation with 100% NIAH-SK1 accuracy at 64k context.
论文提出 Recurrent Looped Transformer (RLT)，将层分为并行因果编码器和循环解码器，使计算路径随序列长度增长而每 token 成本固定，在奇偶校验、S5 置换追踪和模运算等算法任务上显著优于标准 Transformer 的长度外推能力。
This paper introduces Complex KDA, an enhancement to Kimi Delta Attention that improves the expressivity of linear RNNs by enabling 2D rotations and state-tracking capabilities, achieving superior length extrapolation and language modeling performance.
Introduces HiLS Attention, a chunk-wise sparse attention mechanism for LLMs that learns chunk selection end-to-end via LM loss, achieving performance comparable to full attention while enabling ultra-long-context extrapolation and faster inference.
This paper investigates why accumulated token-dependent orthogonal transformations, such as those used in PaTH Attention and a simplified variant with SO(2) rotations, enable length extrapolation in transformers. It proves that such transformations become incoherent after a finite number of steps, suppressing attention to distant tokens, and shows both theoretically and experimentally that this mechanism improves extrapolation but eventually degrades at extreme context lengths.