rope

提出PN-Mix，一种面向RoPE解码器LLM的混合精度键缓存量化方法，通过查询-键对范数识别显著通道对，在约2.26位平均精度下达到或超越现有方法在2.32–2.39位下的精度，且可优雅降级至1.5位。

提出 Kronecker Product Attention (KPA)，用 Kronecker 积紧凑矩阵因子表示键值激活，实现最高 10× 的 KV 缓存压缩，同时在多个模型规模和基准上匹配或超越 TPA、MQA、GQA 和 MLA 等方法。

AttnFuse introduces a composable DSL for compiling custom attention patterns, including RoPE, to fused GPU kernels, achieving speedups over existing methods like PyTorch's flex_attention.

This paper proposes intervention-based metrics to differentiate retrieval and positional heads in RoPE Transformers, leading to a principled hybrid architecture (HwH) that combines full and linear attention for improved language modeling and long-context extrapolation.

A technical survey on position encoding methods in Transformers, covering absolute and relative methods, RoPE, and long-context scaling techniques like Position Interpolation, NTK-aware scaling, YaRN, and LongRoPE.

ATFlash introduces a per-RoPE-wavelength distance window that prunes query-key inner-product terms proportional to each frequency pair's wavelength, cutting 37-48% of attention compute with minimal quality loss and up to 1.31x speedups on long-context LLM inference.

This paper introduces Bifocal Attention, which decouples positional encoding into geometric (standard RoPE) and spectral (learnable harmonic operators) modalities to address the 'Spectral Rigidity' of fixed RoPE, improving algorithmic generalization beyond the training window.

Jet-Long introduces a tuning-free zero-shot method for extending LLM context length by dynamically adjusting RoPE scaling, achieving strong performance on benchmarks up to 128K context with minimal inference overhead.

Introduces Jet-Long, a zero-shot method for long-context extension that dynamically adapts rescaling factors and uses a bifocal attention mechanism, achieving efficient and high-performance processing across varying sequence lengths without retraining.

New paper introduces PoPE, a positional encoding that decouples content and position, addressing a fundamental flaw in RoPE used in many LLMs like Qwen, Gemma, DeepSeek. Presented at ICML2026.

Introduces LPES, a layer-specific positional embedding scaling method that mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors per layer using a genetic algorithm with Bézier curves, achieving up to 11.2% accuracy gain without fine-tuning or latency increase.

This paper investigates why accumulated token-dependent orthogonal transformations, such as those used in PaTH Attention and a simplified variant with SO(2) rotations, enable length extrapolation in transformers. It proves that such transformations become incoherent after a finite number of steps, suppressing attention to distant tokens, and shows both theoretically and experimentally that this mechanism improves extrapolation but eventually degrades at extreme context lengths.

Proposes Block-GTQ, a RoPE-aware bit allocation method for key-value cache quantization that improves long-context performance and memory efficiency by allocating more bits to high-energy RoPE blocks.

Echo-Infinity introduces a learnable evolving memory mechanism for autoregressive video generation, enabling real-time infinite video generation with constant memory cost and state-of-the-art performance.

This post summarizes Efficient AI Lecture 15 on long-context LLMs, covering RoPE position interpolation for context extension, the needle-in-haystack evaluation, and StreamingLLM's attention sink phenomenon and KV cache eviction strategy.

SEGA is a training-free method that improves high-resolution text-to-image generation by adaptively scaling attention across RoPE components based on spatial-frequency structure during denoising steps.

This paper proves that RoPE-based attention fails to distinguish token positions and identity in long contexts, explaining LLM failures within advertised context lengths. Experimental verification shows models optimized for retrieval struggle on simple list tasks.

A social media post discusses the technical implication of applying RoPE rotation directly to KV caches, noting that it leaks positional information into the value matrix V.

This article provides an in-depth technical analysis of the RoPE (Rotary Positional Embedding) design in DeepSeek-V4, focusing on how it handles token compression and shared KV caches in CSA and HCA modules.

Moonshine presents a family of encoder-decoder transformer models for speech recognition that use Rotary Position Embedding (RoPE) and are optimized for live transcription and voice commands, achieving a 5x reduction in compute compared to Whisper tiny.en with no increase in word error rate.