@agopal42: Presenting PoPE today at #ICML2026! We revisit RoPE through the lens of content-position entanglement, and show how pol…
Summary
New paper introduces PoPE, a positional encoding that decouples content and position, addressing a fundamental flaw in RoPE used in many LLMs like Qwen, Gemma, DeepSeek. Presented at ICML2026.
View Cached Full Text
Cached at: 07/06/26, 10:22 PM
Presenting PoPE today at #ICML2026!
We revisit RoPE through the lens of content-position entanglement, and show how polar coordinates can better decouple content from position.
Come by poster #4012 if you’re curious positional embeddings, pretraining or length generalization.
Anand Gopalakrishnan (@agopal42): Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles “what” (content) and “where” (position) information. Our fix (PoPE) is simple but powerful. Paper:
Similar Articles
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
This paper proves that RoPE-based attention fails to distinguish token positions and identity in long contexts, explaining LLM failures within advertised context lengths. Experimental verification shows models optimized for retrieval struggle on simple list tasks.
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
This paper provides a theoretical proof that Rotary Positional Embeddings (RoPE) in Transformer-based language models lose their locality bias and ability to distinguish token order in long contexts, with attention scores becoming no better than random. The authors show that increasing the RoPE base trades off position vs. token distinction and that multi-head, multi-layer architectures cannot compensate for this fundamental limitation.
RIG-RoPE: Relation- and Instance-Gated Rotary Positional Encoding with Duration-Aware Temporal Coordinates
This preliminary technical report proposes RIG-RoPE, a relation- and instance-gated rotary positional encoding with duration-aware temporal coordinates, aiming to address spatial interference and improper temporal scaling in multimodal LLMs. It introduces a gating mechanism for height/width rotations and duration-aware temporal coordinates, but leaves large-scale empirical validation to future work.
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
This paper proposes RoVE, a parameter-free modification to Rotary Position Embeddings that makes value pathways position-sensitive by rotating values simultaneously with keys, transforming RoPE attention into attentive convolution. Experiments on GPT-2 models show consistent gains in few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval.
PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention
PJ-RoPE unifies RoPE's Fourier phase, Jordan-RoPE's finite jets, and ALiBi's affine recency into a single learnable relative-position space, and studies task-driven selection of sectors.