Tag
World-in-Context introduces an interactive video world modeling framework that unifies view, action, and camera controls as explicit tokens within a shared self-attention space via In-Context Conditioning, along with Interaction-Transport RoPE for temporal continuity. Its 5B model matches the navigation performance of the 22B EchoWM on WBench while achieving a 93.8% perspective-switching success rate versus 18.8% for existing 5B baselines.
HRoPE introduces a surface-aware positional encoding that models each patch as a slanted 3D plane and injects plane-induced homographies as relative transformations into attention, improving cross-view alignment and novel view synthesis without ground-truth geometry. Experiments on RealEstate10K, CO3Dv2, and DL3DV show it outperforms existing geometric positional encodings.
提出 LayerRoPE，将 Transformer 中隐式的深度位置编码显式化：用单一共享 γ 向量与深度条件标量替代逐层 γ，在参数减少、FLOPs 几乎不变的情况下，在最高 512 层的深度上实现更稳定的训练与显著更高的计算效率（达到 Pre-Norm 1.3B 损失所需算力减少 3.4 倍）。
The paper introduces FOVEATED, a plug-and-play framework that improves atomic-fact recall in unstructured knowledge editing by perturbing RoPE positions of preceding context during editing to counteract context-induced difficulty underestimation, showing consistent gains across five editors, two LLM backbones, and three benchmarks.
This paper mechanistically analyzes how positional encoding choices (RoPE vs. SWA NoPE hybrids) shape in-context retrieval across 22 open-weight models, showing that PE hybrids shift from positional to semantic retrieval mechanisms, explaining their long-context gains and trade-offs.
MinkowskiPE proposes a Minkowski-geometry-based positional encoding that makes attention scores depend only on relative spacetime displacement, preserving spacetime translation invariance while remaining FlashAttention-compatible, and shows consistent gains on molecular dynamics and video prediction tasks.
This paper introduces hierarchical rotary positional encoding (hRoPE) to represent text hierarchy and uses compression depth as a signature of paragraph structure, showing it correlates with genuine hierarchical organization better than other metrics.
This paper presents Wiola, a 13M parameter decoder-only language model with novel components like Spiral Rotary Positional Encoding and Gated Spiral Attention to enhance parameter efficiency for small-scale, on-device language models.
SCoPE is a model from TencentARC that adds camera sightlines as positional coordinates to a pretrained video diffusion transformer, enabling camera trajectory control while preserving the image-to-video prior. The release includes a self-contained checkpoint for Wan2.2-I2V-A14B inference.
An approachable explanation of why transformers need positional encoding, using a bug report analogy and Python's Counter to illustrate how parallel processing loses word order.
ClockRoPE introduces random Fourier rotations to model temporal periodicity in sequential recommendation, theoretically grounded and validated via online A/B tests at a major video-sharing platform.
This paper reveals that frontier LLMs struggle with exact copying tasks due to limitations in positional encodings, and proposes 2D-RoPE, a method that organizes text into a 2D grid to enable perfect copying, showing advantages in synthetic and large-scale pretraining.
This paper investigates the spectral properties of the QK operator in attention heads, showing that the positional scheme (RoPE, learned absolute, ALiBi) sets a default spectral algebra that acts as a fingerprint consolidated after function rather than a hard constraint.
This paper shows that the geometric symmetry visible from neural network weights depends on the positional encoding and readout observable, and validates this using MLPs trained on 2D signed distance functions with multiple symmetry groups.
New paper introduces PoPE, a positional encoding that decouples content and position, addressing a fundamental flaw in RoPE used in many LLMs like Qwen, Gemma, DeepSeek. Presented at ICML2026.
Wiola is a novel Small Language Model architecture introducing five independently designed components—SRPE, GCLA, ATM, DSFF, and WiolaRMSNorm—aimed at improving efficiency and coherence, released in sizes from 120M to 1.5B parameters and integrated with HuggingFace Transformers.
This tweet shares a well-made explanation of the internal workings of LLMs, covering tokens, embeddings, positional encoding, attention, and feed-forward networks, via a blog post by 0xkato.
LazyAttention introduces a novel attention mechanism that defers positional encoding to enable zero-copy, position-agnostic KV cache reuse across multiple requests. The approach reduces time-to-first-token by 1.37× and increases throughput by 1.40× compared to Block-Attention in RAG settings with skewed document distributions.
Wall Attention generalizes diagonal forget gates to softmax attention, enabling state-of-the-art length extrapolation from 4k to 160k+ context zero-shot and outperforming RoPE and FoX in pretraining. It is released as a drop-in replacement with open-source Triton kernels.
This paper proposes Energy-Gated Attention (EGA) and Morlet Positional Encoding (MoPE) to address missing inductive biases in transformer attention: token salience and scale-adaptive locality. Experiments on TinyShakespeare show superadditive gains when combined, highlighting complementarity.