attention

该论文提出 InfiLoop，一种为循环 Transformer 设计的原生残差连接机制，通过内容加权与学习式时间衰减维持循环状态的运行摘要，使模型在数万次测试时迭代中持续提升推理性能，在 Sudoku-Extreme 上达到 97.9% 精度、ARC-AGI-2 上 13.6% pass@2。

The paper introduces a marginal attention space for language models, showing that token-wise reduction yields a text-intrinsic signal conserved across 60+ LLMs while head-wise reduction forms a model-private signature, enabling offline per-head KV-cache eviction budgets that perform competitively with per-document methods.

The paper introduces ACG-Bench, a benchmark for arm-wise compositional generalization in dual-arm vision-language-action policies, and proposes AE-VLA, which combines skill-specific LoRA adapters with arm-wise attention to significantly outperform π₀.₅-based baselines on unseen task compositions in simulation and on physical SO101 robots.

The paper introduces STCFormer, an adaptive spatio-temporal Transformer that dynamically clusters weather stations per temporal patch using a Cluster-Guided Attention Block combining local intra-cluster attention with global regional summaries, achieving the lowest 24-hour MSE across all eight temperature and wind forecasting tasks.

A Berkeley paper finds that LLMs often retain outdated preferences or deadlines in context and keep attending to stale values despite knowing the update; providing the current state explicitly (or steering attention to the newest value) dramatically improves performance without retraining.

This paper mechanistically analyzes how positional encoding choices (RoPE vs. SWA NoPE hybrids) shape in-context retrieval across 22 open-weight models, showing that PE hybrids shift from positional to semantic retrieval mechanisms, explaining their long-context gains and trade-offs.

A thread discussing DepthBench, a new benchmark designed to fairly compare depth-scaling techniques like Kimi K3's AttnRes, DeepSeek V4's mHC, and ByteDance's HC, which were previously impossible to compare directly due to differing training budgets and architectures.

LoopVL extends Loop Transformers to vision-language models, iteratively updating a unified multimodal state through shared modules (Module-Loop and Model-Loop) and outperforming similarly sized non-recurrent models on understanding and reasoning benchmarks, with notable 'Visual Aha Moments' in attention.

QuantWM 提出一种免训练的 2-bit KV cache 量化框架，通过量化敏感性感知聚类（QSAC）与主子空间注意力补偿（PSAC）来稳定视频世界模型的注意力 logits，缓解时间闪烁与视觉退化，并在多个基准上实现最高 6.20 倍 KV cache 内存压缩。

This paper identifies 'phase sensitivity' in models using chunked KV-cache compression, showing that retrieval accuracy oscillates in cycles matching the compression stride and can flip answers or shift long-context accuracy by up to 40 points at 128K tokens, an effect confirmed via controlled 0.6B pretraining experiments to stem from the compression itself.

MinkowskiPE proposes a Minkowski-geometry-based positional encoding that makes attention scores depend only on relative spacetime displacement, preserving spacetime translation invariance while remaining FlashAttention-compatible, and shows consistent gains on molecular dynamics and video prediction tasks.

提出了 Extender，一种日志结构化的 Transformer 架构，通过增加一条拼接通道 ε 而非仅依赖残差通道 h，在 924M 参数规模下实现与 Transformer 相当的短上下文精度、更优的长上下文（RULER）表现，且持久化注意力内存足迹较多头注意力缩小 104 倍。

The paper introduces CacheBack, a training-free receiver-conditioned latent communication method that filters and compresses an agent's KV cache based on the receiving agent's information needs, reducing transferred state by 75%, improving accuracy by 14.7 points, and cutting latency 3.2x vs text communication on FanOutQA with Qwen3.

The author reflects on deleting Instagram five years ago and discusses how it impacts daily life and relationships through constant comparison and attention, viewing it more extremely from the outside.

AttnFuse introduces a composable DSL for compiling custom attention patterns, including RoPE, to fused GPU kernels, achieving speedups over existing methods like PyTorch's flex_attention.

This paper introduces a constant-size state cache for block diffusion models, demonstrating significant reductions in memory and latency compared to attention-based approaches, enabling efficient long-context generation without quality loss.

The paper proposes computation-conditioned credit transport (CCT) and the CompPO algorithm for architecture-aware credit assignment in large language model reinforcement learning, showing improved performance over baselines like GRPO.

The article shares an intuition that compaction in RNNs represents agentic recurrence while recursion in RLMs represents agentic attention, comparing different context-handling approaches in AI models.

This paper introduces MARCH, a network architecture that scales recurrent state-space models beyond fixed-size dimensions by caching cumulative recurrent-state checkpoints as content-addressable state anchors, enabling efficient long-range memory retrieval that outperforms linear attention variants on long-context benchmarks.

LoKiFormer proposes a new LLM architecture with Local Fusion Attention and a Knowledge Memory Module to improve pretraining efficiency, achieving 1.33x faster convergence than baselines.