attention

The paper proposes computation-conditioned credit transport (CCT) and the CompPO algorithm for architecture-aware credit assignment in large language model reinforcement learning, showing improved performance over baselines like GRPO.

The article shares an intuition that compaction in RNNs represents agentic recurrence while recursion in RLMs represents agentic attention, comparing different context-handling approaches in AI models.

This paper introduces MARCH, a network architecture that scales recurrent state-space models beyond fixed-size dimensions by caching cumulative recurrent-state checkpoints as content-addressable state anchors, enabling efficient long-range memory retrieval that outperforms linear attention variants on long-context benchmarks.

LoKiFormer proposes a new LLM architecture with Local Fusion Attention and a Knowledge Memory Module to improve pretraining efficiency, achieving 1.33x faster convergence than baselines.

This paper introduces SinkFlex-RL, a modular training system for memory-feasible reinforcement learning in long-horizon tool-use agentic tasks. It combines a Gymnasium-compatible environment wrapper, GRPO-based policy optimization, and a sink-aware FlexAttention path, reducing peak VRAM by 19.7% at 4096 tokens and enabling 8192-token runs where eager attention runs out of memory.

This paper applies Marchenko-Pastur random matrix theory to pre-trained attention weights, separating each projection matrix into a random-like bulk and spectral outliers. Causal experiments show zeroing these outliers in Mistral-7B drives performance near random chance, revealing that spectral outliers encode dominant learned structure across 11 transformers.

Motif 3 is a 314B-parameter Mixture-of-Experts language model with 13.2B active parameters per token, featuring Grouped Differential Latent Attention and trained on 12.5T tokens, demonstrating competitive performance across reasoning, coding, and long-context tasks.

This preliminary technical report proposes RIG-RoPE, a relation- and instance-gated rotary positional encoding with duration-aware temporal coordinates, aiming to address spatial interference and improper temporal scaling in multimodal LLMs. It introduces a gating mechanism for height/width rotations and duration-aware temporal coordinates, but leaves large-scale empirical validation to future work.

This paper establishes theoretical bounds on the number of attention heads needed to produce vector representations that support multiple tasks, such as computing min/max and XOR, showing trade-offs between head count, embedding dimension, and precision.

This paper proposes NOVA-KV, a transform-coding approach to KV cache quantization that uses attention-preserving transforms to allocate bits where queries actually attend, improving long-context retrieval accuracy at low bit rates compared to prior methods.

This paper studies how an agent with limited perceptual bandwidth should allocate interoceptive precision across bodily needs in a foraging task, showing that dynamically attending to the most-needed channel improves survival under a fixed precision budget.

Proposes OptR, an output-aware rotation method for INT2 KV-cache quantization that minimizes post-output attention error, improving QuaRot and OSCAR across models and benchmarks.

Su Jianlin reviews the K3 architecture, focusing on the combination of KDA + MLA + Stable LatentMoE + AttnRes. He explains the design trade-offs, MoE stability improvements, why MLA was kept, and the relationship between DSV4 and MLA.

DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.

NVIDIA explains how attention architecture choices (group size, head dimension, KV-cache size, parallelism) set the ceiling for long-context inference performance, with guidelines for co-designing models for faster serving.

ResKV proposes a KV cache compression method that divides a fixed budget into an exact main cache and a compact residual cache to reconstruct omitted attention contributions, improving performance on LongBench and RULER across multiple backbones.

This paper proposes SEDR-Seq2P, a lightweight dilated residual sequence-to-point network for multi-task industrial non-intrusive load monitoring (NILM), achieving improved accuracy and 58% lower inference latency compared to WaveNet.

This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.

This survey presents a systematic taxonomy of memory mechanisms in large language models, classifying along axes of representation, update dynamics, and persistence, and formalizing the underlying mechanistic components.

MM-ShiftKV is a training-free method that improves KV cache selection for multimodal LLMs by approximating decoding-time query behavior during prefilling, reducing memory footprint while preserving performance.