attention

This paper introduces SinkFlex-RL, a modular training system for memory-feasible reinforcement learning in long-horizon tool-use agentic tasks. It combines a Gymnasium-compatible environment wrapper, GRPO-based policy optimization, and a sink-aware FlexAttention path, reducing peak VRAM by 19.7% at 4096 tokens and enabling 8192-token runs where eager attention runs out of memory.

This paper applies Marchenko-Pastur random matrix theory to pre-trained attention weights, separating each projection matrix into a random-like bulk and spectral outliers. Causal experiments show zeroing these outliers in Mistral-7B drives performance near random chance, revealing that spectral outliers encode dominant learned structure across 11 transformers.

Motif 3 is a 314B-parameter Mixture-of-Experts language model with 13.2B active parameters per token, featuring Grouped Differential Latent Attention and trained on 12.5T tokens, demonstrating competitive performance across reasoning, coding, and long-context tasks.

This preliminary technical report proposes RIG-RoPE, a relation- and instance-gated rotary positional encoding with duration-aware temporal coordinates, aiming to address spatial interference and improper temporal scaling in multimodal LLMs. It introduces a gating mechanism for height/width rotations and duration-aware temporal coordinates, but leaves large-scale empirical validation to future work.

This paper establishes theoretical bounds on the number of attention heads needed to produce vector representations that support multiple tasks, such as computing min/max and XOR, showing trade-offs between head count, embedding dimension, and precision.

This paper proposes NOVA-KV, a transform-coding approach to KV cache quantization that uses attention-preserving transforms to allocate bits where queries actually attend, improving long-context retrieval accuracy at low bit rates compared to prior methods.

This paper studies how an agent with limited perceptual bandwidth should allocate interoceptive precision across bodily needs in a foraging task, showing that dynamically attending to the most-needed channel improves survival under a fixed precision budget.

Proposes OptR, an output-aware rotation method for INT2 KV-cache quantization that minimizes post-output attention error, improving QuaRot and OSCAR across models and benchmarks.

Su Jianlin reviews the K3 architecture, focusing on the combination of KDA + MLA + Stable LatentMoE + AttnRes. He explains the design trade-offs, MoE stability improvements, why MLA was kept, and the relationship between DSV4 and MLA.

DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, featuring a hybrid compressed attention mechanism that reduces KV cache needs for 1M-token contexts. It can be served with SGLang on Modal for fast decoding on a single B300.

NVIDIA explains how attention architecture choices (group size, head dimension, KV-cache size, parallelism) set the ceiling for long-context inference performance, with guidelines for co-designing models for faster serving.

ResKV proposes a KV cache compression method that divides a fixed budget into an exact main cache and a compact residual cache to reconstruct omitted attention contributions, improving performance on LongBench and RULER across multiple backbones.

This paper proposes SEDR-Seq2P, a lightweight dilated residual sequence-to-point network for multi-task industrial non-intrusive load monitoring (NILM), achieving improved accuracy and 58% lower inference latency compared to WaveNet.

This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.

This survey presents a systematic taxonomy of memory mechanisms in large language models, classifying along axes of representation, update dynamics, and persistence, and formalizing the underlying mechanistic components.

MM-ShiftKV is a training-free method that improves KV cache selection for multimodal LLMs by approximating decoding-time query behavior during prefilling, reducing memory footprint while preserving performance.

This paper presents the first mechanistic interpretability study of Multi-head Latent Attention (MLA), analyzing how its low-rank bottleneck separates content from positional information and reshapes transformer circuits.

A novel patient-independent multimodal framework for automatic depression detection integrates BiLSTM with intra- and cross-modal attention and domain-adversarial training to improve generalization across speakers. It achieves state-of-the-art accuracy of 93.2% on the Androids-Corpus dataset.

Proposes RED-PIM, an algorithm-architecture co-design that reduces inter-bank data movement from O(N^2) to O(N) and shrinks attention matrices, achieving significant inference time reductions (16% to 99.99%) for transformer models.

This technical report explores adding lightweight depthwise convolution to the query/key/value projections in Transformer blocks for LLMs, providing local inductive bias that improves downstream accuracy with negligible parameter cost.