Tag
The paper introduces two kernel-based interpolatory KV-cache compression methods, GAIM and Q-DAIM, along with linear-time surrogate variants and streaming attention extensions, achieving performance comparable to state-of-the-art on LongBench-E.
Yandex Music introduces Sona, a single transformer that replaced 15+ candidate generators plus pre-ranker and ranker in production A/B tests, using History Compression to halve inference cost while delivering +4.53% Active Users and +6.30% Listening Time.
The paper introduces attention manifolds — learned 2D B-spline surfaces that modulate value dimensions in transformer attention — showing 2–2.5 point perplexity improvements on LLaMA 3.2 with just 0.3% parameter overhead, plus mechanically editable surfaces enabling steering and attention walls for safety.
SMat-Attention introduces structured causal masks with tunable VC-dimension to bridge softmax attention and linear attention, achieving subquadratic prefill, constant-time decoding with O(T^{1-1/d}) cached states, and improved recall over Mamba-2 and Gated DeltaNet backbones.
This survey analyzes the evolution of attention mechanisms in large language models through a five-dimensional lens of contextual memory (representation, update, access, readout, integration), drawing on 59 release-level records across 14 model lineages, and proposes a stateful multidimensional memory-routing hypothesis for future sequence architecture design.
The article introduces two attention mechanisms, CoWindow Attention and MassAlloc Attention, which optimize compute allocation in transformers, achieving significant speedups and reduced training FLOPs in benchmarks.
An interactive diagram comparing self-attention and cross-attention mechanisms in AI models, published as part of an educational library on attention.
This paper presents an interpretability study on video diffusion models, revealing that Rotary Position Embedding (RoPE) induces excessive spatial attention decay, causing physics violations, and proposes a lightweight architectural modification to enhance physical coherence in generated videos.
This paper investigates whether the recurrence in selective state space models drives tokens to consensus similar to attention in transformers, using dynamical systems theory to analyze stability and attraction domains for time-varying weight matrices.
This paper introduces a mean-field analysis of attention that predicts average representation dynamics and reveals context-specific computation in language models, validated across models like GPT-2, Pythia, and Qwen-3-14B.
This article organizes 13 attention mechanisms in AI by the bottleneck they solve, covering KV cache reduction, attention patterns, compute efficiency, and serving efficiency to help AI engineers understand and apply these techniques.
This paper presents a mathematical framework for reverse-engineering transformer models to enhance mechanistic interpretability and address safety concerns in AI systems.
Attention-DP3 enhances 3D diffusion policies by incorporating object-level geometric cues through attention to improve stability in cluttered environments, achieving state-of-the-art performance across benchmarks.
A user announces they have solved all attention-related problems on LeetGPU and compiled them into a resource for easy access.
Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering a cheaper and more reliable inference solution for LLMs.
This paper presents a systematic empirical study of energy consumption in large language model inference, analyzing how attention architectures like Multi-Head Attention, Grouped Query Attention, and Sliding Window Attention affect energy scaling across context lengths and workloads.
ReWorld presents an interactive world model that decouples short-horizon control from long-horizon memory using mixed attention windows and distribution-matching LoRA distillation, enabling real-time video generation with high action fidelity and recall.
ReCache is a framework for efficient KV cache reuse and compression in tool-augmented LLM agents, achieving significant speedup and memory reduction while maintaining performance.
A tweet highlights 'Understanding Transformers and Attention Mechanisms,' a 13-page paper that explains the Transformer architecture and attention from an applied mathematics perspective.
A tweet recommends an arXiv paper that explains the mathematical foundations of Transformers, covering tokenization, embeddings, multi-headed attention, and KV caching for applied mathematicians.