Tag
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
The article speculates that the next major breakthrough after the attention mechanism may involve AI architectures with input-dependent weights, potentially building on DeepSeek's Engram mechanism.
Grouped Value Attention reduces transformer KV cache size by storing grouped values and reconstructing keys via a learned linear map, achieving near-GQA accuracy with a smaller persistent cache.
Random Attention presents a signal-free KV-cache eviction policy for reasoning models that matches or exceeds the performance of learned methods on benchmarks like MATH-500 and LiveCodeBench, while being faster in inference.
RenderFormer-V2 is a transformer-based neural rendering model that handles diverse light-transport effects using a two-stage sequence-to-sequence architecture with improved attention mechanisms and support for heterogeneous scene primitives.
This paper introduces Declarative Attention, a protocol that allows language models to declare where to attend in their chain-of-thought, reducing attended tokens by 52% on Gemma-4-31B and 31.1% on Qwen-3.6-27B with minimal accuracy drops.
This article provides a detailed explanation of the Transformer architecture, covering attention mechanisms, QKV, and residual connections, while tracing its historical development from N-gram to LSTM.
IMPACT is a scalable framework for training interaction-aware world models by using cross-attention as an internal interaction map to reweight denoising supervision, improving performance without external representations or inference-time changes.
This paper introduces Declarative Attention, a method that enables language models to declare relevant context regions during inference, significantly reducing KV cache reads with small accuracy trade-offs in long-context tasks.
The article explains why KV cache stores key and value vectors but not query vectors in autoregressive LLMs, detailing the inference process to optimize token generation.
This paper introduces ClusterAttention, a training-free method to speed up bidirectional attention in transformers by using recursive clustering for block-sparse attention, achieving 2-6x speedups on tabular data and 1.8x on video generation while maintaining high accuracy.
HyQuant improves low-bit LLM attention quantization by preserving critical vertical-line tokens and local windows in high precision while quantizing the rest, maintaining accuracy with low overhead.
A user praises Qwen's model architecture upgrades, which focus on enhancing attention mechanisms for better capability and efficiency.
This article is an educational piece explaining the Transformer architecture, its core components like self-attention and multi-head attention, and its significance in modern AI, as part of a 30-day inference series.
This paper introduces a causal graph-based attention mechanism to enhance retrieval precision in Retrieval-Augmented Generation (RAG) systems, showing improvements in keyword-stuffing regimes of proprietary knowledge bases.
This paper characterizes the lost-in-the-middle effect in clinical LLM applications for EHR processing and introduces Query-Conditioned Clinical Suppression (QCCS) to mitigate it, demonstrating improved accuracy over existing retrieval methods.
A step-by-step walkthrough explaining the core components of the Transformer architecture by manually calculating attention weighting and feed-forward networks to illustrate how Transformers function.
This paper introduces Asymmetric Attention Heads (AAH), a framework that assigns different context windows to attention heads in transformers, with experiments showing improved language modeling performance.
WhiteMatter introduces all-to-all cross-layer connections in Transformers via KV mixing, reducing memory footprint and improving performance over standard architectures in pretraining experiments.
Propose Fractional Decay KV-Cache (FD-KVC), an ownership-aware memory management algorithm for dialog systems that uses dual-channel scoring to improve inference relevancy, outperforming H2O by +6.7% in composite metrics and adapting faster to topic shifts.