Language Models Can Control Their Own Attention
Summary
This paper introduces Declarative Attention, a method that enables language models to declare relevant context regions during inference, significantly reducing KV cache reads with small accuracy trade-offs in long-context tasks.
View Cached Full Text
Cached at: 09/03/26, 03:49 AM
Paper page - Language Models Can Control Their Own Attention
Source: https://huggingface.co/papers/2609.02737
Abstract
Declarative Attention lets language models declare relevant context regions during reasoning to skip most KV cache reads, reducing attended tokens with small accuracy trade-offs.
Language models spend most of their attention on a small fraction of context, yet they read the entireKV cacheto find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation,global attentionlayers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn’t the model already know which parts of the context are relevant? To this end, we introduceDeclarative Attention(DA), a protocol that elicits the model to declare where it needs to attend within itschain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of theKV cacheread. Under zero-shot evaluation across 15long-contexttasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis ofsparse attention, with further potential under training-based methods that future work can explore.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.02737
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.02737 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.02737 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.02737 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@ethantsliu: LLMs can control their own attention for long-context! During text generation, LLMs typically read the full KV cache at…
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
@omarsar0: Banger paper from Google DeepMind and colleagues. (bookmark it) A model reads its entire KV cache on every generated to…
This paper introduces Declarative Attention, a protocol that allows language models to declare where to attend in their chain-of-thought, reducing attended tokens by 52% on Gemma-4-31B and 31.1% on Qwen-3.6-27B with minimal accuracy drops.
focus-llama: a llama.cpp fork implementing Declarative Attention (arXiv:2609.02737)
This article describes a fork of llama.cpp called focus-llama that implements Declarative Attention from a recent paper, allowing models to declare needed context chunks during inference to optimize KV cache usage and reduce decode time.
Dynamic Linear Attention
This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.
Emergent Structure in the Marginal Attention Space of Language Models
The paper introduces a marginal attention space for language models, showing that token-wise reduction yields a text-intrinsic signal conserved across 60+ LLMs while head-wise reduction forms a model-private signature, enabling offline per-head KV-cache eviction budgets that perform competitively with per-document methods.