@ethantsliu: LLMs can control their own attention for long-context! During text generation, LLMs typically read the full KV cache at…

LLMs can control their own attention for long-context! During text generation, LLMs typically read the full KV cache at every step (causing O(N) memory per step). Although attention naturally concentrates on a fraction of the context, identifying which tokens matter beforehand is challenging Existing methods rely on proxy scores or static heuristics that still incur O(N) overheads per step or degrade performance. The authors propose declarative attention (DA) which prompts the model to explicitly declare which parts of the context it needs to attend to in its CoT. DA divides the reasoning process into three distinct attention modes: 1) attend to all context segments (default) 2) attend to specific region (e.g., <focus magic_chunks="2">). Used for reasoning over specific information 3) attend only to model's own recent output. Used for synthesizing the final answer from previously extracted values. to enable targeted attention, the long input context is pre-processed into addressable segments: 1. segment per 2048 (splits at natural boundaries like newlines, sentences, etc) 2. magic chunks (presenting segments as simulated tool-use transcript) and leveraging boundary markers (user, assistant, tool tags) that the model already recognizes from its post-training. the authors even made a DA state machine, implemented via custom hooks in vLLM! it parses mode tags as they are generated and updates the attention mask at block granularity, making it compatible with flashattention (allowing the kernel to skip reading dropped blocks) Tested across 15 long-context tasks, this approach slashes attended tokens during decoding by 52.0% for Gemma4-31B and 31.1% for Qwen3.6-27B, reducing projected wall-clock costs to ~0.7x of vanilla decoding with only marginal accuracy drops!