This article describes a fork of llama.cpp called focus-llama that implements Declarative Attention from a recent paper, allowing models to declare needed context chunks during inference to optimize KV cache usage and reduce decode time.
I forked llama.cpp to implement Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares in its own output which context chunks it needs <focus magic\_chunks="N">), and the engine listens and restricts what the following tokens can attend to. No scorer, no training: just prompting plus an engine that reacts to the tags. According to their paper, the overall decode time for the response can be reduced to 0.71× (Gemma) and 0.77× (Qwen) compared to vanilla. Those are the paper's vLLM numbers; I haven't benchmarked the fork yet (no accuracy benchmark either). = What works = - llama-server can drop KV token ranges for a request da_rm / da_rm_at), either mid-prefill or after prefill. - Tag-driven mode: given a chunk layout from the client, the server parses the model's first <focus magic\_chunks="N"> tag during generation and drops the other chunks at that point (one-shot, one-way). - A second-sequence mode da_b, needs --kv-unified) leaves the original sequence intact. - Checked with first-token logprobs on small smoke tests (a Qwen hybrid/GDN model and a tiny dense one). = Why a fork = - Stock llama-server has no way to touch KV ranges mid-generation, so this needed server changes. - llama.cpp has no paged block table (the paper's vLLM version rewrites it), so masking alone doesn't reduce the KV that gets read. - Real skipping needs kernel support or compaction. From reading the source, single-token decode on CUDA doesn't skip masked chunks today. - On hybrid models only the attention layers are restricted (same as the paper); the recurrent state is untouched. Repo: https://github.com/edwardyoon/focus-llama Feedback welcome, especially from anyone who has worked on llama.cpp KV/recurrent memory.
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
This paper introduces Declarative Attention, a method that enables language models to declare relevant context regions during inference, significantly reducing KV cache reads with small accuracy trade-offs in long-context tasks.
Lighthouse Attention is a selection-based hierarchical attention mechanism that accelerates long-context pretraining by running forward+backward passes ~17× faster at 512K context and delivering 1.4–1.7× end-to-end speedup at 98K context, validated with Llama-3 530M on 50B tokens.
The article presents the official home for llama.cpp, an open-source local LLM inference engine, highlighting integration with the Pi coding agent via the pi-llama plugin and broad hardware optimization.