Tag
This article describes a fork of llama.cpp called focus-llama that implements Declarative Attention from a recent paper, allowing models to declare needed context chunks during inference to optimize KV cache usage and reduce decode time.
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
This paper introduces Declarative Attention, a method that enables language models to declare relevant context regions during inference, significantly reducing KV cache reads with small accuracy trade-offs in long-context tasks.