Context Memorization for Efficient Long Context Generation

Hugging Face Daily Papers Papers
long-context attention-memory prefix-inference efficient-inference lookup-memory llm inference-optimization

Summary

Proposes attention-state memory, a training-free approach that stores precomputed attention states in lightweight memory to improve accuracy and reduce latency for long prefix inference, outperforming traditional methods on benchmarks.

Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. While prefix-augmented inference is effective, it incurs two structural limitations: i) the prefix's influence fades as generation proceeds, and ii) attention computation over the prefix scales linearly with its length. Existing approaches either keep the prefix in attention while compressing it, or internalize it into model parameters through gradient-based training. The former still attends to the prefix at inference, while the latter is training-intensive and ill-suited to prefix updates. To address these issues, we propose attention-state memory, a training-free approach that externalizes the prefix into a lightweight, lookup-based memory of precomputed attention states between prefix and query tokens. On ManyICLBench with LLaMA-3.1-8B, our method improves accuracy over in-context learning at 1K-8K memory budgets while reducing attention latency by 1.36x at 8K, and surpasses full-attention RAG performance on NBA benchmark using only 20% of its memory footprint.
Original Article
View Cached Full Text

Cached at: 05/20/26, 10:37 AM

Paper page - Context Memorization for Efficient Long Context Generation

Source: https://huggingface.co/papers/2605.18226

Abstract

Attention-state memory enables efficient long-prefix inference by storing precomputed attention states in lightweight memory, improving accuracy and reducing latency compared to traditional methods.

Modern large language model (LLM) applications increasingly rely on long conditioning prefixes to control model behavior at inference time. Whileprefix-augmented inferenceis effective, it incurs two structural limitations: i) the prefix’s influence fades as generation proceeds, and ii)attention computationover the prefix scales linearly with its length. Existing approaches either keep the prefix in attention while compressing it, or internalize it into model parameters through gradient-based training. The former still attends to the prefix at inference, while the latter is training-intensive and ill-suited to prefix updates. To address these issues, we proposeattention-state memory, a training-free approach that externalizes the prefix into a lightweight,lookup-based memoryofprecomputed attention statesbetween prefix and query tokens. OnManyICLBenchwithLLaMA-3.1-8B, our method improves accuracy overin-context learningat 1K-8K memory budgets while reducing attention latency by 1.36x at 8K, and surpassesfull-attention RAGperformance onNBA benchmarkusing only 20% of its memory footprint.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2605\.18226

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.18226 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.18226 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.18226 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory

arXiv cs.CL

This paper investigates memory-managed long-context attention, a research direction that separates efficient state compression from explicit editable memory slots. Experiments show that a hybrid approach combining fast recurrent/sparse backbones with explicit memory management outperforms pure fixed-state or pure sparse methods across synthetic tasks and long-context benchmarks.

Dynamic Linear Attention

arXiv cs.CL

This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.

Interdomain Attention: Beyond Token-Level Key-Value Memory

arXiv cs.LG

Proposes Interdomain Attention, a new method that integrates state space models into attention via kernel methods, achieving efficient long-context modeling with a fixed-size state and outperforming SSMs and softmax attention in language modeling experiments up to 1.3B parameters.

Dynamic Linear Attention

Hugging Face Daily Papers

DLA introduces adaptive state merging and capacity-bounded memory modeling for multi-state linear attention, improving long-context LLM performance.