LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

Hugging Face Daily Papers Papers
long-context context-compression attention fine-tuning code-reasoning retrieval nlp

Summary

LongAttnComp adapts AttnComp for long-context reasoning by fine-tuning lightweight cross-attention layers and introducing token-level chunking, a top-p algorithm, positional reordering, and a query parser. It achieves strong performance on long-context tasks like code debugging and transfers across multiple model families.

As real-world applications increasingly require processing inputs of 100k+ tokens, the gap between context length and inference efficiency has become a critical bottleneck. Context compression offers a way to reduce prefill costs while preserving task accuracy. However, existing training-free attention-based methods leave substantial gaps in demanding long-context tasks such as code reasoning. We present LongAttnComp, a long-context adaptation of AttnComp that fine-tunes a lightweight cross-attention scoring layer and introduces tokenlevel chunking, a token-budget top-p algorithm, positional reordering, and a formatagnostic query parser. We further design a two-stage fine-tuning recipe for the compressor: Stage 1 builds a general retrieval foundation from NIAH-style data, and Stage 2 extends it with multi-hop and reasoning data for broader long-context task coverage. On InfiniteBench Code-Debug, LongAttnComp matches or exceeds full-context accuracy, substantially outperforms training-free baselines, and transfers across four target models from three families. On LongBench v2, the two-stage recipe largely closes the Stage 1 gap on multi-document reasoning while preserving Code-Debug performance.
Original Article
View Cached Full Text

Cached at: 06/02/26, 03:37 PM

Paper page - LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

Source: https://huggingface.co/papers/2606.01336

Abstract

LongAttnComp adapts AttnComp for long-context processing by fine-tuning lightweight attention layers and implementing token-level chunking and positional reordering techniques.

As real-world applications increasingly require processing inputs of 100k+ tokens, the gap between context length and inference efficiency has become a critical bottleneck.Context compressionoffers a way to reduceprefill costswhile preserving task accuracy. However, existing training-freeattention-based methodsleave substantial gaps in demandinglong-context taskssuch as code reasoning. We present LongAttnComp, a long-context adaptation of AttnComp that fine-tunes a lightweight cross-attention scoring layer and introduces tokenlevel chunking, a token-budgettop-p algorithm,positional reordering, and a formatagnosticquery parser. We further design atwo-stage fine-tuningrecipe for the compressor: Stage 1 builds a generalretrieval foundationfrom NIAH-style data, and Stage 2 extends it with multi-hop and reasoning data for broader long-context task coverage. On InfiniteBench Code-Debug, LongAttnComp matches or exceeds full-context accuracy, substantially outperforms training-free baselines, and transfers across four target models from three families. On LongBench v2, the two-stage recipe largely closes the Stage 1 gap on multi-document reasoning while preserving Code-Debug performance.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2606\.01336

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.01336 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.01336 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.01336 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ACC: Compiling Agent Trajectories for Long-Context Training

arXiv cs.CL

ACC converts multi-turn agent trajectories into long-context QA pairs to train LLMs on long-range reasoning without additional annotation, achieving significant gains on MRCR and GraphWalks benchmarks while preserving general capabilities.

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

arXiv cs.AI

Introduces OctoLong, a context engineering pipeline for curating dependency-rich cross-repository code contexts, and OctoLong-Instruct, a suite of long-context open LMs trained on this data. Experiments show that replacing 12% of traditional long-context corpora with OctoLong data yields substantial gains in long-range retrieval, state tracking, repository-level code understanding, and agentic tasks.

End-to-End Context Compression at Scale

Hugging Face Daily Papers

This paper presents Latent Context Language Models (LCLMs), a family of encoder-decoder compressors that efficiently handle long contexts through architectural search and large-scale pretraining, outperforming traditional KV cache methods in accuracy, speed, and memory usage.

@VukRosic99: Long-context Transformers hit two walls: quadratic attention compute and a KV cache that reaches hundreds of GB at 1M t…

X AI KOLs Timeline

MiniCPM-SALA is a 9B-parameter hybrid attention model that interleaves sparse and linear attention to overcome the quadratic compute and large KV cache bottlenecks of long-context Transformers. It achieves 3.5x faster inference than Qwen3-8B at 256K tokens and supports up to 1M tokens on consumer GPUs, with a cost-effective continual training approach that reduces training costs by ~75%.