TokenPilot: Cache-Efficient Context Management for LLM Agents
Summary
TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments, achieving 61-87% cost reduction on benchmarks while maintaining competitive performance.
View Cached Full Text
Cached at: 06/16/26, 11:34 AM
Paper page - TokenPilot: Cache-Efficient Context Management for LLM Agents
Source: https://huggingface.co/papers/2606.17016 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments.
AsLLM agentsare deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimizetoken footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity andprompt cache continuity. To address this, we present TokenPilot, a dual-granularitycontext managementframework. Globally,Ingestion-Aware Compactionacts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally,Lifecycle-Aware Evictionmonitors the ongoingresidual utilityof context segments, enforcing a conservativebatch-turn scheduleto offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated andcontinuous modes demonstrate that TokenPilot reduces costs by 61% and 56% inisolated mode, and 61% and 87% incontinuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.17016
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.17016 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.17016 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.17016 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
@rohanpaul_ai: TokenPilot reduces LLM agent costs via ingestion-aware compaction and lifecycle-aware eviction. Achieves 61–87% cost re…
TokenPilot reduces LLM agent costs via ingestion-aware compaction and lifecycle-aware eviction, achieving 61–87% cost reduction on PinchBench and Claw-Eval with competitive scores.
@omarsar0: // The Efficiency Frontier // Cool paper on context management. As agents reuse the same documents and histories across…
This paper introduces The Efficiency Frontier, a unified framework for cost–performance optimization in LLM context management that models context strategy selection as a deployment-aware optimization problem, achieving 25% reduction in token usage and over 50% lower token cost with amortized memory compression compared to full-context prompting.
Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents
This paper evaluates context engineering configurations for LLM agents in enterprise tool-use workflows, showing that summarization with selective pruning achieves 91.6% accuracy while reducing token usage by over 60% compared to full-context baselines.
@IntuitMachine: PEEK: The 1k-Token Map That Just Killed the Long-Context Tax Your LLM agent is reading the same 50k-token codebase for …
Microsoft introduces PEEK, a 1,024-token 'context map' that caches orientation knowledge for LLM agents, cutting redundant reasoning and achieving up to 34% accuracy gains with 93–145 fewer retries and 5.8× cost reduction.
Use context profiler to optimize your LLM calls and reduce token use
ContextSpy is a local proxy tool that profiles how LLM applications use their context window, breaking down token usage by category to help developers optimize and reduce costs.