TokenPilot: Cache-Efficient Context Management for LLM Agents
Summary
TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments, achieving 61-87% cost reduction on benchmarks while maintaining competitive performance.
View Cached Full Text
Cached at: 06/16/26, 11:34 AM
Paper page - TokenPilot: Cache-Efficient Context Management for LLM Agents
Source: https://huggingface.co/papers/2606.17016 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments.
AsLLM agentsare deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimizetoken footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity andprompt cache continuity. To address this, we present TokenPilot, a dual-granularitycontext managementframework. Globally,Ingestion-Aware Compactionacts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally,Lifecycle-Aware Evictionmonitors the ongoingresidual utilityof context segments, enforcing a conservativebatch-turn scheduleto offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated andcontinuous modes demonstrate that TokenPilot reduces costs by 61% and 56% inisolated mode, and 61% and 87% incontinuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.17016
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.17016 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.17016 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.17016 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
@rohanpaul_ai: TokenPilot reduces LLM agent costs via ingestion-aware compaction and lifecycle-aware eviction. Achieves 61–87% cost re…
TokenPilot reduces LLM agent costs via ingestion-aware compaction and lifecycle-aware eviction, achieving 61–87% cost reduction on PinchBench and Claw-Eval with competitive scores.
@omarsar0: // The Efficiency Frontier // Cool paper on context management. As agents reuse the same documents and histories across…
This paper introduces The Efficiency Frontier, a unified framework for cost–performance optimization in LLM context management that models context strategy selection as a deployment-aware optimization problem, achieving 25% reduction in token usage and over 50% lower token cost with amortized memory compression compared to full-context prompting.
built a token-budget-aware context orchestration for long-horizon LLM agents
ContextOS is an open-source token-budget-aware context orchestration layer for long-horizon LLM agents, using hybrid retrieval and deterministic planning to select memories for context with execution traces for inspection.
Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents
This paper evaluates context engineering configurations for LLM agents in enterprise tool-use workflows, showing that summarization with selective pruning achieves 91.6% accuracy while reducing token usage by over 60% compared to full-context baselines.
@bojie_li: If you run LLM agents in production you know the tax. Every turn re-reads the same long context — a system policy, tool…
Researchers introduce Programmable KV Cache, a method for editing and composing KV caches to avoid re-prefilling long contexts during LLM agent inference, achieving 53–398× reduction in p90 time-to-first-token while maintaining decision identity.