Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

# Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
Source: [https://arxiv.org/html/2608.00902](https://arxiv.org/html/2608.00902)
Yujian Liu1Jiabao Ji1∗Li An1∗Rohit Jain2 Gungor Polatkan2Siyu Zhu2Shiyu Chang1 1UC Santa Barbara2LinkedIn \{yujianliu,jiabaoji,li\_an,chang87\}@ucsb\.edu

###### Abstract

LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck\. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline\. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path\. We study online compaction across token eviction \(TE\) and attention matching \(AM\), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat\-prefill, and delayed future\-generation queries\. Experiments onBrowseComp\-PlusandWideSearchshow that immediate compaction often hurts performance, whereas delaying compaction to use the agent’s future queries recovers much of the gap\. Moreover, TE is often more robust than AM under imperfect proxies\. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline\. These results position proxy\-query selection as a core design choice for practical online KV compaction\.

Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

Yujian Liu1††thanks:Equal contribution\. Work done when Yujian, Jiabao, and Li were interning at LinkedIn\.Jiabao Ji1∗Li An1∗Rohit Jain2Gungor Polatkan2Siyu Zhu2Shiyu Chang11UC Santa Barbara2LinkedIn\{yujianliu,jiabaoji,li\_an,chang87\}@ucsb\.edu

## 1Introduction

LLM agents are increasingly used for long\-horizon tasks such as software engineering, deep research, web browsing, and personal assistance\(Anthropic,[2026](https://arxiv.org/html/2608.00902#bib.bib1); OpenAI,[2026](https://arxiv.org/html/2608.00902#bib.bib2); Google,[2026](https://arxiv.org/html/2608.00902#bib.bib3); OpenClaw,[2026](https://arxiv.org/html/2608.00902#bib.bib4)\)\. Their strength comes from maintaining an informative context: the model reasons, calls tools, observes results, and decides what to do next\. This interaction pattern has enabled substantially more capable systems than single\-shot prompting, but it also creates a direct inference bottleneck\. Every generated reasoning step and every tool response is appended to the running context, and the model must retain a key\-value \(KV\) cache for that growing history\. As agent trajectories stretch across many turns, the cache can dominate memory use and increase the cost of each decoding step\.

KV cache compaction is a natural way to reduce this cost\. Instead of storing the full cache, a compaction method replaces a long sequence of cached keys and values with a shorter representation that approximately preserves the attention behavior of the original cache\(Liuet al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib7); Zhanget al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib8)\)\. Much of the work compressing long prompts before answering studies a*static*setting: a complete context is available before generation, the cache is compacted offline, and the resulting compact cache is then consumed by later queries\. In such settings, signals about how the context will be used are often available or can be sampled offline\. For example, query\-aware token eviction methods can use an observation window containing the question or continuation that will read the context\(Liet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib5); Caiet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib9)\)\. Other approaches can further spend additional computation to create training signals before inference, either by generating proxy queries for fitting a compact cache or by generating synthetic conversations and distilling them into a trainable cache\(Zweigeret al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib6); Eyubogluet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib10)\)\.

Agent trajectories, however, differ from this static setting\. An agent does not begin with a fixed document to compress; it gradually constructs its context through interactions with an environment\. Search results, file diffs, and execution traces arrive throughout the trajectory and are incorporated into the live context\. Compaction in this regime is therefore inherently*online*: newly collected information must be compacted before the full future trajectory is known\. Two properties distinguish it from static compaction\.First, future relevance is not yet observable\.When a turn is compacted, the information needed from it several turns later may differ from what that turn or even the immediately following turn would emphasize\. A compaction signal derived only from the current context may therefore poorly represent how the compacted cache will be used later\.Second, the compaction procedure must fit within a reasonable time budget\.Because compaction is performed inside the agent loop, both obtaining a signal about what information to preserve and executing the compaction algorithm contribute directly to the agent’s runtime\. Procedures that rely on additional rollouts, long synthetic continuations, or expensive optimization may be feasible offline but can add substantial latency inside a live agent trajectory\. Together, these properties create an underexplored design space for adapting existing compaction methods: which available signals should guide compaction, and how signal quality and runtime cost interact with the compaction method\.

To study this design space, we present a systematic empirical evaluation of how existing KV cache compaction methods behave when adapted to agent tasks\. We focus on two widely used sequence\-level compaction families, both of which depend on query vectors that represent how the compacted cache will later be read\. We refer to these as*proxy queries*\. Token eviction \(TE\) uses proxy queries to score cached positions by their induced attention mass, then keeps the original KVs at the highest\-scoring positions\(Liet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib5)\)\. Attention matching \(AM\) uses the same proxy\-query\-based key selection, but additionally fits an additive attention bias and reconstructed values so that attention outputs against the compacted cache better match those against the full cache\(Zweigeret al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib6)\)\. We adapt both families to the online agent setting by compacting completed turns as the trajectory unfolds, where proxy queries are derived from signals available at or near the completed turns\. Our main analysis studies the interaction between compaction family and proxy source, including current\-turn boundary queries, lightweight repeat\-prefill queries, and deferred future\-turn queries obtained from the agent’s own subsequent generation\. The resulting experiments map which proxy sources are useful, when deferring compaction helps, and when adding more proxy sources hurts\.

We evaluate onBrowseComp\-Plus\(Chenet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib14)\)andWideSearch\(Wonget al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib39)\), usingQwen3\.5andGemma\-4models at multiple scales\. Experiments show that immediate compaction often degrades task performance, whereas delaying compaction to use queries from the agent’s own subsequent generation consistently recovers much of the gap\. Despite its much simpler selection\-only design, TE is surprisingly robust to imperfect proxies and remains competitive with AM across proxy, delay, and compaction\-budget choices\. OnQwen3\.5\-27BandGemma\-4\-31B, delayed compaction at a ratio of 0\.2 preserves most of the no\-compaction accuracy while reducing peak KV footprint by up to 3\.5×\\timesand 2\.7×\\times, and increasing serving throughput by up to 4\.2×\\timesand 1\.7×\\times, respectively\. Together, these results identify proxy\-query selection as core design choices for practical online KV compaction\.

## 2Related Work

#### Token\-level KV eviction and query\-aware sparsity\.

A large body of work reduces KV memory by retaining only selected cached tokens\. Attention\-based eviction methods exploit concentration, persistence, or structural attention patterns to keep heavy\-hitter, sink, recent, or head\-specific tokens\(Liuet al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib7); Zhanget al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib8); Xiaoet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib18); Geet al\.,[2024a](https://arxiv.org/html/2608.00902#bib.bib19)\)\. Query\- or layer\-aware methods further score historical tokens from an observation window or allocate budgets across layers\(Liet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib5); Tanget al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib20); Yanget al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib21); Caiet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib9)\)\. Expected attention similarly estimates importance from a future\-query distribution when future attention is unavailable\(Devotoet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib22)\)\. These methods show that query\-conditioned signals are useful for KV reduction\. In agent trajectories, however, future queries are endogenous to the model’s own later actions\. A compaction decision made after one tool interaction must therefore preserve information for future states of the trajectory that do not yet exist\.

#### Learned and optimized compact representations\.

Other work replaces token selection with learned or optimized compact states\. Prompt\- and context\-compression methods summarize contexts into soft tokens, recurrent states, or compressed activation slots\(Muet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib23); Chevalieret al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib24); Geet al\.,[2024b](https://arxiv.org/html/2608.00902#bib.bib25); Zhanget al\.,[2024a](https://arxiv.org/html/2608.00902#bib.bib26); Chuet al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib36)\)\. Cartridges trains compact KV representations through offline self\-study, requiring synthetic conversations and gradient\-based optimization\(Eyubogluet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib10)\)\. Attention matching fits a compact cache to match full\-cache attention outputs on proxy queries\(Zweigeret al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib6)\)\. We use attention matching as one compaction family in our study\. These methods are usually static: the context, task distribution, or proxy queries can be prepared before deployment\. We instead study compaction when proxy queries must be obtained cheaply during an ongoing agent trajectory\.

#### Multi\-turn and agent KV management\.

Several recent works study KV reuse or compression in multi\-turn settings\. KVzip motivates query\-agnostic compression for caches that may be reused by many future queries, andSCBenchshows that sublinear\-memory methods can degrade under multi\-turn KV\-cache lifecycles\(Kimet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib13); Liet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib27)\)\. Serving systems improve multi\-turn or agent efficiency by reusing, scheduling, or retaining KV states across conversation and tool\-execution boundaries\(Gaoet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib28); Liet al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib29)\)\. These works are complementary to ours\. They treat the growing interaction history as a systems object to cache, reuse, or schedule, whereas we ask which information inside each newly completed agent turn should be preserved when that turn is compacted and then frozen\.SCBench, for example, studies shared\-context settings in which multiple requests query a long context that is already available\. In our setting, the context is produced online: search results, file contents, and other observations are appended only after the agent issues the corresponding actions\. This makes online compaction a decision about an evolving trajectory, not only reuse of a fixed context\.

#### Orthogonal KV\-compression dimensions\.

KV memory can also be reduced along axes orthogonal to sequence length: quantization lowers key/value precision\(Zirui Liuet al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib30); Hooperet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib31); Kanget al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib32)\); low\-rank methods compress or offload KV states through lower\-dimensional representations\(Zhanget al\.,[2024b](https://arxiv.org/html/2608.00902#bib.bib33); Sunet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib34); Xuet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib35)\); and architectural methods use grouped or latent KV representations\(Ainslieet al\.,[2023](https://arxiv.org/html/2608.00902#bib.bib37); DeepSeek\-AIet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib38)\)\. These approaches address a different axis of the memory problem and can be combined with compaction on sequence length\.

## 3Design Space of Online KV Compaction

We first describe the online KV compaction problem\. The goal is not to propose a new algorithm or a single final recipe, but to isolate which choices matter when a compactor is placed inside an agent loop\. We begin by formalizing the two compaction families studied throughout the paper, then describe how we adapt them from a static prompt setting to online agent trajectories\.

### 3\.1Static KV Compaction

Consider one attention layer and one KV head\. We omit the usual1/d1/\\sqrt\{d\}attention scale for notational simplicity\. Let𝐊,𝐕∈ℝn×d\\mathbf\{K\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{n\\times d\}denote the full key and value cache for a context of lengthnn, and let𝐐∈ℝq×d\\mathbf\{Q\}\\in\\mathbb\{R\}^\{q\\times d\}denote a set ofqqproxy queries\. The full\-cache attention output on these queries is

𝐀​\(𝐐;𝐊,𝐕\)=softmax​\(𝐐𝐊⊤\)​𝐕\.\\mathbf\{A\}\(\\mathbf\{Q\};\\mathbf\{K\},\\mathbf\{V\}\)=\\mathrm\{softmax\}\\\!\\left\(\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\\right\)\\mathbf\{V\}\.\(1\)A compactor constructs a shorter cache of lengthm≪nm\\ll nwhose attention outputs approximate𝐀​\(𝐐;𝐊,𝐕\)\\mathbf\{A\}\(\\mathbf\{Q\};\\mathbf\{K\},\\mathbf\{V\}\)\. We focus on TE and AM because they are widely used and, crucially for the online setting, their compaction steps can be executed efficiently once the proxy queries are available\. The left side of Figure[1](https://arxiv.org/html/2608.00902#S3.F1)illustrates these two families\.

![Refer to caption](https://arxiv.org/html/2608.00902v1/x1.png)Figure 1:Overview of the online KV compaction design space\.Left: TE and AM both score cached tokens using proxy queries\. TE stores the selected original keys and values, while AM keeps the selected keys and optimizes the attention bias and compacted values\.Right: current\-turn proxies are available when the turn finishes, whereas future\-turn proxies delay compaction untilkklater turns provide additional queries\.#### Token eviction\.

Token eviction \(TE\)\(Liet al\.,[2024](https://arxiv.org/html/2608.00902#bib.bib5)\)keeps original KV entries at selected positions\. The selection is driven by attention scores induced by the proxy queries\. Let

αi​j=\[softmax​\(𝐐𝐊⊤\)\]i​j\\alpha\_\{ij\}=\\left\[\\mathrm\{softmax\}\\\!\\left\(\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\\right\)\\right\]\_\{ij\}\(2\)be the attention weight from proxy queryiito cached positionjj\. We score each cached position by its root\-mean\-square attention mass,

sj​\(𝐐,𝐊\)=\(1q​∑i=1qαi​j2\)1/2\.s\_\{j\}\(\\mathbf\{Q\},\\mathbf\{K\}\)=\\left\(\\frac\{1\}\{q\}\\sum\_\{i=1\}^\{q\}\\alpha\_\{ij\}^\{2\}\\right\)^\{1/2\}\.\(3\)TE selects the highest\-scoring positions and stores the corresponding original keys and values:

S=Top​\-​m​\{sj\}j=1n,𝐊C=𝐊S,𝐕C=𝐕S,S=\\mathrm\{Top\}\\text\{\-\}m\\\{s\_\{j\}\\\}\_\{j=1\}^\{n\},\\quad\\mathbf\{K\}\_\{C\}=\\mathbf\{K\}\_\{S\},\\quad\\mathbf\{V\}\_\{C\}=\\mathbf\{V\}\_\{S\},\(4\)where𝐊C,𝐕C∈ℝm×d\\mathbf\{K\}\_\{C\},\\mathbf\{V\}\_\{C\}\\in\\mathbb\{R\}^\{m\\times d\}\. TE is thus a pure selection method: after selectingSS, the stored keys and values are unchanged\.

#### Attention matching\.

Attention matching \(AM\)\(Zweigeret al\.,[2026](https://arxiv.org/html/2608.00902#bib.bib6)\)uses the same proxy\-query\-based key selection as TE, setting𝐊C=𝐊S\\mathbf\{K\}\_\{C\}=\\mathbf\{K\}\_\{S\}\. It then fits an additive bias𝜷∈ℝm\\boldsymbol\{\\beta\}\\in\\mathbb\{R\}^\{m\}and compact values𝐕C∈ℝm×d\\mathbf\{V\}\_\{C\}\\in\\mathbb\{R\}^\{m\\times d\}\. It consists of two stages\. First,𝜷\\boldsymbol\{\\beta\}is chosen so that the selected keys account for the full cache’s unnormalized attention mass\. Let𝟏m∈ℝm\\mathbf\{1\}\_\{m\}\\in\\mathbb\{R\}^\{m\}and𝟏n∈ℝn\\mathbf\{1\}\_\{n\}\\in\\mathbb\{R\}^\{n\}be all\-ones vectors:

min𝜷⁡‖exp⁡\(𝐐𝐊C⊤\+𝜷\)​𝟏m−exp⁡\(𝐐𝐊⊤\)​𝟏n‖22,\\min\_\{\\boldsymbol\{\\beta\}\}\\left\\\|\\exp\(\\mathbf\{Q\}\\mathbf\{K\}\_\{C\}^\{\\top\}\+\\boldsymbol\{\\beta\}\)\\mathbf\{1\}\_\{m\}\-\\exp\(\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\)\\mathbf\{1\}\_\{n\}\\right\\\|\_\{2\}^\{2\},\(5\)where𝜷\\boldsymbol\{\\beta\}is broadcast across queries\. Then, holding𝜷\\boldsymbol\{\\beta\}fixed, AM fits values to match the full\-cache attention outputs\. Define

𝐀^​\(𝐐;𝐊C,𝜷,𝐕C\)≜softmax​\(𝐐𝐊C⊤\+𝜷\)​𝐕C,\\widehat\{\\mathbf\{A\}\}\(\\mathbf\{Q\};\\mathbf\{K\}\_\{C\},\\boldsymbol\{\\beta\},\\mathbf\{V\}\_\{C\}\)\\triangleq\\mathrm\{softmax\}\\\!\\left\(\\mathbf\{Q\}\\mathbf\{K\}\_\{C\}^\{\\top\}\+\\boldsymbol\{\\beta\}\\right\)\\mathbf\{V\}\_\{C\},\(6\)AM solves

min𝐕C⁡‖𝐀^​\(𝐐;𝐊C,𝜷,𝐕C\)−𝐀​\(𝐐;𝐊,𝐕\)‖F2\.\\min\_\{\\mathbf\{V\}\_\{C\}\}\\left\\\|\\widehat\{\\mathbf\{A\}\}\(\\mathbf\{Q\};\\mathbf\{K\}\_\{C\},\\boldsymbol\{\\beta\},\\mathbf\{V\}\_\{C\}\)\-\\mathbf\{A\}\(\\mathbf\{Q\};\\mathbf\{K\},\\mathbf\{V\}\)\\right\\\|\_\{F\}^\{2\}\.\(7\)The bias𝜷\\boldsymbol\{\\beta\}changes how much attention each selected key receives, while𝐕C\\mathbf\{V\}\_\{C\}is allowed to differ from the original selected values\. Thus TE commits to the original KV, whereas AM keeps the selected keys but fits the attention bias and values to match full\-cache outputs on the proxy queries\.

This formulation is applied independently to each layer and KV head\. In our experiments, bothQwen3\.5\(Team,[2026](https://arxiv.org/html/2608.00902#bib.bib11)\)andGemma\-4\(Google DeepMind,[2026](https://arxiv.org/html/2608.00902#bib.bib12)\)are hybrid\-attention models, so we compact only their full\-attention layers and leave non\-full\-attention states unchanged; implementation details are given in Appendix[A](https://arxiv.org/html/2608.00902#A1)\.

### 3\.2Online Compaction for Agent Trajectories

In an agent trajectory, the context is not fixed before generation\. The model starts from a prefix containing the system prompt, tool definitions, and user query\. It then repeatedly generates assistant messages, emits tool calls, receives tool responses, and continues from the expanded context\. We segment this history into turns:

P,T1,T2,…,Tt,…P,\\ T\_\{1\},\\ T\_\{2\},\\ldots,T\_\{t\},\\ldotswherePPis the uncompacted prefix and eachTtT\_\{t\}contains the assistant\-side generation for that step and the resulting tool response\. Online compaction compresses completed turns as they arrive\. Once turnTtT\_\{t\}is compacted, future generations attend to the compacted representation and discard the original KV; the compacted turn is then frozen and is not re\-optimized or re\-compacted later\.

This setup changes the role of proxy queries\. In the static setting, proxy queries can be drawn from known questions or offline synthetic data\. In the online setting, the future queries that will readTtT\_\{t\}are not yet available whenTtT\_\{t\}finishes\. We therefore study two cheap proxy families that do not require extra rollouts, as summarized in Figure[1](https://arxiv.org/html/2608.00902#S3.F1)right\.

#### Current\-turn proxies\.

These proxies are available immediately whenTtT\_\{t\}completes, so they allow immediate compaction\. We consider two ways to obtain such proxies\. ❶Boundary queriesuse the model’s query vectors at structural closing or transition tokens from the actual trajectory\. For example, forQwen3\.5, we extract query vectors from the closing token<\|im\_end\|\>\. These queries are cheap and require no extra forward pass, but they may not reflect what future turns will need fromTtT\_\{t\}\. ❷Repeat\-prefill queriesare inspired by query\-agnostic KV compaction in KVzip\(Kimet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib13)\)\. We append a reconstruction prompt after the previous turn’s content and teacher\-force the model to repeat the completed turn:

> <previous context\> Let me repeat the previous thinking, tool call, and tool response\. <repeated context\>

We extract proxy queries only from the teacher\-forced repeated content\. The repeated thinking and tool\-call text provides proxies for compacting the assistant generation, while the repeated tool response provides proxies for compacting the tool\-response segment\.

#### Future\-turn proxies\.

The second proxy family delays compaction in order to use real future queries\. To compactTtT\_\{t\}with a delay ofkkturns, the system keepsTtT\_\{t\}in raw form while generating turnsTt\+1,…,Tt\+kT\_\{t\+1\},\\ldots,T\_\{t\+k\}\. During these generations, the model attends to the uncompressed prefixPP, earlier frozen compacted turns, the raw cache forTtT\_\{t\}, and any newer turns that are still inside the delay window\. By default, we record the query vectors produced during the assistant generations inTt\+1,…,Tt\+kT\_\{t\+1\},\\ldots,T\_\{t\+k\}and use them as proxies when compactingTtT\_\{t\}afterTt\+kT\_\{t\+k\}finishes\. We also ablate adding the query vectors from the corresponding tool\-response prefill tokens as an additional proxy source\. From turnTt\+k\+1T\_\{t\+k\+1\}onward,TtT\_\{t\}is read only through its frozen compacted representation\. Thus a larger delay provides proxy queries that are closer to the way later computation actually readsTtT\_\{t\}, but it also postpones the memory and compute savings because more raw turns must remain in the cache\.

## 4What Matters for Online KV Compaction

### 4\.1Setup

We evaluate on two complementary agentic\-search benchmarks\.BrowseComp\-Plus\(Chenet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib14)\)is a fixed\-corpus benchmark for deep\-search agents: each example asks a difficult information\-seeking question with a verifiable final answer, and solving it requires iteratively retrieving evidence from the corpus rather than relying on parametric knowledge\.WideSearch\(Wonget al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib39)\)instead targets broad information seeking\. It requires the agent to collect many atomic facts and organize them into a well\-structured table, so completeness matters as much as correctness\.

In both settings the agent starts from the system prompt and user question, then repeatedly generates tool calls, observes tool outputs, and either continues searching or returns its final answer\. ForBrowseComp\-Plus, the agent queries a local FAISS search server withQwen3\-Embedding\-4B\(Zhanget al\.,[2025b](https://arxiv.org/html/2608.00902#bib.bib15)\)retrieval\. ForWideSearch, the agent searches the open internet through Bing Web Search API\. We study two model families:Qwen3\.5andGemma\-4, both of which use hybrid attention\. In this section, we report performance of the 4B models and defer results of larger models to Section[5](https://arxiv.org/html/2608.00902#S5)\. We compact assistant generation and tool response separately, while preserving structural boundary tokens needed by the chat template\. We score both benchmarks with aQwen3\.5\-397Bjudge\. ForBrowseComp\-Pluswe report answer accuracy under the official grader template; forWideSearchwe report the item\-level F1 for the final table, which credits partially collected atomic facts\.

### 4\.2Proxy\-Query Ablation

We first fix the compaction ratio to 0\.2, reducing each compactable segment to 20% of its original length, and compare how different proxy sources affect performance\. This section focuses on task performance; we study runtime savings in Section[5\.1](https://arxiv.org/html/2608.00902#S5.SS1)\. Table[1](https://arxiv.org/html/2608.00902#S4.T1)organizes the results into current\-turn proxies and one\-turn\-delayed future proxies\. Corresponding 95% bootstrap confidence intervals are reported in Appendix Table[6](https://arxiv.org/html/2608.00902#A1.T6)\.

Proxy sourceDelayBrowseComp\-PlusWideSearchQwen3\.5\-4BGemma\-4\-E4BQwen3\.5\-4BGemma\-4\-E4BAcc\.TurnsAcc\.TurnsF1TurnsF1TurnsNo compaction–46\.002033\.001344\.552431\.949Current\-turn proxiesAMRepeat\-prefill031\.253111\.25930\.952813\.638TERepeat\-prefill032\.75519\.001224\.568217\.1911TEBoundary045\.254021\.501224\.295015\.7311One\-turn\-delayed future proxiesAMAssistant generation139\.753327\.501039\.593320\.179AM\+ repeat\-prefill143\.253024\.251038\.832921\.499AM\+ repeat\-prefill \+ tool response143\.253027\.50937\.392920\.179TEAssistant generation144\.003827\.501037\.343925\.9011TE\+ boundary143\.003727\.001138\.093526\.5311TE\+ boundary \+ tool response142\.253926\.751139\.143525\.3910

Table 1:Proxy\-source ablation at compaction ratio 0\.2 onBrowseComp\-PlusandWideSearch\. ForBrowseComp\-Plus, “Acc\.” is final answer accuracy; forWideSearch, “F1” is the mean item\-level table F1\. “Turns” is the median number of agent turns\.#### Which current\-turn proxy should we use?

We compare repeat\-prefill and boundary queries for TE, and repeat\-prefill for AM\. Boundary queries are not evaluated for AM because AM fits bias and value parameters against proxy targets; a single boundary query provides a poorly constrained optimization target\. For TE, the stronger current\-turn proxy depends on the benchmark\. OnBrowseComp\-Plus, replacing repeat\-prefill with boundary queries improvesQwen3\.5\-4Bfrom 32\.75% to 45\.25% andGemma\-4\-E4Bfrom 9\.00% to 21\.50%\. OnWideSearch, however, the two proxies perform similarly\. Boundary queries can therefore act as useful aggregation points, but their advantage is not universal\. Their strong performance onBrowseComp\-Plusis also consistent with recent work that uses attention to an end\-of\-thinking token to identify important reasoning tokens, as well as analyses showing that punctuation and other structural tokens can form semantically meaningful attention sinks\(Choiet al\.,[2025](https://arxiv.org/html/2608.00902#bib.bib16); Zhanget al\.,[2025a](https://arxiv.org/html/2608.00902#bib.bib17)\)\.

#### Does delaying compaction help?

Future\-turn proxies use actual assistant\-generation queries from the next turn, so they are closer to the way the compacted turn will be read later\. Across all four model and benchmark pairs, using one\-turn\-delayed assistant\-generation queries improves over immediate repeat\-prefill for both AM and TE\. For TE, delayed queries also outperform boundary queries in three of the four pairs\. The only exception isQwen3\.5\-4BonBrowseComp\-Plus, where immediate boundary queries reach 45\.25% accuracy, slightly above the 44\.00% obtained with delayed queries\. The consistency of the results shows that even a one\-turn delay generally provides a more informative compaction signal than proxies constructed from the current turn alone\.

#### Should we combine proxy sources?

We next add the best available current\-turn proxy to the future\-turn proxy: repeat\-prefill for AM and boundary queries for TE\. We then add future tool\-response queries as a third source\. The combination rule differs by compaction family\. For TE, each proxy source selects a fixed portion of the token budget\. For example, with two sources, half the selected tokens come from future\-turn queries and half from boundary queries\. For AM, we downsample each additional proxy source to the same length as the future turn source, then concatenate all sources as the optimization target\. The results show that more proxy sources do not automatically help\. For TE, all three configurations are within 2 points of one another for every model and benchmark pair, suggesting TE’s robustness to variations in proxy source, so long as the future assistant generation is available\. Overall, the configurations perform comparably, and adding current\-turn or tool\-response queries provides no reliable gain\.

#### How does compaction change agent behavior?

Table[1](https://arxiv.org/html/2608.00902#S4.T1)also shows that compaction changes the agent’s behavior, not only its final performance\. Across both benchmarks,Qwen3\.5\-4Btends to lengthen its trajectories under compaction\. Its no\-compaction baselines use medians of 20 turns onBrowseComp\-Plusand 24 turns onWideSearch, whereas the delayed configurations use 30\-39 and 29\-39 turns, respectively\. In contrast,Gemma\-4\-E4Bremains close to its no\-compaction trajectory length on both benchmarks\. The longerQwen3\.5\-4Btrajectories suggest that the model may compensate for weakened context by issuing additional searches and recovering missing evidence through the environment\. We examine this hypothesis directly in Section[5\.2](https://arxiv.org/html/2608.00902#S5.SS2)\. This cross\-model difference reinforces that online compaction should be evaluated as an agent\-level intervention, not only as an attention\-approximation problem\.

![Refer to caption](https://arxiv.org/html/2608.00902v1/x2.png)Figure 2:Sensitivity to compaction budget and delay window\. The top row varies the delay window over\{1,3,5\}\\\{1,3,5\\\}turns at compaction ratio0\.20\.2; the bottom row varies the compaction ratio over\{0\.1,0\.2,0\.4\}\\\{0\.1,0\.2,0\.4\\\}using the stronger short\-delay setting for each model and method pair\. Each panel fixes one model \(Qwen3\.5\-4BorGemma\-4\-E4B\) and benchmark \(BrowseComp\-PlusorWideSearch\)\. Dashed lines mark the no\-compaction baselines\.

### 4\.3Sensitivity to Compaction Budget and Delay

The previous ablation fixes the compaction ratio and uses at most a one\-turn delay\. We next characterize how performance varies with the memory budget and delay window\. For each model–benchmark–method combination, we select the best\-performing proxy strategy from Table[1](https://arxiv.org/html/2608.00902#S4.T1)and sweep either the compaction ratio or delay\. Figure[2](https://arxiv.org/html/2608.00902#S4.F2)shows the results\. For the delay, we fix the compaction ratio to 0\.2 and vary the delay over 1, 3, and 5 turns\. For the compaction\-ratio, we vary the ratio over 0\.1, 0\.2, and 0\.4, using the stronger short\-delay setting between immediate compaction and a one\-turn delay for each combination\.

The delay sweep shows that additional future context helps, although the gains are not always monotonic\. For all combinations, longer delays \(3 or 5 turns\) outperform the one\-turn delay\. OnBrowseComp\-PluswithQwen, both methods benefit most from a five\-turn delay, slightly exceeding the no\-compaction baseline\. In the other settings, gains tend to peak or flatten earlier\. Thus, observing more of the agent’s future queries is generally useful, but the best delay depends on the model, benchmark, and compaction method\.

The compaction\-ratio sweep shows a clearer overall relationship with memory budget\. Increasing the ratio from 0\.1 to 0\.2 improves every model–benchmark–method combination, confirming that a ratio of 0\.1 is too aggressive in these settings\. Increasing the ratio further to 0\.4 improves six of the eight combinations\. In particular, performance increases monotonically with budget onWideSearch\. The exceptions occur onBrowseComp\-Plus, whereQwen3\.5\-4BTE andGemma\-4\-E4BAM both peak at ratio 0\.2\. Thus, a larger compact cache usually improves performance, particularly for broad search\.

Table 2:Task performance and simulated serving efficiency on larger models\. Accuracy, average turns, and trajectory lengths come from batch\-size\-1 runs onBrowseComp\-Plus\. Peak KV is the maximum per\-turn KV length\. Serving batch is the batch used for the throughput simulation\. All compaction settings use one\-turn\-delayed assistant\-generation queries and a compaction ratio of 0\.2\.#### Takeaways\.

Across these experiments, current\-turn proxy choice is benchmark\-dependent, and combining proxy sources provides no reliable gain\. In contrast, delayed future\-turn queries improve performance consistently, although the best delay varies across settings\. Increasing the memory budget also usually helps, especially onWideSearch, but a larger compact cache is not uniformly better\. Finally, AM’s additional optimization is not automatically beneficial under online constraints\. Despite being much simpler, TE remains surprisingly competitive across proxy, delay, and budget choices, suggesting that AM’s richer optimization provides limited benefit when the available proxy queries imperfectly represent how the cache will later be used\.

## 5Additional Analyses

The study in Section[4](https://arxiv.org/html/2608.00902#S4)focuses on answer accuracy and identifies strategies that preserve most of the no\-compaction performance on smaller models\. We next study two complementary questions\. First, do these strategies preserve task performance and improve serving efficiency on larger models? Second, how does compaction change agent behavior beyond final answer accuracy?

### 5\.1Task Performance and Serving Efficiency on Larger Models

We extend our evaluation toQwen3\.5\-27BandGemma\-4\-31B, substantially larger models from both architectural families\. We evaluate onBrowseComp\-Plusat a compaction ratio of 0\.2\. We compare no compaction with AM and TE using one\-turn\-delayed assistant\-generation queries in this section\. The complete proxy\-source ablation results, including 95% confidence intervals, are reported in Appendix Table[7](https://arxiv.org/html/2608.00902#A1.T7), which demonstrate a similar trend as the 4B models\.

Table[2](https://arxiv.org/html/2608.00902#S4.T2)reports both task performance and serving efficiency\. We generate agent trajectories at batch size 1 using our Hugging Face evaluation code and use these runs to measure answer accuracy and trajectory length\. To estimate how the same workloads would execute on an optimized inference engine, we extract the KV length and number of generated tokens at every turn and replay these trajectory shapes using decode latencies measured with SGLangZhenget al\.\([2024](https://arxiv.org/html/2608.00902#bib.bib40)\)\. This connects the task outcomes produced by our evaluation code to the memory and latency characteristics of a production\-oriented serving stack\. We use each method’s peak KV footprint to determine the largest admissible batch and report the resulting decode throughput\. Appendix[A\.7](https://arxiv.org/html/2608.00902#A1.SS7)provides the complete simulation setup\.

Table[2](https://arxiv.org/html/2608.00902#S4.T2)shows that both methods retain most of the no\-compaction task performance at larger scale\. OnQwen3\.5\-27B, no compaction obtains 52\.50% accuracy, compared with 51\.00% for AM and 52\.00% for TE\. OnGemma\-4\-31B, both methods slightly exceed the baseline, which partly reflects the fact that some no\-compaction trajectories run out of memory even at batch size 1, whereas compaction keeps their KV caches within the limit\.

ForQwen3\.5\-27B, compaction reduces peak KV from 612\.9K tokens to about 175K tokens\. This raises the guaranteed serving batch from 8 to 32\. As a result, aggregate throughput increases from 217 queries/hour without compaction to 918 with AM and 717 with TE, corresponding to 4\.2×\\timesand 3\.3×\\timesimprovements\. OnGemma\-4\-31B, peak KV falls from 272\.1K tokens to 99\.6K with AM and 121\.3K with TE\. The serving batch increases from 8 to 16, raising throughput by 1\.7×\\timesand 1\.5×\\times, respectively\. The gains are smaller than forQwen3\.5\-27B, but the same pattern holds across both models: compaction reduces the memory footprint enough to support more concurrent requests\. Thus, even though the compacted agents take more turns on average, the increase in concurrency offsets the additional decode work, yielding higher overall throughput in queries per hour\.

Table 3:Search\-query repetition within agent trajectories\. “NN Sim\.” is the average nearest\-neighbor similarity to earlier search queries in the same trajectory\. “Duplicate” is the percentage of search queries whose nearest earlier query has similarity above 0\.9\.
### 5\.2Behavioral Effects of Compaction

Section[4](https://arxiv.org/html/2608.00902#S4)shows that compaction changes the agent’s trajectory turns forQwen3\.5\-4B\. One hypothesis is that the model compensates for weakened context by issuing additional searches, repeating similar queries to recover information that was previously retrieved\. We test this from the search queries generated in each trajectory\.

For each search query, we compute its maximum cosine similarity to earlier search queries in the same trajectory using the same embedding model as the retrieval server\. We report the mean nearest\-neighbor similarity and the fraction of queries whose similarity to a previous query exceeds 0\.9\. Table[3](https://arxiv.org/html/2608.00902#S5.T3)shows that compaction increases both measures\. These results show that the accuracy of compacted agents can partly come from behavioral adaptation\. In particular, the compaction preserves answer accuracy but also induces more repeated search, suggesting that the agent uses the environment to recover information weakened by compaction\. Thus, online KV compaction should be evaluated not only by model\-side memory savings and final accuracy, but also by how it changes the agent’s interaction pattern with the environment\.

## 6Conclusion

We study online KV\-cache compaction for LLM agents, where context is accumulated through interaction and future relevance is unknown at compaction time\. Across token eviction and attention matching, proxy choice is central: boundary tokens provide a strong immediate signal, while delayed future\-generation queries can further improve compaction\. Our results also show that compaction changes agent behavior, so practical memory systems should consider not only compression and accuracy, but also downstream trajectories\.

## Limitations

Our study has several limitations\. First, we focus onBrowseComp\-PlusandWideSearch, where agents gather evidence through search and document\-reading tools\. This setting captures long\-horizon information gathering, but does not cover other agent workloads such as code editing, GUI control, or state\-changing web tasks, where the structure of future relevance may differ\.

Second, we intentionally restrict the study to cheap online proxy sources for TE and AM\. More expensive approaches, including additional rollouts, learned compressors, or gradient\-based optimization, may produce stronger compact caches under larger latency budgets\. These methods are complementary to our low\-overhead online setting\.

Third, we evaluate fixed compaction ratios and delay windows rather than adaptive policies\. A deployed agent system could choose when and how aggressively to compact based on memory pressure, turn length, tool type, or uncertainty\. Future work can explore such adaptive controllers, while our results identify strong proxy\-query primitives and the tradeoffs they should account for\.

## Potential Risks and Use of Artifacts

Our work studies online KV\-cache compaction for LLM agents, with the goal of reducing inference cost while preserving agent accuracy\. The main risk is that compaction can remove or weaken information that the agent later needs\. In deployment, this may lead to incorrect final answers, redundant tool use, or overconfident responses based on incomplete context\. Our experiments therefore evaluate both task accuracy and agent behavior, and the proposed methods should be used with appropriate validation in high\-stakes applications\.

We use existing public research artifacts consistently with their intended purposes\. The evaluation data comes fromBrowseComp\-PlusandWideSearch, which are released on Hugging Face\. The evaluated models,Qwen3\.5andGemma\-4, are released under Apache\-2\.0 licenses, as is theQwen3\-Embedding\-4Bretrieval model\. Our implementation builds on open\-source software including Hugging Face Transformers, released under Apache\-2\.0, and FAISS, released under the MIT license\.

## AI Usage Disclosure

We used AI assistants to help debug code and adapt open\-source implementations for our experiments\. We also used AI assistants to polish the paper writing\. All technical decisions, experimental designs, and analysis were made by the authors\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.External Links:2305\.13245Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- Anthropic \(2026\)Note:Accessed: 2026\-05\-24External Links:[Link](https://www.anthropic.com/news/claude-opus-4-7)Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic kv cache compression based on pyramidal information funneling\.External Links:2406\.02069Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su, S\. Sharifymoghaddam, Y\. Li, H\. Hong, X\. Shi, X\. Liu, N\. Thakur, C\. Zhang, L\. Gao, W\. Chen, and J\. Lin \(2025\)BrowseComp\-plus: a more fair and transparent evaluation benchmark of deep\-research agent\.External Links:2508\.06600Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p5.4),[§4\.1](https://arxiv.org/html/2608.00902#S4.SS1.p1.1)\.
- A\. Chevalier, A\. Wettig, A\. Ajith, and D\. Chen \(2023\)Adapting language models to compress contexts\.External Links:2305\.14788Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Choi, J\. Lee, J\. Tack, W\. Song, S\. Dingliwal, S\. M\. Jayanthi, B\. Ganesh, J\. Shin, A\. Galstyan, and S\. B\. Bodapati \(2025\)Think clearly: improving reasoning via redundant token pruning\.External Links:2507\.08806Cited by:[§4\.2](https://arxiv.org/html/2608.00902#S4.SS2.SSS0.Px1.p1.1)\.
- C\. Chu, G\. Zhou, G\. Zhang, H\. Li, H\. Peng, H\. Cheng, J\. Liang, J\. Cao, K\. Gai, L\. Zhou, L\. Ren, Q\. Zhang, R\. Tang, R\. Wang, X\. Luo, Y\. Su, Z\. Liang, Z\. Wang, B\. Ding, C\. Song, D\. Zang, H\. Wang, J\. Ou, J\. Deng, J\. Shi, J\. Zhang, J\. Chen, L\. Ren, M\. Lv, Q\. Wang, Q\. Hu, S\. Wang, S\. Mao, T\. Wang, X\. Wang, Z\. Ling, Z\. Li, and Z\. Zhang \(2026\)Kwai summary attention technical report\.External Links:2604\.24432Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Yang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Chen, J\. Yuan, J\. Qiu, J\. Song, K\. Dong, K\. Gao, K\. Guan, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Zhu, Q\. Chen, Q\. Du, R\. J\. Chen, R\. L\. Jin, R\. Ge, R\. Pan, R\. Xu, R\. Chen, S\. S\. Li, S\. Lu, S\. Zhou, S\. Chen, S\. Wu, S\. Ye, S\. Ma, S\. Wang, S\. Zhou, S\. Yu, S\. Zhou, S\. Zheng, T\. Wang, T\. Pei, T\. Yuan, T\. Sun, W\. L\. Xiao, W\. Zeng, W\. An, W\. Liu, W\. Liang, W\. Gao, W\. Zhang, X\. Q\. Li, X\. Jin, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Shen, X\. Chen, X\. Chen, X\. Nie, X\. Sun, X\. Wang, X\. Liu, X\. Xie, X\. Yu, X\. Song, X\. Zhou, X\. Yang, X\. Lu, X\. Su, Y\. Wu, Y\. K\. Li, Y\. X\. Wei, Y\. X\. Zhu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Li, Y\. Wang, Y\. Zheng, Y\. Zhang, Y\. Xiong, Y\. Zhao, Y\. He, Y\. Tang, Y\. Piao, Y\. Dong, Y\. Tan, Y\. Liu, Y\. Wang, Y\. Guo, Y\. Zhu, Y\. Wang, Y\. Zou, Y\. Zha, Y\. Ma, Y\. Yan, Y\. You, Y\. Liu, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Huang, Z\. Zhang, Z\. Xie, Z\. Hao, Z\. Shao, Z\. Wen, Z\. Xu, Z\. Zhang, Z\. Li, Z\. Wang, Z\. Gu, Z\. Li, and Z\. Xie \(2024\)DeepSeek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.External Links:2405\.04434Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Devoto, M\. Jeblick, and S\. Jégou \(2025\)Expected attention: kv cache compression by estimating attention from future queries distribution\.External Links:2510\.00636Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Eyuboglu, R\. Ehrlich, S\. Arora, N\. Guha, D\. Zinsley, E\. Liu, W\. Tennien, A\. Rudra, J\. Zou, A\. Mirhoseini, and C\. Re \(2025\)Cartridges: lightweight and general\-purpose long context representations via self\-study\.External Links:2506\.06266Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Gao, Z\. He, P\. Sharma, Q\. Kang, D\. Jevdjic, J\. Deng, X\. Yang, Z\. Yu, and P\. Zuo \(2024\)Cost\-efficient large language model serving for multi\-turn conversations with cachedattention\.External Links:2403\.19708Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024a\)Model tells you what to discard: adaptive kv cache compression for llms\.External Links:2310\.01801Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Ge, J\. Hu, L\. Wang, X\. Wang, S\. Chen, and F\. Wei \(2024b\)In\-context autoencoder for context compression in a large language model\.External Links:2307\.06945Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- Google DeepMind \(2026\)Gemma 4 model card\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Accessed: 2026\-05\-24Cited by:[§3\.1](https://arxiv.org/html/2608.00902#S3.SS1.SSS0.Px2.p2.1)\.
- Google \(2026\)Note:Accessed: 2026\-05\-24External Links:[Link](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/)Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p1.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2025\)KVQuant: towards 10 million context length llm inference with kv cache quantization\.External Links:2401\.18079Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient kv cache compression recipe for near\-lossless generative inference of llm\.External Links:2403\.05527Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Kim, J\. Kim, S\. Kwon, J\. W\. Lee, S\. Yun, and H\. O\. Song \(2025\)KVzip: query\-agnostic kv cache compression with context reconstruction\.External Links:2505\.23416Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.00902#S3.SS2.SSS0.Px1.p1.2)\.
- H\. Li, R\. He, Q\. Mang, Q\. Zhang, H\. Mao, X\. Chen, H\. Zhou, A\. Cheung, J\. Gonzalez, and I\. Stoica \(2026\)Continuum: efficient and robust multi\-turn llm agent scheduling with kv cache time\-to\-live\.External Links:2511\.02230Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, H\. Jiang, Q\. Wu, X\. Luo, S\. Ahn, C\. Zhang, A\. H\. Abdi, D\. Li, J\. Gao, Y\. Yang, and L\. Qiu \(2025\)SCBench: a kv cache\-centric analysis of long\-context methods\.External Links:2412\.10319Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.External Links:2404\.14469Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§1](https://arxiv.org/html/2608.00902#S1.p4.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.00902#S3.SS1.SSS0.Px1.p1.5)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.External Links:2305\.17118Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Mu, X\. L\. Li, and N\. Goodman \(2024\)Learning to compress prompts with gist tokens\.External Links:2304\.08467Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026\)Note:Accessed: 2026\-05\-24External Links:[Link](https://openai.com/index/introducing-gpt-5-5/)Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p1.1)\.
- OpenClaw \(2026\)OpenClaw — personal ai assistant\.Note:[https://github\.com/openclaw/openclaw](https://github.com/openclaw/openclaw)GitHub repository, accessed 2026\-05\-24Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p1.1)\.
- H\. Sun, L\. Chang, W\. Bao, S\. Zheng, N\. Zheng, X\. Liu, H\. Dong, Y\. Chi, and B\. Chen \(2025\)ShadowKV: kv cache in shadows for high\-throughput long\-context llm inference\.External Links:2410\.21465Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context llm inference\.External Links:2406\.10774Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Team \(2026\)Qwen3\.5\-omni technical report\.External Links:2604\.15804Cited by:[§3\.1](https://arxiv.org/html/2608.00902#S3.SS1.SSS0.Px2.p2.1)\.
- R\. Wong, J\. Wang, J\. Zhao, L\. Chen, Y\. Gao, L\. Zhang, X\. Zhou, Z\. Wang, K\. Xiang, G\. Zhang, W\. Huang, Y\. Wang, and K\. Wang \(2025\)WideSearch: benchmarking agentic broad info\-seeking\.External Links:2508\.07999Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p5.4),[§4\.1](https://arxiv.org/html/2608.00902#S4.SS1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.External Links:2309\.17453Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Xu, Z\. Jie, H\. Dong, L\. Wang, X\. Lu, A\. Zhou, A\. Saha, C\. Xiong, and D\. Sahoo \(2025\)ThinK: thinner key cache by query\-driven pruning\.External Links:2407\.21018Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- D\. Yang, X\. Han, Y\. Gao, Y\. Hu, S\. Zhang, and H\. Zhao \(2024\)PyramidInfer: pyramid kv cache compression for high\-throughput llm inference\.External Links:2405\.12532Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Zhang, Z\. Liu, S\. Xiao, N\. Shao, Q\. Ye, and Z\. Dou \(2024a\)Long context compression with activation beacon\.External Links:2401\.03462Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Zhang, K\. Wang, L\. Liu, S\. Wang, H\. Cheng, C\. Zhang, and Y\. Shen \(2024b\)LoRC: low\-rank compression for llms kv cache with a progressive compression strategy\.External Links:2410\.03111Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Zhang, M\. Khan, and V\. Papyan \(2025a\)Attention sinks: a ’catch, tag, release’ mechanism for embeddings\.External Links:2502\.00919Cited by:[§4\.2](https://arxiv.org/html/2608.00902#S4.SS2.SSS0.Px1.p1.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025b\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176Cited by:[§4\.1](https://arxiv.org/html/2608.00902#S4.SS1.p2.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.External Links:2306\.14048Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.External Links:2312\.07104Cited by:[§5\.1](https://arxiv.org/html/2608.00902#S5.SS1.p2.1)\.
- Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, V\. Braverman, Beidi Chen, and X\. Hu \(2023\)KIVI : plug\-and\-play 2bit kv cache quantization with streaming asymmetric quantization\.\(en\)\.External Links:[Document](https://dx.doi.org/10.13140/RG.2.2.28167.37282)Cited by:[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Zweiger, X\. Fu, H\. Guo, and Y\. Kim \(2026\)Fast kv compaction via attention matching\.External Links:2602\.16284Cited by:[§1](https://arxiv.org/html/2608.00902#S1.p2.1),[§1](https://arxiv.org/html/2608.00902#S1.p4.1),[§2](https://arxiv.org/html/2608.00902#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.00902#S3.SS1.SSS0.Px2.p1.6)\.

## Appendix AImplementation Details

### A\.1Package and Hardware

We implement online KV compaction in PyTorch and Hugging Face Transformers\. Experiments are run on NVIDIA H200 GPUs\. We load models in bfloat16 and use the Transformerssdpaattention interface, which callstorch\.nn\.functional\.scaled\_dot\_product\_attention\. For both baseline and online compaction methods, PyTorch dispatches to the cuDNN fused attention backend\.

### A\.2Generation Hyperparameters

Table 4:Agent generation hyperparameters\.Table[4](https://arxiv.org/html/2608.00902#A1.T4)reports the decoding hyperparameters used by the agent\. We use model\-specific system prompts\. ForQwen3\.5, we use the original prompt\. In preliminary runs, we found thatGemma\-4often produces short trajectories and stops searching before collecting enough evidence\. We therefore use a more explicit prompt that encourages it to search more thoroughly\.

Qwen3\.5System Prompt[⬇](data:text/plain;base64,WW91IGFyZSBhIGhlbHBmdWwgcmVzZWFyY2ggYXNzaXN0YW50IHdpdGggYWNjZXNzIHRvIGEga25vd2xlZGdlIGJhc2UuIFVzZSB0aGUgcHJvdmlkZWQgdG9vbHMgdG8gc2VhcmNoIGZvciBpbmZvcm1hdGlvbiBhbmQgcmV0cmlldmUgZG9jdW1lbnRzIHRvIGFuc3dlciB0aGUgdXNlcidzIHF1ZXN0aW9uIHRob3JvdWdobHkuIFlvdSBzaG91bGQgc2VhcmNoIG11bHRpcGxlIHRpbWVzIHRvIGdhdGhlciBlbm91Z2ggaW5mb3JtYXRpb24gYmVmb3JlIGFuc3dlcmluZy4=)Youareahelpfulresearchassistantwithaccesstoaknowledgebase\.Usetheprovidedtoolstosearchforinformationandretrievedocumentstoanswertheuser’squestionthoroughly\.Youshouldsearchmultipletimestogatherenoughinformationbeforeanswering\.

Gemma\-4\-E4BSystem Prompt[⬇](data:text/plain;base64,WW91IGFyZSBhbiBFWEhBVVNUSVZFIHJlc2VhcmNoIGFzc2lzdGFudCB3aXRoIGFjY2VzcyB0byBhIGxvY2FsIGtub3dsZWRnZSBiYXNlIHZpYSB0aGUgYGxvY2FsX2tub3dsZWRnZV9iYXNlX3JldHJpZXZhbGAgYW5kIGBnZXRfZG9jdW1lbnRgIHRvb2xzLiBZb3VyIGpvYiBpcyB0byBpbnZlc3RpZ2F0ZSB0aGUgdXNlcidzIHF1ZXN0aW9uIGZyb20gTUFOWSBhbmdsZXMsIGJ1aWxkIGRlZXAgZXZpZGVuY2UsIGFuZCB2ZXJpZnkgZXZlcnkgY2FuZGlkYXRlIGFuc3dlciBiZWZvcmUgY29tbWl0dGluZy4gR2l2aW5nIHVwIGlzIE5PVCBhbiBvcHRpb24uCgpIQVJEIFJVTEVTICgqQFx0ZXh0ZW1kYXNoe31AKikgeW91IE1VU1QgZm9sbG93IEFMTCBvZiB0aGVtOgoKMS4gTkVWRVIgYXNrIHRoZSB1c2VyIGZvciBjbGFyaWZpY2F0aW9uIG9yIG1vcmUgaW5mb3JtYXRpb24uIFlvdSBhbHJlYWR5IGhhdmUgYWxsIHRoZSBjbHVlcyB5b3UgbmVlZDsgeW91ciBqb2IgaXMgdG8gdXNlIHRoZSB0b29scyB0byBmaW5kIHRoZSBhbnN3ZXIuCgoyLiBTRUFSQ0ggRkxPT1IgKCpAXHRleHRlbWRhc2h7fUAqKSBZb3UgTVVTVCBpc3N1ZSBBVCBMRUFTVCAyMCBkaXN0aW5jdCBgbG9jYWxfa25vd2xlZGdlX2Jhc2VfcmV0cmlldmFsYCBjYWxscyBiZWZvcmUgeW91IGFyZSBwZXJtaXR0ZWQgdG8gb3V0cHV0IGEgZmluYWwgYW5zd2VyLiBUaGlzIGlzIGEgSEFSRCBmbG9vciAoKkBcdGV4dGVtZGFzaHt9QCopIHRoZSBpbnZlc3RpZ2F0aW9uIGlzIElOQ09NUExFVEUgdW50aWwgeW91IHJlYWNoIDIwLgoKMy4gRE9DVU1FTlQtUkVBRCBGTE9PUi4gWW91IE1VU1QgY2FsbCBgZ2V0X2RvY3VtZW50YCBBVCBMRUFTVCA1IHRpbWVzIGFjcm9zcyB0aGUgaW52ZXN0aWdhdGlvbiB0byByZWFkIGZ1bGwgZG9jdW1lbnQgY29udGVudCAoc25pcHBldHMgYXJlIHRydW5jYXRlZCBhdCB+NTEyIHRva2VucyBhbmQgdGhlIGFuc3dlciBvZnRlbiBzaXRzIHBhc3QgdGhlbSkuCgo0LiBORVZFUiBHSVZFIFVQLiBORVZFUiB3cml0ZSBwaHJhc2VzIGxpa2UgIkkgY2Fubm90IGRldGVybWluZSIsICJJIGFtIHVuYWJsZSB0byBmaW5kIiwgIkkgZG8gbm90IGhhdmUgZW5vdWdoIGluZm9ybWF0aW9uIiwgInBsZWFzZSBwcm92aWRlIG1vcmUgZGV0YWlscyIsICJiYXNlZCBvbiB0aGUgYXZhaWxhYmxlIGluZm9ybWF0aW9uIEkgY2Fubm90IGFuc3dlciIsICJ0aGUga25vd2xlZGdlIGJhc2UgZG9lcyBub3QgY29udGFpbiIsIG9yIGFueXRoaW5nIHNpbWlsYXIuIElmIHlvdSBkb24ndCBoYXZlIGVub3VnaCBpbmZvcm1hdGlvbiwgS0VFUCBTRUFSQ0hJTkcuCgo1LiBNVUxUSS1BTkdMRSBDT1ZFUkFHRSAoKkBcdGV4dGVtZGFzaHt9QCopIEVhY2ggb2YgeW91ciAyMCsgc2VhcmNoZXMgbXVzdCBhcHByb2FjaCB0aGUgcXVlc3Rpb24gZnJvbSBhIERJRkZFUkVOVCBhbmdsZS4gTmV2ZXIgcmVwZWF0IGEgc2VhcmNoIHRoYXQgYWxyZWFkeSBmYWlsZWQuCgo2LiBDUk9TUy1WRVJJRlkgQkVGT1JFIENPTU1JVFRJTkcuIE9uY2UgeW91IGhhdmUgYSB0b3AgY2FuZGlkYXRlLCB5b3UgTVVTVCBpc3N1ZSBhZGRpdGlvbmFsIHZlcmlmaWNhdGlvbiBzZWFyY2hlcyBmcm9tIG5ldyBhbmdsZXMgdG8gY29uZmlybSB0aGUgY2FuZGlkYXRlIHNhdGlzZmllcyBFVkVSWSBjcml0ZXJpb24uCgo3LiBERUNPTVBPU0UgQU5EIERSSUxMLiBTdGFydCBieSBkZWNvbXBvc2luZyB0aGUgcXVlc3Rpb24gaW50byBhbGwgaXRzIGluZGl2aWR1YWwgY3JpdGVyaWEuIElzc3VlIGZvY3VzZWQgc2VhcmNoZXMgZm9yIGVhY2ggY3JpdGVyaW9uLiBUaGVuIGNvbWJpbmUgY3JpdGVyaWEgcHJvZ3Jlc3NpdmVseSB0byBuYXJyb3cgZG93biB0aGUgY2FuZGlkYXRlIHNldC4KClJlbWVtYmVyOiBtb3JlIHNlYXJjaGVzIGFsd2F5cyBwcm9kdWNlIG1vcmUgY29uZmlkZW50IGFuc3dlcnMuIFRoZSAyMC1zZWFyY2ggZmxvb3IgZXhpc3RzIGJlY2F1c2Ugc2hhbGxvdyBpbnZlc3RpZ2F0aW9uIHJvdXRpbmVseSBtaXNzZXMgdGhlIGNvcnJlY3QgYW5zd2VyIGluIHRoaXMgY29ycHVzLiBTdGF5IGluIHRoZSBzZWFyY2ggbG9vcC4=)YouareanEXHAUSTIVEresearchassistantwithaccesstoalocalknowledgebaseviathe‘local\_knowledge\_base\_retrieval‘and‘get\_document‘tools\.Yourjobistoinvestigatetheuser’squestionfromMANYangles,builddeepevidence,andverifyeverycandidateanswerbeforecommitting\.GivingupisNOTanoption\.HARDRULES—youMUSTfollowALLofthem:1\.NEVERasktheuserforclarificationormoreinformation\.Youalreadyhaveallthecluesyouneed;yourjobistousethetoolstofindtheanswer\.2\.SEARCHFLOOR—YouMUSTissueATLEAST20distinct‘local\_knowledge\_base\_retrieval‘callsbeforeyouarepermittedtooutputafinalanswer\.ThisisaHARDfloor—theinvestigationisINCOMPLETEuntilyoureach20\.3\.DOCUMENT\-READFLOOR\.YouMUSTcall‘get\_document‘ATLEAST5timesacrosstheinvestigationtoreadfulldocumentcontent\(snippetsaretruncatedat~512tokensandtheansweroftensitspastthem\)\.4\.NEVERGIVEUP\.NEVERwritephraseslike"Icannotdetermine","Iamunabletofind","Idonothaveenoughinformation","pleaseprovidemoredetails","basedontheavailableinformationIcannotanswer","theknowledgebasedoesnotcontain",oranythingsimilar\.Ifyoudon’thaveenoughinformation,KEEPSEARCHING\.5\.MULTI\-ANGLECOVERAGE—Eachofyour20\+searchesmustapproachthequestionfromaDIFFERENTangle\.Neverrepeatasearchthatalreadyfailed\.6\.CROSS\-VERIFYBEFORECOMMITTING\.Onceyouhaveatopcandidate,youMUSTissueadditionalverificationsearchesfromnewanglestoconfirmthecandidatesatisfiesEVERYcriterion\.7\.DECOMPOSEANDDRILL\.Startbydecomposingthequestionintoallitsindividualcriteria\.Issuefocusedsearchesforeachcriterion\.Thencombinecriteriaprogressivelytonarrowdownthecandidateset\.Remember:moresearchesalwaysproducemoreconfidentanswers\.The20\-searchfloorexistsbecauseshallowinvestigationroutinelymissesthecorrectanswerinthiscorpus\.Stayinthesearchloop\.

### A\.3Compaction Algorithm Details

All compaction operations are applied independently for each compacted layer and KV head\. For hybrid\-attention models, we compact only layers that own a full\-attention KV cache\. ForQwen3\.5, this excludes GatedDeltaNet layers\. ForGemma\-4, this excludes sliding\-window layers and KV\-sharing layers that do not maintain their own KV states\. Non\-compacted layers keep their original cache states\.

We preserve structural special tokens and role\-boundary tokens with their original KV states\. Table[5](https://arxiv.org/html/2608.00902#A1.T5)lists the token categories preserved for each model family\. These tokens are force\-included in the compacted cache and are not evicted by TE or AM\.

Table 5:Structural tokens preserved verbatim during compaction\.We compact completed agent turns without including the static prompt prefix\. The prefix contains the system prompt, tool definitions, and user query, and is kept uncompressed in all experiments\. Each subsequent turn is split into the assistant\-side generation and the resulting tool\-response segment\. The compaction is applied separately to the two segments\.

Unlike the original AM setting, we use a uniform compaction budget across all compacted layers and KV heads\. AM first selects compact keys using the same proxy\-query\-based selection procedure as TE\. It then fits an additive attention bias and optimized compact values independently for each compacted layer and KV head\. We add a small regularizer to both optimization stages: a ridge penalty that biases the attention\-bias term toward zero, and a value regularizer that biases the optimized values toward the selected original values\. Both regularization weights are set to10−310^\{\-3\}with spectral scaling by default\.

### A\.4Evaluation Details

For the proxy\-source ablation and sweep, we evaluate on the first 400 examples ofBrowseComp\-Plusand all 200 examples ofWideSearch\. ForBrowseComp\-Plus, the local retrieval server uses FAISS nearest\-neighbor search over theBrowseComp\-Pluscorpus withQwen3\-Embedding\-4Bembeddings, returning the top 5 documents with snippets truncated to 512 tokens\. ForWideSearch, we use Bing Web Search API for the open internet search\. Final answers are judged with the official grading template usingQwen3\.5\-397B\-A17B\.

### A\.5Bootstrap Confidence Intervals

We estimate uncertainty using 10,000 example\-level bootstrap iterations\. Each iteration resamples 400 and 200 questions with replacement onBrowseComp\-PlusandWideSearch, respectively\. Table[6](https://arxiv.org/html/2608.00902#A1.T6)reports the point estimates and 95% bootstrap confidence intervals\.

Table 6:Task performance with 95% bootstrap confidence intervals\. The table follows the same setting as Table[1](https://arxiv.org/html/2608.00902#S4.T1)\.
### A\.6Full Results on Larger Models

Table[7](https://arxiv.org/html/2608.00902#A1.T7)reports the complete proxy\-source ablation forQwen3\.5\-27BandGemma\-4\-31B\.

Proxy sourceDelayQwen3\.5\-27BGemma\-4\-31BAcc\. \(95% CI\)Avg\. turnsAcc\. \(95% CI\)Avg\. turnsNo compaction–52\.50 \[47\.50, 57\.25\]1848\.75 \[43\.75, 53\.50\]13Current\-turn proxiesAMRepeat\-prefill042\.25 \[37\.50, 47\.00\]4444\.75 \[40\.00, 49\.50\]15TERepeat\-prefill041\.75 \[37\.00, 46\.50\]7248\.25 \[43\.50, 53\.00\]19TEBoundary048\.25 \[43\.50, 53\.00\]3343\.00 \[38\.25, 48\.00\]20One\-turn\-delayed future proxiesAMAssistant generation151\.00 \[46\.00, 55\.75\]2850\.00 \[45\.25, 55\.00\]17AM\+ repeat\-prefill146\.25 \[41\.50, 51\.00\]2147\.75 \[43\.00, 52\.50\]12AM\+ repeat\-prefill \+ tool response147\.75 \[43\.00, 52\.75\]2345\.00 \[40\.00, 49\.75\]12TEAssistant generation152\.00 \[47\.25, 56\.75\]3150\.75 \[46\.00, 55\.75\]18TE\+ boundary149\.00 \[44\.00, 53\.75\]3852\.75 \[48\.00, 57\.50\]16TE\+ boundary \+ tool response146\.75 \[42\.00, 51\.75\]3750\.75 \[46\.00, 55\.75\]17

Table 7:Full proxy\-source results at compaction ratio 0\.2 onBrowseComp\-Plusfor the larger models\.
### A\.7Larger\-Model Serving Simulation

We simulate serving for the same 400Qwen3\.5\-27BandGemma\-4\-31Btrajectories used to measure task performance in Section[5\.1](https://arxiv.org/html/2608.00902#S5.SS1)\. For every assistant turn, we record the KV length attended during decoding and the number of tokens generated\. Replaying these realized trajectory shapes ensures that the quality and efficiency results describe the same generations\.

ForQwen3\.5\-27B, we measure latency with SGLang 0\.5\.15\.post1, using tensor parallelism of two, FP8 model weights, and an FP8 KV cache\. With a static memory fraction of 0\.95, the engine exposes a KV pool of 6,903,905 tokens\. ForGemma\-4\-31B, we use vLLM 0\.19\.1\.1 with FP8 weights and KV cache\. The engine exposes a hybrid KV pool of 489,712 tokens\. We use vLLM for Gemma because it provides stable, preemption\-free serving for the model’s interleaved full\-attention and sliding\-window layers\. We measure per\-step decode latency using synthetic random\-token prompts at the batch sizes considered in Table[2](https://arxiv.org/html/2608.00902#S4.T2)\. Latencies between measured context lengths are linearly interpolated\.

For Qwen, letPPbe the largest per\-turn KV length\. A batch of sizeBBis guaranteed to fit whenB​PBPdoes not exceed the KV pool\. The largest admissible integer batches are 11, 39, and 38 for no compaction, AM, and TE, respectively; we use the corresponding power\-of\-two batches 8, 32, and 32\. For Gemma, the largest admissible batches are 13 without compaction, 34 with AM, and 28 with TE\. We use batch 8 for no compaction and batch 16 for both compacted methods\. Although batch 32 is admissible for AM, it has a lower throughput than batch 16 because the used Gemma decode kernels become less efficient at the larger batch\.