IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

# Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
Source: [https://arxiv.org/html/2606.09916](https://arxiv.org/html/2606.09916)
###### Abstract

Multi\-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning\. Both KV memory and KV read bandwidth grow by orders of magnitude across a single trajectory, making the key–value \(KV\) cache, not parameter compute, the dominant serving bottleneck for long\-horizon agents\. We introduce IntentKV, learned KV pruning that keeps the base LLM frozen\. IntentKV maintains a session\-level QueryMemory of cross\-turn intent, scores live history tokens with a memory\-attention rule, and adds a zero\-initialized residual head with cross\-attention over current\-query K\-vectors\. To stay composable with prefix caches, eviction is a slot\-map redirection: dropped positions route to a sentinel*dead slot*while surviving K/V rows, RoPE phases, and slot identities stay in place\. IntentKV matches the no\-pruning full\-cache baseline with almost no accuracy drop under tight KV budgets: at an 8k KV budget, mean peak request tokens drop 23\.9% on Qwen3\-8B and 30\.7% on Qwen2\.5\-14B\. On the 100 longest BCP queries that all methods complete on Qwen2\.5\-14B, IntentKV\-8k further cuts worst\-case peak request tokens from 92\.3k to 20\.5k \(↓\\downarrow77\.8%\) and worst\-case raw KV reads from 411M to 31M \(↓\\downarrow92\.6%\)\.

IntentKV: Cross\-Turn Intent\-Aware KV Cache Pruning for Agent Inference

Junjie Li Jiong Lou Jie LiShanghai Jiao Tong University

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.09916v1/x1.png)Figure 1:Prompt\-local KV pruning becomes stale in multi\-turn agents\.\(a\) Single\-shot compressors score history against one query; agent sessions accumulate new intents across turns\. \(b\) Signal mismatch between prompt\-local scoring and IntentKV’s session\-aware retention\.Modern LLM agents increasingly execute multi\-turn browsing, deep\-research, and tool\-use workflows\(Yaoet al\.,[2022](https://arxiv.org/html/2606.09916#bib.bib25)\)\. In these settings, a short user request often expands into a long trajectory of retrieved documents, tool outputs, intermediate decisions, and generated arguments\. This creates an asymmetric serving profile: although the original user query may contain only a few dozen tokens, the agent state can grow to tens of thousands of prompt tokens, and each generated token must attend to the KV tensors of all previous tokens at every layer\. As a result, KV memory capacity and KV read bandwidth, rather than parameter computation alone, become the dominant bottlenecks in serving long\-running agent sessions\(Liet al\.,[2024a](https://arxiv.org/html/2606.09916#bib.bib19); Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)\.

A natural response is to apply existing KV\-cache pruning methods, but most were designed for single\-prompt compression rather than multi\-turn agent serving\. StreamingLLM\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib1)\)keeps attention sinks and a recent window; SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2606.09916#bib.bib2)\)estimates token importance from prompt\-local tail attention; H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib3)\)retains heavy hitters\. These methods decide which past tokens to keep from signals available within a single prompt, and their standard use typically materializes the kept KV rows as a compacted request\-local cache\.

When these methods are applied to multi\-turn agent serving, two assumptions from single\-prompt compression become problematic\. First, token importance changes across turns: later queries, retrieved evidence, and intermediate decisions can make early history useful again, so a ranking computed from one prompt grows stale and may discard future\-use evidence\. Second, the pruned state must remain reusable: agent sessions repeatedly submit histories with long shared prefixes, which systems such as SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)and vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib15)\)exploit through radix or prefix caches\. Relocating surviving KV rows may serve the current request, but it changes the cache identity that later turns would otherwise reuse\.[Figure˜1](https://arxiv.org/html/2606.09916#S1.F1)illustrates both failure modes: prompt\-local scoring can drop future\-use evidence, and relocation\-based pruning loses the prefix identity needed across turns\.

These two constraints suggest separating what pruning keeps from how the pruned state is represented\. The retention policy should track information needs as they evolve across the session, rather than scoring history only against a single prompt\. The layout policy should evict tokens without relocating surviving K/V rows, preserving their logical positions and cache identities for later turns\. We instantiate this separation in IntentKV: a session\-aware heuristic provides the primary retention signal, a small learned residual handles cases the rule misses, and a layout\-preserving eviction recipe keeps pruning compatible with prefix\-cache reuse\.

On the retention side, IntentKV maintains a QueryMemory that aggregates query\-like signals across the session and scores live history tokens against this evolving state\. A small zero\-initialized residual head, trained from frozen\-model features, learns a correction for cases the rule scorer misses while recovering the rule score at initialization\.

On the layout side, IntentKV evicts without compaction\. Dropped positions are redirected to a sentinel*dead slot*, while surviving K/V rows, RoPE phases, and logical slot identities remain unchanged\. Agent sessions can therefore combine KV pruning with cross\-turn prefix reuse, rather than choosing one or the other\.

#### Contributions\.

- •We formulate agent KV pruning as*multi\-query retention*and instantiate it with QueryMemory, a per\-session state aggregating queries, tool calls, search intents, and action spans across turns\.
- •A learned\-residual pruner keeps the session\-aware heuristic as the primary signal and adds a frozen\-feature correction grounded in future actions, without updating the base LLM\.
- •An eviction recipe for paged KV caches \(SGLang/vLLM\) that redirects dropped positions to a sentinel slot rather than relocating survivors, so pruning stays composable with radix\-prefix reuse without changes to existing attention kernels:20\.7%20\.7\\%prefix\-hit rate at an88k budget where compaction baselines fall to0–3%3\\%\.
- •On BCP atC=8C\{=\}8k, IntentKV\-Phase\-2 matches Full\-cache True Acc within0\.960\.96points on Qwen3\-8B and surpasses the strongest heuristic by10\.3610\.36points on Qwen2\.5\-14B; on the100100longest queries it cuts worst\-case peak request tokens by77\.8%77\.8\\%–81\.7%81\.7\\%and worst\-case raw KV reads by36\.9%36\.9\\%–92\.6%92\.6\\%\([Table˜3](https://arxiv.org/html/2606.09916#S4.T3)\)\.

## 2Related Work

#### From prompt\-local scores to cross\-turn intent\.

Most KV eviction methods score tokens against a single user prompt\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib1); Liuet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib28); Zhanget al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib3); Liet al\.,[2024b](https://arxiv.org/html/2606.09916#bib.bib2); Zhouet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib6); Ahnet al\.,[2026](https://arxiv.org/html/2606.09916#bib.bib23)\), with orthogonal refinements for per\-layer budgets, query\-aware decode\-time sparsity, and future\-query pre\-scoring\(Geet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib29); Caiet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib7); Tanget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib8); Devotoet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib9)\)\. All assume a single request available at prefill\. Multi\-turn agents break this: later turns surface fresh query signals in tool outputs and intermediate plans that a one\-shot scorer cannot anticipate\. IntentKV instead formulates pruning as multi\-query retention, scoring history against a session\-level QueryMemory that absorbs these signals as they arrive\. KV quantization\(Liuet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib10); Kanget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib11)\)reduces per\-token precision rather than token count and is orthogonal to IntentKV\.

#### Semantic pruning without changing the base model\.

A second line learns token\-importance policies on frozen\-model features\(Ahnet al\.,[2026](https://arxiv.org/html/2606.09916#bib.bib23); Zhouet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib6)\), though pure learned scores can be brittle under weak or shifted query signal\. TRIM\-KV\(Buiet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib4)\)is the closest peer: it trains a small predictor and biases retention via a*positional time\-decay*prior, well\-suited to chain\-of\-thought trajectories where relevant context is temporally local\. Agent loops invert this regime—useful tokens are often the oldest \(initial request, early retrieved page, first tool result\) while youngest decode tokens are scaffolding—so a time\-decay prior is tuned to a different workload\. IntentKV trains a comparable head but conditions it on QueryMemory and supervises with which positions are re\-attended in later turns, yielding a*query\-decay*policy anchored to evolving intent\. SideQuest\(Kariyappa and Suh,[2026](https://arxiv.org/html/2606.09916#bib.bib20)\)and Activation Beacon\(Zhanget al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib12)\)modify the base model \(deletion commands and learned summary tokens respectively\) and so are complementary rather than substitutable; IntentKV keeps the base LLM frozen and adds only a zero\-initialized residual head\.

#### Composable serving\- and orchestration\-layer systems\.

Efficient agent serving reuses KV state via paged allocation and prefix/radix caches when consecutive requests share a prefix\(Kwonet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib15); Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\), and Continuum\(Liet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib5)\)extends this to multi\-turn agents through KV time\-to\-live scheduling\. Conventional compaction shifts the physical slots and RoPE phases of surviving tokens and invalidates the prefix identity such systems would reuse; IntentKV keeps surviving K/V rows in place and redirects dropped logical positions to a sentinel slot, so request\-local compression composes with radix\-prefix reuse\. At a higher layer, MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib13)\)and A\-MEM\(Xuet al\.,[2026](https://arxiv.org/html/2606.09916#bib.bib14)\)manage agent state via OS\-style swapping and long\-term graph memory, deciding what to remember at the orchestration layer; IntentKV operates inside the cache with the base LLM and agent loop unchanged and so composes with such memories rather than competing\.

## 3Method

![Refer to caption](https://arxiv.org/html/2606.09916v1/x2.png)Figure 2:IntentKV method overview\.\(a\) Rule score against QueryMemory; \(b\) zero\-init residual correction via cross\-attention over current\-query K\-vectors; \(c\) final score combination feeding top\-kkretention\. Eviction redirects dropped positions to a sentinel*dead slot*on the existing paged KV cache, preserving radix prefix identity\.IntentKV \([Figure˜2](https://arxiv.org/html/2606.09916#S3.F2)\) intervenes once per request after prefill, when the sequence lengthNNexceeds a retention budgetCC\. It produces a kept set𝒦⊆\[0,N\)\\mathcal\{K\}\\\!\\subseteq\\\!\[0,N\)that always retains the current actionable query span and the protected system prefix, holds every other surviving token at its original logical position and physical KV slot, and lets the cross\-request radix tree\(Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)match prefixes despite eviction\. The design factors into two layers:*retention*ranks history under cross\-turn intent, and*layout*keeps the surviving prefix reusable\.

### 3\.1Problem Formulation

We frame KV pruning as a budgeted top\-kkselection over post\-prefill tokens, with a small protected set fixed in advance\. Let𝐒r∈ℤN\\mathbf\{S\}\_\{r\}\\\!\\in\\\!\\mathbb\{Z\}^\{N\}denote the slot map of requestrr,𝐊,𝐕∈ℝL×Hk​v×N×D\\mathbf\{K\},\\mathbf\{V\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times H\_\{kv\}\\times N\\times D\}the post\-RoPE\(Suet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib26)\)keys and values, and\[qs,qe\)\[q\_\{s\},q\_\{e\}\)the actionable query span \(the latest user, tool, or function message that can affect the next decision, resolved after template rendering\)\. The protected prefixπ\\picovers the chat\-template system span\. With the forced setℱ=\[0,π\)∪\[qs,qe\)\\mathcal\{F\}\\\!=\\\!\[0,\\pi\)\\cup\[q\_\{s\},q\_\{e\}\), residual budgetC⋆=max⁡\(0,C−\|ℱ\|\)C^\{\\star\}\\\!=\\\!\\max\(0,C\-\|\\mathcal\{F\}\|\), and candidate historyℋ=\[0,N\)∖ℱ\\mathcal\{H\}\\\!=\\\!\[0,N\)\\setminus\\mathcal\{F\}, compression selects

𝒦⋆=ℱ∪arg​top𝒦⊆ℋ\|𝒦\|=C⋆​∑j∈𝒦sj,\\mathcal\{K\}^\{\\star\}=\\mathcal\{F\}\\,\\cup\\,\\operatorname\*\{arg\\,top\}\_\{\\begin\{subarray\}\{c\}\\mathcal\{K\}\\subseteq\\mathcal\{H\}\\\\ \|\\mathcal\{K\}\|=C^\{\\star\}\\end\{subarray\}\}\\sum\_\{j\\in\\mathcal\{K\}\}s\_\{j\},\(1\)wheresjs\_\{j\}is produced by one of three scorer modes:queryuses only the current query encoding,memoryaccumulates queries across the session via a recurrence, andlearnableadds a residual head on top ofmemory\. ThusCCcontrols compressible history rather than serving as a hard cap on prompt\-plus\-generation tokens; implementation details are in[Appendix˜H](https://arxiv.org/html/2606.09916#A8)\.

### 3\.2Cross\-Turn Query Memory

Within one agent session, attention from later queries no longer concentrates on the regions that the first user prompt singled out, so a scorer trained onq0q\_\{0\}alone is mis\-calibrated forq1:tq\_\{1\{:\}t\}\([Figure˜3](https://arxiv.org/html/2606.09916#S3.F3)\)\. Per session, IntentKV maintains a query memory𝐌t∈ℝL×Hq×D\\mathbf\{M\}\_\{t\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times H\_\{q\}\\times D\}with the shape of one post\-RoPE query row, updated by a geometric recurrence:

𝐌t=e−λ​𝐌t−1\+Enc​\(𝐪t\),\\mathbf\{M\}\_\{t\}=e^\{\-\\lambda\}\\,\\mathbf\{M\}\_\{t\-1\}\+\\mathrm\{Enc\}\(\\mathbf\{q\}\_\{t\}\),\(2\)
![Refer to caption](https://arxiv.org/html/2606.09916v1/x3.png)Figure 3:Cross\-turn attention drifts away from the initial prompt\.On a BCP rollout \(qid861861, Qwen3\-8B; turn\-aware synthetic preview\), each queryq0→q4q\_\{0\}\\\!\\to\\\!q\_\{4\}activates a different region of the prefix \(positions0–20002000\)\.whereEnc​\(𝐪t\)\\mathrm\{Enc\}\(\\mathbf\{q\}\_\{t\}\)averages the post\-RoPE Q rows over the actionable query span; the decay rateλ\\lambdais fixed across the session \(value set in[Appendix˜A](https://arxiv.org/html/2606.09916#A1)\)\. After each update,𝐌t\\mathbf\{M\}\_\{t\}is projected to unit norm along the head dimension; without this, its norm grows roughly linearly with turn count and the softmax below collapses onto a single coordinate\. With𝐊∈ℝL×Hk​v×N×D\\mathbf\{K\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times H\_\{kv\}\\times N\\times D\}from the current request and KV heads expanded toHqH\_\{q\}under GQA, candidate positionj∈ℋj\\\!\\in\\\!\\mathcal\{H\}receives the rule score

rulej\\displaystyle\\mathrm\{rule\}\_\{j\}=∑l,hal,h,j,\\displaystyle=\\sum\_\{l,h\}a\_\{l,h,j\},\(3\)al,h,j\\displaystyle a\_\{l,h,j\}=exp⁡\(𝐌t​\[l,h\]⋅𝐊​\[l,h,j\]/D\)∑i∈ℋexp⁡\(𝐌t​\[l,h\]⋅𝐊​\[l,h,i\]/D\)\.\\displaystyle=\\frac\{\\exp\(\\mathbf\{M\}\_\{t\}\[l,h\]\\\!\\cdot\\\!\\mathbf\{K\}\[l,h,j\]/\\sqrt\{D\}\)\}\{\\sum\_\{i\\in\\mathcal\{H\}\}\\exp\(\\mathbf\{M\}\_\{t\}\[l,h\]\\\!\\cdot\\\!\\mathbf\{K\}\[l,h,i\]/\\sqrt\{D\}\)\}\.[Equation˜3](https://arxiv.org/html/2606.09916#S3.E3)replaces SnapKV’s window ofWWtail Q rows with a single accumulated query state, reducing per\-event scoring from𝒪​\(W​L​Hq​N\)\\mathcal\{O\}\(WLH\_\{q\}N\)to𝒪​\(L​Hq​N\)\\mathcal\{O\}\(LH\_\{q\}N\)while incorporating evidence from prior turns\.

### 3\.3Residual Learnable Pruner

We refer to the rule scorer of[Equation˜3](https://arxiv.org/html/2606.09916#S3.E3)as the Phase\-1 configuration and to the rule\-plus\-residual scorer defined below as the Phase\-2 configuration\. IntentKV optionally augments the rule score with a learned residual that produces thesjs\_\{j\}consumed by[Equation˜1](https://arxiv.org/html/2606.09916#S3.E1),

sj=rulej\+α⋅MLP​\(ϕj⊕𝐜j\),s\_\{j\}=\\mathrm\{rule\}\_\{j\}\+\\alpha\\cdot\\mathrm\{MLP\}\\\!\\big\(\\bm\{\\phi\}\_\{j\}\\oplus\\mathbf\{c\}\_\{j\}\\big\),\(4\)whereα∈\[−5,5\]\\alpha\\\!\\in\\\!\[\-5,5\]is a learned scalar clipped at inference,ϕj∈ℝ3​D\+1\\bm\{\\phi\}\_\{j\}\\\!\\in\\\!\\mathbb\{R\}^\{3D\+1\}is the per\-token feature vector, and𝐜j∈ℝdc\\mathbf\{c\}\_\{j\}\\\!\\in\\\!\\mathbb\{R\}^\{d\_\{c\}\}is a cross\-attention readout over the current query’s K\-vectors\.

#### Per\-token feature vector\.

Let𝐤¯j\\bar\{\\mathbf\{k\}\}\_\{j\}be the mean of𝐊​\[l,h,j\]\\mathbf\{K\}\[l,h,j\]over GQA\-expanded heads and layers, and𝐦¯\\bar\{\\mathbf\{m\}\}the analogous mean over𝐌t\\mathbf\{M\}\_\{t\}\. Then

ϕj=\[𝐤¯j​‖𝐦¯‖​𝐤¯j⊙𝐦¯∥rulej\]\.\\bm\{\\phi\}\_\{j\}=\\big\[\\,\\bar\{\\mathbf\{k\}\}\_\{j\}\\,\\\|\\,\\bar\{\\mathbf\{m\}\}\\,\\\|\\,\\bar\{\\mathbf\{k\}\}\_\{j\}\\\!\\odot\\\!\\bar\{\\mathbf\{m\}\}\\,\\\|\\,\\mathrm\{rule\}\_\{j\}\\,\\big\]\.\(5\)Mean aggregation keeps every column ofϕj\\bm\{\\phi\}\_\{j\}at𝒪​\(1\)\\mathcal\{O\}\(1\)regardless of depth or head count, so one residual topology covers backbones with the sameDD\. The element\-wise interaction𝐤¯j⊙𝐦¯\\bar\{\\mathbf\{k\}\}\_\{j\}\\\!\\odot\\\!\\bar\{\\mathbf\{m\}\}supplies an explicit token–memory alignment channel that a linear concatenation cannot represent\.

#### Cross\-attention over query K\-vectors\.

Per\-token features collapse every layer and head, losing which current query token a candidate matches\. The residual therefore reads the query span back through a multi\-head cross\-attention\. Let𝐊¯q∈ℝnq×D\\bar\{\\mathbf\{K\}\}^\{q\}\\\!\\in\\\!\\mathbb\{R\}^\{n\_\{q\}\\times D\}aggregate K\-vectors at the live query positions \(positions evicted in a prior round are excluded, so the attention consumes only live evidence\)\. WithHc=4H\_\{c\}\\\!=\\\!4heads and widthdc=128d\_\{c\}\\\!=\\\!128,

𝐜j=CrossAttn​\(WQ​ϕj,WK​𝐊¯q,WV​𝐊¯q\)\.\\mathbf\{c\}\_\{j\}=\\mathrm\{CrossAttn\}\\big\(W\_\{Q\}\\bm\{\\phi\}\_\{j\},\\;W\_\{K\}\\bar\{\\mathbf\{K\}\}^\{q\},\\;W\_\{V\}\\bar\{\\mathbf\{K\}\}^\{q\}\\big\)\.\(6\)Candidate features form the queries; past query K\-vectors form keys and values\.

#### Initialisation and lower bound\.

The MLP body is a two\-layer GELU network of hidden widthh=256h\\\!=\\\!256\. Its output projection is zero\-initialised in both weight and bias, whileα=1\\alpha\\\!=\\\!1at start\. Under\(α=1,fc2=0\)\(\\alpha\\\!=\\\!1,\\,\\mathrm\{fc\}\_\{2\}\\\!=\\\!0\)the pruner exactly recovers[Equation˜3](https://arxiv.org/html/2606.09916#S3.E3)on day zero, gradients still flow throughfc2\\mathrm\{fc\}\_\{2\}, and the rule scorer remains a strict floor under the learned head; pairingα=0\\alpha\\\!=\\\!0withfc2=0\\mathrm\{fc\}\_\{2\}\\\!=\\\!0would freeze the gradient path at a\(0,0\)\(0,0\)saddle\. ForD=128D\\\!=\\\!128, the residual head has214,274214\{,\}274trainable parameters\.

#### Ablation knobs\.

The full configuration*Full Phase\-2*activates all three additions in[Equation˜4](https://arxiv.org/html/2606.09916#S3.E4)\.[Section˜4\.4](https://arxiv.org/html/2606.09916#S4.SS4)ablates them independently:*−\-memory*replaces𝐌t\\mathbf\{M\}\_\{t\}withEnc​\(𝐪t\)\\mathrm\{Enc\}\(\\mathbf\{q\}\_\{t\}\),*−\-x\-attn*forces𝐜j=𝟎\\mathbf\{c\}\_\{j\}\\\!=\\\!\\mathbf\{0\}in[Equation˜6](https://arxiv.org/html/2606.09916#S3.E6), and*−\-residual*clampsα=0\\alpha\\\!=\\\!0, exactly recovering the rule scorer\.

### 3\.4Slot\-Map Eviction

Serving stacks already decouple logical token positions from physical KV slots through a paged allocator and a per\-request slot map\(Kwonet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib15); Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)\. Standard compactors copy kept rows into a contiguous range and re\-rotate keys, which changes both slot identity and RoPE phase and breaks the radix\-tree match against later turns\. IntentKV instead leaves every kept K/V row at its original slot and modifies only𝐒r\\mathbf\{S\}\_\{r\}:

𝐒r​\[j\]←s†,∀j∈\[0,N\)∖𝒦⋆,\\mathbf\{S\}\_\{r\}\[j\]\\leftarrow s^\{\\dagger\},\\quad\\forall j\\in\[0,N\)\\setminus\\mathcal\{K\}^\{\\star\},\(7\)with the token\-id stream𝐟𝐢𝐥𝐥​\_​𝐢𝐝𝐬r\\mathbf\{fill\\\_ids\}\_\{r\}and sequence lengthNNuntouched\. The radix tree therefore continues to match this request against later turns sharing its prefix, exactly as if no pruning had happened\.

#### Sentinel as in\-place mask\.

The redirect targets†s^\{\\dagger\}in[Equation˜7](https://arxiv.org/html/2606.09916#S3.E7)is a single sentinel slot allocated at compressor startup and held in a never\-free set, with𝐊​\[s†\]=−104​𝟏D\\mathbf\{K\}\[s^\{\\dagger\}\]\\\!=\\\!\-10^\{4\}\\mathbf\{1\}\_\{D\}on every layer and𝐕​\[s†\]=𝟎D\\mathbf\{V\}\[s^\{\\dagger\}\]\\\!=\\\!\\mathbf\{0\}\_\{D\}\. The pre\-softmax logit at the sentinel is thus−104​D\-10^\{4\}\\sqrt\{D\}, driving its softmax weight toexp⁡\(−104​D\)\\exp\(\-10^\{4\}\\sqrt\{D\}\), far below the smallest representable bf16/fp16 value; the zero value then nulls any residual contribution\. The sentinel is numerically equivalent to a hard visibility mask, but it lives at the slot\-map level rather than inside the attention kernel, so existing flashinfer and FA3 paths run unmodified\.

#### Alias\-aware deallocation\.

A dropped physical slotu=𝐒r​\[j\]u\\\!=\\\!\\mathbf\{S\}\_\{r\}\[j\]is returned to the allocator only if \(a\) positionjjlies outside the radix\-protected prefix, \(b\)u≠s†u\\\!\\neq\\\!s^\{\\dagger\}, and \(c\)u∉\{𝐒r​\[i\]:i∈𝒦⋆\}u\\notin\\\{\\mathbf\{S\}\_\{r\}\[i\]:i\\\!\\in\\\!\\mathcal\{K\}^\{\\star\}\\\}\. Condition \(c\) catches two alias patterns: positions already redirected tos†s^\{\\dagger\}in a prior compression round, and slots shared with sibling requests on the same radix branch\. Without this check, freeinguuwould corrupt KV reads from those siblings, which is the practical obstacle that forces prior compaction\-based pruners to disable prefix\-cache reuse\. The slot\-map indirection is borrowed from PagedAttention\(Kwonet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib15)\)and the sentinel\-as\-mask construction is the standard attention\-masking idiom; what we add is the integration itself: combining them so per\-request pruning becomes composable with radix\-prefix reuse without modifying the attention kernel\.

### 3\.5Training

#### Frozen\-model feature extraction\.

Training reads post\-RoPE Q and K from a frozen copy of the deployed model\. We register a RoPE hook that captures the post\-rotation tensors exactly where the attention backend consumes them, so the offline feature distribution matches inference\. Long sequences are processed in4,0964\{,\}096\-token chunks with KV caching enabled to bound attention\-matrix memory; chunk boundaries respect the actionable query span so each example sees its full encoding\.

#### Future\-action grounding\.

For each\(session,turn\)\(\\texttt\{session\},\\,\\texttt\{turn\}\)row, the labeller scans the next five tool calls in the same session and marks any history token whose content matches a literal substring of a future tool argument as positive\. Tokens inside the current query span and special\-template tokens carry label−100\-100and are excluded from the loss; rows with no matched evidence are dropped rather than back\-filled with a recency proxy, which improves label precision at the cost of some textually unrecoverable turns\.

#### Loss\.

The total objective sums four terms,

ℒ\\displaystyle\\mathcal\{L\}=ℒBCE​\(s,y\)\+ρ​ℒrank\\displaystyle=\\mathcal\{L\}\_\{\\text\{BCE\}\}\(s,y\)\+\\rho\\,\\mathcal\{L\}\_\{\\text\{rank\}\}\(8\)\+γ​σ​\(s\)¯\+η​\(‖𝐌t‖2−1\)2,\\displaystyle\\quad\+\\gamma\\,\\overline\{\\sigma\(s\)\}\+\\eta\\,\(\\\|\\mathbf\{M\}\_\{t\}\\\|\_\{2\}\-1\)^\{2\},with defaultsρ=0\.05\\rho\\\!=\\\!0\.05,γ=0\.01\\gamma\\\!=\\\!0\.01,η=0\.001\\eta\\\!=\\\!0\.001\.ℒBCE\\mathcal\{L\}\_\{\\text\{BCE\}\}uses an adaptive positive weight\(1−p~\)/p~\(1\-\\tilde\{p\}\)/\\tilde\{p\}withp~=clip​\(p,0\.02,0\.5\)\\tilde\{p\}\\\!=\\\!\\mathrm\{clip\}\(p,0\.02,0\.5\)to absorb the long\-tailed positive rate;ℒrank\\mathcal\{L\}\_\{\\text\{rank\}\}is a pairwisesoftplus\\operatorname\{softplus\}surrogate that compares each positive against the top\-6464hard negatives; the third term discourages an unbounded keep probability; the last term softly enforces the unit\-norm projection on𝐌t\\mathbf\{M\}\_\{t\}used at inference\.

#### Optimisation\.

Sessions are replayed in increasingturn\_indexso𝐌t\\mathbf\{M\}\_\{t\}accumulates as at inference and are shuffled between epochs\. One residual head is trained per backbone; the topology is identical and the differences are captured by separately trained weights\. See[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)for hyperparameters\.

## 4Experiments

![Refer to caption](https://arxiv.org/html/2606.09916v1/x4.png)Figure 4:BCP accuracy under two distinct cost metrics\.Each column fixes one \(model, budget\) configuration\. The top row plots True Acc against the*KV memory footprint*\(PT, k tokens\)\. The bottom row plots True Acc against the*decode\-side attention workload*\(Eff\. Live KV, k tokens\)\. IntentKV\-Phase\-2 is Pareto\-dominant on both axes atC=8C\{=\}8k\.### 4\.1Experimental Setup

#### Models, serving, and benchmark\.

We evaluate Qwen3\-8B\-Instruct\(Yanget al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib16)\)and Qwen2\.5\-14B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib17)\)on a single8080GiB A100, both Neox\-RoPE GQA models\. The restriction to Qwen backbones is empirically motivated rather than methodological: a seven\-model cross\-architecture smoke study spanning Llama\-3\.1\-8B, Mistral\-Nemo\-2407, Mistral\-Small\-3\.2\-24B, Hermes\-3\-Llama\-3\.1\-8B, watt\-tool\-8B, gpt\-oss\-20b, and GLM\-4\-9B found that no non\-Qwen open\-source backbone in our sample reliably engages in autonomous multi\-step tool calling on BCP, even after we fixed the corresponding parser and tokenizer\-protocol incompatibilities; the failures sit at the agentic\-behavior layer, not at the KV\-compression layer \([Appendix˜G](https://arxiv.org/html/2606.09916#A7)\)\. All inference uses SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)with radix prefix caching, deterministic decoding \(T=0T\{=\}0\), and at most3232tool\-use turns per query\. The benchmark is BrowseComp\-Plus \(BCP;Chenet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib18)\), an830830\-query deep\-research suite over a fixed∼\\sim100K\-document corpus; the retriever is frozen via a per\-query cache so compression effects are not confounded by retrieval drift\.

#### Training\.

The Phase\-2 pruner is trained on strict\-cleaned ToolBench\(Qinet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib21); Guoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib22)\)multi\-turn traces with future\-action labels obtained by literal substring matching against the next five tool\-call arguments \([Section˜3\.5](https://arxiv.org/html/2606.09916#S3.SS5)\)\. Training uses reservoir\-sampled33K–55K examples per epoch for two epochs with AdamW \(lr10−310^\{\-3\}, weight decay0\.010\.01, grad\-accum88, seed4242\); the base LLM is frozen in bf16 and only the FP32 residual head is updated\. Trigger size is512512tokens and tail windowW=32W\{=\}32\. The Qwen3\-8B \(resp\. Qwen2\.5\-14B\) run completes in∼\\sim4040\(∼\\sim5555\) minutes on4×804\{\\times\}80GiB A100s with final loss0\.540\.54\(0\.510\.51\); training curves are in[Appendix˜F](https://arxiv.org/html/2606.09916#A6)\.

#### Baselines\.

We compare IntentKV\-Phase\-2 against a no\-pruningFull\-cacheceiling,StreamingLLM\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib1)\)\(sink44\+ recent window\),SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2606.09916#bib.bib2)\)\(kernel77,W=32W\{=\}32, max\-pool\),H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib3)\)\(ratio mode with recent floor\), and a publicTrimKV\(Buiet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib4)\)checkpoint that uses positional time\-decay rather than query\-conditioned retention\. All compressors run atC∈\{8192,16384\}C\\in\\\{8192,16384\\\}applied to the compressible history at each pruning event\.

#### Metrics\.

*Accuracy\.*Following BCP, a Qwen3\-32B LLM judge\(Zhenget al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib27)\)labels each completed trajectory; we reportCompl\(queries finishing without OOM, overflow, turn cap, error, or empty output\),Raw\(Correct/Completed\\textsc\{Correct\}/\\textsc\{Completed\}\), and the headlineTrueAcc \(Correct/Total=\\textsc\{Correct\}/\\textsc\{Total\}=Raw×\\timesCompl, with non\-completion counted as wrong\)\.*System cost\.*All system\-side metrics are lower\-is\-better\. LetRtR\_\{t\}denote the raw \(pre\-pruning\) attended\-to set at decode stepttandKt⊆RtK\_\{t\}\\subseteq R\_\{t\}the*effective*subset that survives pruning and actually participates in attention;\|Rt\|\|R\_\{t\}\|and\|Kt\|\|K\_\{t\}\|are the corresponding sizes\. We report:

- •PT\(peak request tokens, k\): per\-query maximummaxt⁡\|Rt\|\\max\_\{t\}\|R\_\{t\}\|of the uncompressed request size; a memory\-footprint upper bound\.
- •Eff\. Live KV\(k tokens\): per\-query peak of the live \(post\-pruning\) effective KV,maxt⁡\|Kt\|\\max\_\{t\}\|K\_\{t\}\|; the attention\-workload metric reported on the bottom row of[Figure˜4](https://arxiv.org/html/2606.09916#S4.F4)\.
- •Raw KV Reads\(10610^\{6\}tokens\): cumulative pre\-pruning KV reads∑t\|Rt\|\\sum\_\{t\}\|R\_\{t\}\|\.
- •Eff\. KV Reads\(10610^\{6\}tokens\): cumulative effective KV reads∑t\|Kt\|\\sum\_\{t\}\|K\_\{t\}\|; proxies attention bandwidth\.

We additionally report wall\-clock latency \(Wall, s\) and the radix prefix\-hit rate \(in %, quoted inline\)\. Cohort statistics in[Tables˜2](https://arxiv.org/html/2606.09916#S4.T2),[4](https://arxiv.org/html/2606.09916#A2.T4)and[6](https://arxiv.org/html/2606.09916#A4.T6)are means across the relevant query set;[Table˜3](https://arxiv.org/html/2606.09916#S4.T3)reports cohort maximaP​Tmax=maxq⁡PT​\(q\)PT\_\{\\max\}\{=\}\\max\_\{q\}\\mathrm\{PT\}\(q\)andRmax=maxq⁡Raw​KV​Reads​\(q\)R\_\{\\max\}\{=\}\\max\_\{q\}\\mathrm\{Raw\\,KV\\,Reads\}\(q\)\.

### 4\.2Main Results: Accuracy

#### The 8k budget is the stress regime\.

[Table˜1](https://arxiv.org/html/2606.09916#S4.T1)reports the full830830\-query evaluation\. On Qwen3\-8B atC=8C\{=\}8k, the four compaction baselines collapse on completion: StreamingLLM finishes65\.42%65\.42\\%\(−23\.6\-23\.6vs\. Full\-cache\); H2O, SnapKV, TrimKV drop further to45\.6645\.66–51\.57%51\.57\\%\. IntentKV\-Phase\-2 keeps84\.58%84\.58\\%completion and reaches14\.1014\.10True Acc, surpassing StreamingLLM by2\.292\.29, TrimKV by3\.143\.14, H2O by4\.344\.34, SnapKV by4\.944\.94\. The Qwen2\.5\-14B margin is larger: IntentKV\-Phase\-2 atC=8C\{=\}8k recovers18\.5518\.55True Acc vs\. a20\.6020\.60Full\-cache ceiling, while the best heuristic \(StreamingLLM\) reaches only8\.198\.19—a10\.3610\.36\-point gap\.

#### At 16k the gap to the ceiling closes\.

On Qwen3\-8B atC=16C\{=\}16k, IntentKV\-Phase\-2 attains14\.9414\.94True Acc, within0\.120\.12of no\-pruning and ahead of every heuristic \(StreamingLLM14\.4614\.46, H2O13\.8613\.86, TrimKV13\.2513\.25, SnapKV12\.7712\.77\)\. On Qwen2\.5\-14B the margin narrows \(14\.5814\.58vs\. StreamingLLM16\.1416\.14\): Qwen2\.5\-14B BCP trajectories average∼\\sim1717k peak request tokens, so a1616k budget rarely triggers eviction— consistent with the same model’s10\.3610\.36\-point IntentKV advantage at the tighter88k budget where retention decisions actually matter\.

Table 1:IntentKV\-Phase\-2 lands within0\.960\.96True Acc of Full\-cache on Qwen3\-8B and beats the strongest heuristic by10\.3610\.36points on Qwen2\.5\-14B\(830830\-query BCP eval\)\. Best non\-Full\-cache cell per column inbold, second\-bestunderlined; Full\-cache is shown only atC=16C\{=\}16k as the uncompressed ceiling\.

### 4\.3Main Results: Efficiency

#### Efficiency gains come from both tighter retention and preserved prefix reuse\.

[Table˜2](https://arxiv.org/html/2606.09916#S4.T2)shows IntentKV’s Eff\. KV Reads sits at or below every compaction baseline on both backbones: it is within88M of StreamingLLM on Qwen3\-8B and roughly2×2\\timeslower than every compaction baseline on Qwen2\.5\-14B \(21\.021\.0–24\.124\.1vs\.44\.544\.5–58\.758\.7M\), indicating that QueryMemory selects a tighter live working set than tail\-attention or heavy\-hitter heuristics\. On top of this, the hole\-preserving layout keeps radix prefix caches valid:20\.7%20\.7\\%/26\.0%26\.0\\%prefix\-hit atC=8C\{=\}8k/1616k, while compaction baselines fall to0–3%3\\%because relocation renumbers positions\. With both effects active, IntentKV\-Phase\-2 atC=16C\{=\}16k on Qwen3\-8B issues32\.032\.0M Raw KV Reads \(matching Full\-cache’s32\.232\.2M;−51\.1%\-51\.1\\%vs\. H2O’s65\.565\.5M\), with wall time139\.2139\.2s \(−43\.1%\-43\.1\\%vs\. H2O, within5\.9%5\.9\\%of no\-pruning\)\. AtC=8C\{=\}8k, Raw KV Reads drop to31\.631\.6M \(−59\.1%\-59\.1\\%vs\. H2O/SnapKV\)\. The prefix\-reuse contribution is isolable in[Appendix˜E](https://arxiv.org/html/2606.09916#A5): re\-running SnapKV and H2O on IntentKV’s dead\-slot substrate cuts wall\-time4444–46%46\\%and Raw KV Reads3939–47%47\\%on both baselines with no change to scoring or budget, while IntentKV\-Phase\-2 retains a1\.5×1\.5\\timesreduction in Raw KV Reads over SnapKV under the matched substrate at indistinguishable True Acc \(Δ=0\.35​σ\\Delta\{=\}0\.35\\sigmaon830830queries\)\.

#### Worst\-case KV pressure \(vs\. uncompressed\)\.

[Table˜3](https://arxiv.org/html/2606.09916#S4.T3)stress\-tests IntentKV against the*uncompressed*ceiling on the100100BCP queries with the largest Full\-cacheP​TmaxPT\_\{\\max\}\. IntentKV\-88k cuts worst\-case peak request tokens by77\.877\.8–81\.7%81\.7\\%and worst\-case Raw KV Reads by36\.936\.9–92\.6%92\.6\\%across the two backbones, while True Acc stays within0\.960\.96–2\.052\.05points of the uncompressed ceiling\. IntentKV therefore absorbs the heaviest BCP trajectories at a11–22order\-of\-magnitude smaller KV footprint without sacrificing full\-cache accuracy\.

Table 2:IntentKV\-Phase\-2 cuts wall time by up to43%43\\%and Raw KV reads by up to59%59\\%versus compaction baselines on both backbones\(lower is better\)\. Column definitions in[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1); best non\-ceiling cell per column inbold\.Table 3:IntentKV\-88k cuts worst\-case peak request tokens by7878–82%82\\%and worst\-case Raw KV reads by up to93%93\\%against the uncompressed ceiling\.Cohort: the100100BCP queries with the largest uncompressedP​TmaxPT\_\{\\max\}where all three methods complete; symbols defined in[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)\.

### 4\.4Ablations

We ablate the three additions of[Equation˜4](https://arxiv.org/html/2606.09916#S3.E4)on Qwen3\-8B/BCP; full breakdown in[Appendix˜B](https://arxiv.org/html/2606.09916#A2)\. Cross\-turn memory is the dominant signal: replacing𝐌t\\mathbf\{M\}\_\{t\}with the current query reduces True Acc by3\.133\.13points atC=16C\{=\}16k and1\.331\.33atC=8C\{=\}8k\. Cross\-attention and the learned residual each contribute under11point, with the residual adding\+0\.60\+0\.60True Acc at1616k while the rule prior is preferred by0\.600\.60points at the tighter88k budget—the rule scorer saturates the dead\-slot ceiling at88k, while the residual lifts only once1616k frees spare capacity\. A TrimKV head\-to\-head and FRAMES cross\-benchmark validation appear in[Appendices˜D](https://arxiv.org/html/2606.09916#A4)and[C](https://arxiv.org/html/2606.09916#A3)\.

## 5Conclusion

Multi\-turn agents need KV compression that survives cross\-turn intent shifts and composes with radix prefix reuse\. IntentKV factors pruning into a cross\-turn QueryMemory retention head and a sentinel\-slot layout\. On Qwen3\-8B/BCP this recovers Full\-cache True Acc within0\.960\.96points and holds a20\.7%20\.7\\%prefix\-hit rate where compaction baselines collapse to0–3%3\\%; the same dead\-slot substrate transfers4444–46%46\\%wall\-time savings to SnapKV/H2O\. Retention and layout are modular axes, and the substrate accepts any future prompt\-local scorer\. Per\-query budget control and≥70\\geq\\\!70B backbones remain for future work\.

## Limitations

#### No autonomous budget selection\.

IntentKV uses a single global KV budgetCCthat is fixed at deployment time and applied uniformly to every query: the method cannot currently choose its own compression ratio from the trajectory\. Short queries therefore pay an eviction overhead they do not need, and the small subset of queries whose peak request tokens grow far beyond the budget \(the9292–115115k\-token tail in[Table˜3](https://arxiv.org/html/2606.09916#S4.T3)\) is forced through the same compression curve as the median query\. A learned per\-query controller that picksCCfrom early\-turn signals, or a schedule that expands with realized peak usage, would let IntentKV spend memory only on queries that need it; we leave this to future work\.

#### Supervision and scope\.

The Phase\-2 residual is trained only on strict\-cleaned ToolBench multi\-turn traces with substring\-matched future\-action labels, so expected gains are smaller on workloads further from agentic tool use; the rule scorer protects against regression and IntentKV falls back to its heuristic on single\-turn inputs\. IntentKV is validated on Qwen3\-8B and Qwen2\.5\-14B \(Neox\-RoPE GQA\), and the dead\-slot sentinel currently requires fp16/bf16 KV pools\. Other open backbones \(Llama\-3\.1\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib30)\), Mistral\-Nemo/Small, Hermes\-3, GLM\-4, gpt\-oss, Watt\-Tool\) failed prerequisite agentic behaviour on BCP under our serving harness and so cannot isolate KV\-compression effects on this workload; see[Appendix˜G](https://arxiv.org/html/2606.09916#A7)for the smoke study and failure\-mode taxonomy\. Evaluation on≥\\geq70B models is left to future work\.

## References

- J\. Ahn, I\. Seong, A\. Kedia, J\. Kim, H\. Jang, K\. Lee, and Y\. Jeon \(2026\)LookaheadKV: fast and accurate kv cache eviction by glimpsing into the future without generation\.arXiv preprint arXiv:2603\.10899\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Bui, S\. Sharma, S\. Lamba, S\. Mishra, and R\. Ying \(2025\)Cache what lasts: token retention for memory\-bounded kv cache in llms\.arXiv preprint arXiv:2512\.03324\.Cited by:[Appendix D](https://arxiv.org/html/2606.09916#A4.p1.10),[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px3.p1.4)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Chen, X\. Ma, S\. Zhuang, P\. Nie, K\. Zou, A\. Liu, J\. Green, K\. Patel, R\. Meng, M\. Su,et al\.\(2025\)Browsecomp\-plus: a more fair and transparent evaluation benchmark of deep\-research agent\.arXiv preprint arXiv:2508\.06600\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px1.p1.5)\.
- A\. Devoto, M\. Jeblick, and S\. Jégou \(2025\)Expected attention: kv cache compression by estimating attention from future queries distribution\.arXiv preprint arXiv:2510\.00636\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024\)Model tells you what to discard: adaptive kv cache compression for llms\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 22975–22988\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px1.p1.1),[Supervision and scope\.](https://arxiv.org/html/2606.09916#Sx1.SS0.SSS0.Px2.p1.1)\.
- Z\. Guo, S\. Cheng, H\. Wang, S\. Liang, Y\. Qin, P\. Li, Z\. Liu, M\. Sun, and Y\. Liu \(2024\)Stabletoolbench: towards stable large\-scale benchmarking on tool learning of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 11143–11156\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px2.p1.15)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)Gear: an efficient kv cache compression recipe for near\-lossless generative inference of llm\.arXiv preprint arXiv:2403\.05527\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kariyappa and G\. E\. Suh \(2026\)SideQuest: model\-driven kv cache management for long\-horizon agentic reasoning\.arXiv preprint arXiv:2602\.22603\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui \(2024\)Fact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.arXiv preprint arXiv:2409\.12941\.Cited by:[Appendix C](https://arxiv.org/html/2606.09916#A3.p1.1),[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p3.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2606.09916#S3.SS4.SSS0.Px2.p1.6),[§3\.4](https://arxiv.org/html/2606.09916#S3.SS4.p1.1)\.
- H\. Li, R\. He, Q\. Mang, Q\. Zhang, H\. Mao, X\. Chen, H\. Zhou, A\. Cheung, J\. Gonzalez, and I\. Stoica \(2025\)Continuum: efficient and robust multi\-turn llm agent scheduling with kv cache time\-to\-live\.arXiv preprint arXiv:2511\.02230\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Li, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. Hu, W\. Dong, Q\. Li, and L\. Chen \(2024a\)A survey on large language model acceleration based on kv cache management\.arXiv preprint arXiv:2412\.19442\.Cited by:[§1](https://arxiv.org/html/2606.09916#S1.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024b\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§C\.2](https://arxiv.org/html/2606.09916#A3.SS2.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p2.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px3.p1.4)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.Advances in Neural Information Processing Systems36,pp\. 52342–52364\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024\)Kivi: a tuning\-free asymmetric 2bit quantization for kv cache\.arXiv preprint arXiv:2402\.02750\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Packer, V\. Fang, S\. Patil, K\. Lin, S\. Wooders, and J\. Gonzalez \(2023\)MemGPT: towards llms as operating systems\.\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px3.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§G\.3](https://arxiv.org/html/2606.09916#A7.SS3.SSS0.Px2.p1.4)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)Toolllm: facilitating large language models to master 16000\+ real\-world apis\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9695–9717\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px2.p1.15)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px1.p1.5)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§3\.1](https://arxiv.org/html/2606.09916#S3.SS1.p1.9)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context llm inference\.arXiv preprint arXiv:2406\.10774\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21875–21895\.Cited by:[§C\.2](https://arxiv.org/html/2606.09916#A3.SS2.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p2.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px3.p1.4)\.
- W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang \(2026\)A\-mem: agentic memory for llm agents\.Advances in Neural Information Processing Systems38,pp\. 17577–17604\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px1.p1.5)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§G\.3](https://arxiv.org/html/2606.09916#A7.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p1.1)\.
- P\. Zhang, Z\. Liu, S\. Xiao, N\. Shao, Q\. Ye, and Z\. Dou \(2025\)Long context compression with activation beacon\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 101932–101948\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[§C\.2](https://arxiv.org/html/2606.09916#A3.SS2.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p2.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px3.p1.4)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px4.p1.8)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.\(2024\)Sglang: efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[Appendix I](https://arxiv.org/html/2606.09916#A9.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p1.1),[§1](https://arxiv.org/html/2606.09916#S1.p3.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2606.09916#S3.SS4.p1.1),[§3](https://arxiv.org/html/2606.09916#S3.p1.3),[§4\.1](https://arxiv.org/html/2606.09916#S4.SS1.SSS0.Px1.p1.5)\.
- X\. Zhou, W\. Wang, M\. Zeng, J\. Guo, X\. Liu, L\. Shen, M\. Zhang, and L\. Ding \(2024\)DynamicKV: task\-aware adaptive kv cache compression for long context llms\.arXiv preprint arXiv:2412\.14838\.Cited by:[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.09916#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AExperimental Details

#### Session identification and concurrency\.

The session key is resolved in priority order: an explicitsession\_idfield; the token tuple in a designated session span \(typically the first user task message after the chat template\); a hash of the firstP=256P\\\!=\\\!256input tokens; finally, a per\-request key that never matches across requests\. An LRU store of1,0241\{,\}024memories per compressor caps memory; concurrent requests update disjoint keys, so multi\-tenant serving never cross\-contaminates intent\. The cross\-turn EMA decay \([Equation˜2](https://arxiv.org/html/2606.09916#S3.E2)\) isλ=0\.5\\lambda\{=\}0\.5for all reported runs\.

#### Serving setup\.

All BCP runs use SGLang with radix prefix caching enabled whenever the compression layout preserves prefix identity\. Compaction baselines that left\-pack K/V rows are run with radix reuse disabled after eviction to avoid incorrect RoPE positions\. IntentKV uses the same attention kernels as the baselines; only the slot map differs\. The evaluation harness controls this through the KV compaction mode: main comparison baselines use compact apply paths, while IntentKV uses the dead\-slot layout\.

#### Compression settings\.

All compressors are evaluated at nominal retention budgetsC∈\{8192,16384\}C\\in\\\{8192,16384\\\}\. The budget is applied to compressible history at each pruning event, not to the entire prompt\-plus\-generation request\. IntentKV forces the resolved actionable query span to remain live and assigns the remaining budget by score\. TheW=32W\{=\}32buffer is retained only as a bounded query\-tail fallback\. StreamingLLM uses sink size44plus a recent window, SnapKV usesW=32W\{=\}32with kernel size77and max\-pool smoothing, and H2O uses ratio mode with a recent\-window floor\.

#### Training setup\.

The residual head is trained from strict\-cleaned, ToolBench\-only multi\-turn trajectories with future\-action labels over a five\-call lookahead window\. We use AdamW with learning rate10−310^\{\-3\}, weight decay0\.010\.01, gradient accumulation88,3,0003\{,\}000sampled examples per epoch for Qwen3\-8B and5,0005\{,\}000for Qwen2\.5\-14B \(the smaller budget matches Qwen3\-8B’s manifest support;[Section˜3\.5](https://arxiv.org/html/2606.09916#S3.SS5)\), and two epochs\. Loss coefficients areρ=0\.05\\rho=0\.05,γ=0\.01\\gamma=0\.01, andη=0\.001\\eta=0\.001; the BCE positive\-rate clip is\[0\.02,0\.5\]\[0\.02,0\.5\]\. Qwen3\-8B and Qwen2\.5\-14B each train one214,274214\{,\}274\-parameter pruner with the sameD=128D\{=\}128topology\.

## Appendix BComponent Ablations

[Table˜4](https://arxiv.org/html/2606.09916#A2.T4)ablates the three additions of[Equation˜4](https://arxiv.org/html/2606.09916#S3.E4)on Qwen3\-8B/BCP \(inference\-time only on the same checkpoint; the Eff\. Live KV column shifts by at most0\.50\.5k across variants\)\. Cross\-turn memory is the dominant signal: replacing𝐌t\\mathbf\{M\}\_\{t\}with the current query reduces True Acc by3\.133\.13points atC=16C\{=\}16k \(14\.94→11\.8114\.94\\\!\\to\\\!11\.81\) and1\.331\.33atC=8C\{=\}8k \(14\.10→12\.7714\.10\\\!\\to\\\!12\.77\)\. Removing cross\-attention drops True Acc by0\.240\.24and0\.490\.49points, respectively\. Disabling the residual recovers Phase\-1 exactly: the learned correction adds0\.600\.60True Acc at1616k, while at88k the rule prior is preferred by0\.600\.60points—a tight\-budget regime where an unconstrained residual can move score mass away from the rule prior\.

Table 4:Component ablations on Qwen3\-8B / BCP\.Same checkpoint, inference\-time config only\. “Eff\. LK” is*Eff\. Live KV*\(k tokens; defined in[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)\); ablations only shift accuracy\.
## Appendix CCross\-Benchmark Validation on FRAMES

The main results \([Sections˜4\.2](https://arxiv.org/html/2606.09916#S4.SS2)and[4\.3](https://arxiv.org/html/2606.09916#S4.SS3)\) use BrowseComp\-Plus as the primary benchmark\. To verify that IntentKV’s accuracy and efficiency gains transfer beyond BCP’s native agent protocol and corpus, we replicate the agent loop on FRAMES\(Krishnaet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib32)\), an824824\-query multi\-hop question answering suite over Wikipedia developed for the fact\-retrieval / multi\-hop reasoning literature\. We deliberately*adapt*FRAMES rather than wholesale\-port it, so the resulting numbers measure pruner behaviour under a BCP\-style agentic distribution while keeping the FRAMES question pool, gold answers, and gold document URLs unchanged\.

### C\.1Protocol adaptation

Out of the box FRAMES is shipped as a single\-shot QA TSV with no agent harness; the original release uses a free\-form ReAct loop with<search\>/<answer\>text tags\. On Qwen3\-8B we observe this protocol yields a∼\\sim2121%*unparsable*rate \(the model emits an unclosed tag and the judge sees no answer\), which artificially caps accuracy and confounds compressor effects with parser brittleness\. Our adapted FRAMES loop differs from the native release on four axes, each chosen to bring the per\-turn KV\-pressure distribution into a BCP\-compatible regime:

#### \(i\) Native function calling, BCP\-shaped tools\.

We replace text\-tag ReAct with native OpenAI Chat Completions function calling, registering the same two tools used in our BCP runs verbatim:local\_knowledge\_base\_retrievalfor snippet search andget\_documentfor full\-text drill\-down\. Structured tool\-calljsonarguments are robust to mid\-string truncation, dropping the unparsable rate to∼\\sim0%\.

#### \(ii\) BCPQUERY\_TEMPLATEand grader\.

We reuse BCP’sExplanation / Exact Answer / Confidenceprompt verbatim, with inline\[docid\]citations, and grade with the same Qwen3\-32B LLM judge under BCP’sGRADER\_TEMPLATE\([Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)\)\. This makes the resulting accuracy numbers commensurable across BCP and FRAMES without rubric\-level confounds\.

#### \(iii\) Plaintext Wikipedia corpus\.

We build a corpus of2,4792\{,\}479Wikipedia articles—the union of allwikipedia\_link\_\*URLs across the824824test queries—fetched via the officialaction=query&prop=extracts&explaintext=1endpoint and capped at3,0003\{,\}000characters per document\. The cap is chosen so that a typicalget\_documentcall returns roughly700700–800800decoded tokens—substantially shorter than BCP’s∼\\sim77–88k\-character tool returns, but already an order of magnitude longer than the∼\\sim400400\-character Wikipedia\-intro corpus shipped with FRAMES—which is enough to makeget\_documenta meaningful drill\-down on top of the800800\-character snippet preview rather than a no\-op\. After fetching,99\.499\.4% of documents carry non\-empty text\.

#### \(iv\) Stress profile\.

FRAMES questions are short enough that an unconstrained agent answers∼\\sim3030% of queries in one turn, which leaves the KV cache too shallow for compression effects to surface\. We impose a stress profile that pushes the agent into a multi\-turn regime without altering the question or answer:k=8k\\\!=\\\!8retrieved hits per search joined by44injected distractor documents drawn from low\-BM25\-score regions of the corpus, a multi\-hop preamble that requires the agent to maintain an inline evidence ledger, and minimum\-action constraints of33search calls plus11get\_documentcall before answering\. These knobs convert FRAMES into a multi\-turn KV\-pressure benchmark while keeping the gold answers and document URLs unchanged\.

What is intentionally*not*aligned to BCP: the corpus size \(2,4792\{,\}479vs\.∼\\sim100100k docs\), the corpus retrieval modality \(BM25 plaintext vs\. FAISS dense\), the per\-document length, and the question pool itself\. These are dataset identities of FRAMES that the cross\-benchmark check is meant to preserve\.

### C\.2Substrate integration: IntentKV’s dead\-slot eviction is shared by all compressors

All compression methods evaluated below—StreamingLLM, SnapKV, H2O, and IntentKV—execute on the same eviction substrate introduced in[Section˜3\.4](https://arxiv.org/html/2606.09916#S3.SS4): dropped positions are rewritten to a reserved sentinel slot whose K drives softmax weight to zero, and the flashinfer backend strips the sentinel before attention so decode reads only the kept entries while token\-position identity stays intact\. Concretely, the harness defaultsKV\_COMPACTION\_MODEtodead\_slotfor every score\-based method, and the publishedcompact\-layout RoPE\-reindex apply path used in the original StreamingLLM\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib1)\), SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2606.09916#bib.bib2)\), and H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib3)\)implementations is only retained as a reproducibility\-only opt\-in \(it additionally forces\-\-disable\-radix\-cache\)\. This means every method in[Table˜5](https://arxiv.org/html/2606.09916#A3.T5)shares IntentKV’s memory profile and radix prefix\-cache eligibility: the only varying component across rows is the scorer\. The same property holds for the BCP main table; we restate it here so that the FRAMES system metrics can be read against a fair\-substrate background rather than against each baseline’s original published memory layout\.

### C\.3Results atC=12C\{=\}12k

[Table˜5](https://arxiv.org/html/2606.09916#A3.T5)reports True Acc, gold\-document recall, and the system\-side metrics of[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)under the four compressors plus the no\-pruning Full\-cache ceiling, atC=12,288C\\\!=\\\!12\{,\}288tokens,N=824N\\\!=\\\!824per cell\. The FRAMES grader uses the same denominator convention as BCP \(non\-completion counted as wrong\), so the True Acc column is directly comparable with the BCP main table\. Full\-cache is budget\-agnostic and serves as the no\-pruning ceiling; PT and Eff\. LK are reported in10310^\{3\}tokens, Raw and Eff\. KV reads in10610^\{6\}tokens, Wall in seconds, and Turns is the mean ReAct round count\. Wall\-clock aggregates across the824824queries with deterministic decoding \(T=0T\{=\}0\), at most3232tool\-use turns, and an agent thread parallelism of88\.

Table 5:FRAMES atC=12C\{=\}12k\.Best non\-Full\-cache value per column inbold; lower is better in system\-cost columns\.Two effects stand out\. First, on both backbones IntentKV\-Phase\-2 matches or exceeds Full\-cache accuracy while substantially reducing per\-query KV reads:\+0\.73\+0\.73True Acc with2\.6×2\.6\\timesfewer effective KV reads on Qwen3\-8B \(6\.06\.0vs\.15\.615\.6M tokens\), and\+0\.24\+0\.24True Acc with2\.9×2\.9\\timesfewer effective KV reads on Qwen2\.5\-14B \(13\.913\.9vs\.39\.939\.9M tokens\)\. This is consistent with the stress\-profile distractor injection making the attended\-to set noisier than the unprejudiced model can profitably use, so that retention\-aware pruning acts as implicit context denoising in this regime\. Second, against the strongest heuristic baseline, IntentKV\-Phase\-2 dominates every system axis on Qwen3\-8B \(−24\-24% PT,−24\-24% effective live KV,−54\-54% effective KV reads,−41\-41% wall\-clock,−15\-15% ReAct turns\), while on Qwen2\.5\-14B it retains the headline accuracy lead and reduces effective KV reads by−65\-65% relative to H2O at comparable wall\-clock\. StreamingLLM’s lower system numbers on the 14B row come at the price of−5\.10\-5\.10True Acc against the Full\-cache ceiling, illustrating the accuracy/efficiency frontier that the headline metric obscures\.

## Appendix DComparison with TrimKV: Query\-Decay vs\. Positional Time\-Decay

We compare IntentKV\-Phase\-2 against a public TrimKV\(Buiet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib4)\)checkpoint that uses a*positional time\-decay*retention prior trained for math chain\-of\-thought reasoning, while IntentKV uses*query\-decay*conditioned on cross\-turn QueryMemory\. Both are evaluated on Qwen3\-8B/BCP atC=8C\{=\}8k with a matched Eff\. Live KV budget \(∼\\sim10\.110\.1–10\.210\.2k tokens\) so that any accuracy difference reflects the retention prior, not the budget\.[Table˜6](https://arxiv.org/html/2606.09916#A4.T6)shows that TrimKV achieves slightly higher per\-completion correctness \(21\.2521\.25Raw vs\.16\.6716\.67\), but its completion rate collapses to51\.57%51\.57\\%\(vs\.84\.58%84\.58\\%for IntentKV\)\. The result is a3\.143\.14\-point True Acc gap \(10\.96→14\.1010\.96\\\!\\to\\\!14\.10\) once non\-completion is counted as wrong, indicating that on multi\-turn tool\-use trajectories—where the most useful tokens are often the oldest \(initial request, early retrieval, first tool result\)—a time\-decay prior tuned for locally relevant chain\-of\-thought reasoning misaligns with future tool\-call evidence, while QueryMemory\-driven retention tracks it\.

Table 6:TrimKV vs\. IntentKV atC=8C\{=\}8k\.“Eff\. LK” is*Eff\. Live KV*\(k tokens; defined in[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)\)\.
## Appendix ELayout Ablation: Compact vs\. Dead\-Slot

[Table˜7](https://arxiv.org/html/2606.09916#A5.T7)isolates IntentKV’s hole\-preserving \(dead\-slot\) eviction substrate from QueryMemory scoring\. We re\-run SnapKV \(observation\-window scoring\) and H2O \(heavy\-hitter scoring\) on Qwen3\-8B / BCP atC=8C\{=\}8k under two apply paths: the standard compact \(RoPE\-reindex\) path used in each method’s published implementation, and IntentKV’s dead\-slot path, in which dropped positions are redirected to a sentinel slot whose K is filled with−104\-10^\{4\}\([Section˜3\.4](https://arxiv.org/html/2606.09916#S3.SS4)\)\. Budget, scoring kernel, attention backend, prefix\-protect marker, and agent threading are held fixed across the four heuristic rows; only the eviction substrate varies\.

Switching the substrate cuts wall\-time4444–46%46\\%and Raw KV Reads3939–47%47\\%on both baselines, while Eff\. KV Reads moves by less than10%10\\%—confirming that the wall\-time savings flow through preserved radix\-prefix reuse, not tighter retention\. True Acc rises by2\.42\.4–5\.55\.5points on both baselines, since radix\-prefix reuse keeps per\-turn ReAct prompts short and trajectories reach a final answer before exhaustingmax\_tokens\. Under the matched dead\-slot substrate, IntentKV\-Phase\-2 retains a1\.5×1\.5\\timesreduction in Raw and Eff\. KV Reads over SnapKV \(31\.631\.6vs46\.946\.9M Raw;39\.839\.8vs59\.059\.0M Eff\.\) at True Acc within1​σ1\\sigma\(14\.1014\.10vs14\.7014\.70;Δ=0\.35​σ\\Delta\{=\}0\.35\\sigmaon830830queries\), isolating QueryMemory’s retention\-side gain on top of the layout substrate\.

Table 7:Dead\-slot eviction transfers4444–46%46\\%wall\-time and3939–47%47\\%Raw KV Reads savings to H2O / SnapKV under matched scoring\.Qwen3\-8B / BCP atC=8C\{=\}8k; only the eviction substrate varies\. Wall in seconds; Raw / Eff\. in10610^\{6\}KV reads; True in %\.
## Appendix FTraining Curves

[Figure˜5](https://arxiv.org/html/2606.09916#A6.F5)plots the optimisation trajectories for the two Phase\-2 pruners released with IntentKV: the BCE loss converges within the first half of epoch 1 and remains bounded across epoch 2, and the clipped residual gainα\\alphadrifts only mildly from its initialisation, consistent with the rule\-prior floor preventing the residual from dominating the scorer\.

![Refer to caption](https://arxiv.org/html/2606.09916v1/x5.png)Figure 5:Training trajectories for the two trained pruners\.Loss falls quickly and remains bounded; the learned residual gainα\\alpharises from about1\.01\.0to1\.11\.1on both models\.
## Appendix GCross\-Architecture Evaluation Gap

### G\.1Motivation

To check whether IntentKV generalises beyond the Qwen family used in the main evaluation, we ran smoke tests on seven additional open\-source instruction\-tuned models in the 7B–20B range, covering four base\-model families: Meta Llama \(including the NousResearch Hermes\-3 fine\-tune and the watt\-ai watt\-tool\-8B tool\-specialist fine\-tune, both built on Llama\-3\.1\-8B\-Instruct\), Mistral, OpenAI gpt\-oss, and ZhipuAI GLM\.Only Qwen\-family models reliably engage in multi\-step autonomous tool\-calling on BrowseComp\-Plusunder our serving harness; the other backbones fail at the agentic\-behavior level—not at IntentKV’s KV\-compression level—which prevents a meaningful comparison of compression methods on them\.

### G\.2Empirical Findings

[Table˜8](https://arxiv.org/html/2606.09916#A7.T8)summarizes the smoke\-test results\. “Avg\. tools” is per\-query tool\-call count averaged over the smoke set; “Acc” is the judged accuracy on the full830830\-query BrowseComp\-Plus split when smoke succeeded\.

Table 8:Cross\-architecture smoke results on BrowseComp\-Plus\.Models marked×\\timesfail at the agentic\-behavior layer and cannot isolate KV\-compression effects; failure mode is summarised in the last column\.ModelStatusToolsAccFailureQwen3\-8B\-Instruct✓3\.33\.315\.5%15\.5\\%—Qwen2\.5\-14B\-Instruct✓3\.33\.3——Qwen2\.5\-7B\-Instruct✓3\.33\.3——watt\-tool\-8B×\\times2\.02\.04\.1%4\.1\\%1\-shot biasLlama\-3\.1\-8B\-Instruct×\\times0\.00\.0—No toolsMistral\-Nemo\-2407×\\times0\.00\.0—No toolsMistral\-Small\-3\.2\-24B×\\times0\.00\.0—No toolsHermes\-3\-Llama\-3\.1\-8B×\\times0\.00\.0—Forced\-onlygpt\-oss\-20b \(MXFP4\)×\\times——No FP4 HWgpt\-oss\-20b\-bf16×\\times0\.00\.0—No toolsGLM\-4\-9B\-Chat×\\times0\.00\.0—Parser
### G\.3Failure Mode Taxonomy

We categorize the observed failures into four structural types\.

#### F1: Refusal to use tools\.

Llama\-3\.1\-8B\-InstructandMistral\-Nemo\-Instruct\-2407load and respond fluently, but*never*emit a tool\-call token on BCP queries; they instead hallucinate direct answers from parametric memory\. As an example, the correct answer to one query is a specific Florence hat\-shop brand from the 1930s, yet Mistral\-Nemo answers “Explanation: The individual who meets all the given criteria isDr\. Jane Smith\. …” — no retrieval was attempted; the entity was fabricated\. We attribute the gap to differences in post\-training: Qwen’s instruction tuning explicitly includes multi\-turn ReAct\(Yaoet al\.,[2022](https://arxiv.org/html/2606.09916#bib.bib25)\)trajectories and agentic data, whereas the Llama\-3\.1 and Mistral\-Nemo post\-training prioritizes single\-turn helpfulness\.

The pattern persists at larger scale and despite native tool\-call training\.Mistral\-Small\-3\.2\-24B\-Instruct\-2506—a 24B\-parameter model that ships an\[AVAILABLE\_TOOLS\] / \[TOOL\_CALLS\] / \[TOOL\_RESULTS\]dialogue protocol and a Tekken tokenizer with built\-in function\-calling support—also returns zero tool calls on the BCP smoke set across three prompt configurations \(default, “MUST search” system prompt, and the WebSailor agent persona at temperature0\.70\.7\)\. In a controlled probe outside the BCP harness, the same model*does*emit a well\-formed\[TOOL\_CALLS\]on a short single\-fact query \(“find the SGLang v0\.4 release date”\), so the failure is not a lack of capability but a lack of propensity to initiate retrieval when the question is presented in BCP’s long multi\-criterion format\. To eliminate infrastructure as a confounder we resolved a SGLang↔\\leftrightarrowmistral\-common interop bug \(the OpenAI protocol layer generatedcall\_<24hex\>tool\-call IDs that mistral\-common’s request validator rejects as non\-conformant to its 9\-character alphanumeric schema, breaking the second turn of every multi\-turn loop with a500500\)\. After this fix multi\-turn round\-trips succeed end\-to\-end at the protocol level, yet the model still does not spontaneously initiate retrieval on BCP queries—confirming that the gap sits at the agentic\-behavior layer, not at the serving layer\.

#### F2: Single\-shot tool use\.

watt\-tool\-8B, a specialist fine\-tune of Llama\-3\.1\-8B\-Instruct for BFCL\-style function calling\(Patilet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib31)\), exhibits a rigid behavior pattern: exactly one retrieval call followed by one document fetch followed by a final answer \(avg\.1\.01\.0retrieval \+1\.01\.0fetch per query\)\. The pattern reflects the BFCL training data distribution, which is dominated by single\-turn and two\-turn benchmarks, and cannot support BCP’s typical 3–5 hop information need\. The resulting4\.1%4\.1\\%True Acc—well below the15\.5%15\.5\\%Qwen3\-8B reference in[Table˜8](https://arxiv.org/html/2606.09916#A7.T8)under the same harness—demonstrates that specialized function\-calling fine\-tunes can lock the model into a single\-turn behaviour profile incompatible with multi\-hop agent workloads, even when the upstream base \(Llama\-3\.1\-8B, itself an F1 zero\-tool case above\) lacks autonomous tool use to begin with\.

#### F3: Hardware\-bound quantization mismatch \(not a model deficiency\)\.

OpenAI’sgpt\-oss\-20bis released in MXFP4 and requires Hopper\-class \(H100/H200/B200\) tensor cores for native FP4 execution;*our 80 GiB A100 cluster has no FP4 tensor cores and therefore cannot serve the native release at all*, so it appears in[Table˜8](https://arxiv.org/html/2606.09916#A7.T8)as the first rowgpt\-oss\-20b \(MXFP4\)with theToolscolumn left blank—the model was literally not runnable in our environment\. Lacking access to Hopper\-class hardware, we additionally tried thelmsys/gpt\-oss\-20b\-bf16community MXFP4→\\rightarrowbf16 re\-upload as a proxy \(second row\)\. The proxy never emits a tool call on BCP queries and produces only short free\-text refusals that degenerate into repetitive token loops on continuation \(e\.g\., “\[ \! \] \[ \! \] \[ \! \] …”\), which we attribute to precision loss during the community dequantization—specifically in the layers that emit OpenAI’s Harmony channel\-control tokens \(<\|channel\|\>analysis\|\>,<\|channel\|\>final\|\>\)\. Both rows therefore reflect infrastructure constraints on our side rather than deficiencies in the underlying model; the native MXFP4 release remains untested on BCP and is not evidence for or against gpt\-oss\-20b’s BCP capability\.

#### F4: Parser / chat\-template incompatibility\.

GLM\-4\-9B\-Chatproduced parser\-level garbage in our SGLang deployment\. The model emits structured tool calls in its own proprietary format, but theglm45parser in SGLang is calibrated for the GLM\-4\-MoE / GLM\-4\.5 release—a different tokenizer and chat\-template family\. Adapting the parser to the GLM\-4\-9B dense variant would require approximately 100–150 lines of detector code plus chat\-template adaptation, which we leave to future work\. The same parser\-incompatibility risk applies to other promising agent\-tuned models \(Cohere Command R\-7B, IBM Granite\-3\.1\-Instruct, InternLM2\.5\) that currently lack native SGLang detectors\.

### G\.4Forcing\-Prompt Robustness Probe

The four failure modes above are observed under each model’s default system prompt\. We separately check whether stronger system prompts recover F1\-class refusals, since otherwise the cross\-architecture gap could be dismissed as a prompt\-engineering artifact\. We re\-ran two representative models under explicit forcing variants—a “you MUST call search before answering” system message, the WebSailor agent persona, and a*tool\-choice*\-style preamble that directly instructs the model to emit a tool call on the first turn\.Mistral\-Small\-3\.2\-24Breturned zero tool calls across all forcing variants on the same three smoke queries on which it failed by default—under temperature0\.00\.0the model collapses into degenerate template loops \(“Your Answer: \{your answer\} \| Confidence: \.\.\.”\) and under temperature0\.70\.7it still apologizes without retrieving\.Hermes\-3\-Llama\-3\.1\-8B\(NousResearch\), a Llama\-3\.1\-8B fine\-tune marketed for tool use, behaved more leniently but still unreliably: across seven prompt configurations \(plain, chat\-template override,*tool\-choice*forcing,*direct\-prompt*, and*must\-search*\) the best result was a single tool call on11of33smoke queries \(*direct\-prompt*\), with*must\-search*yielding zero across all three\. Neither model approached the33\+ spontaneous calls per query that Qwen issues at default prompting\. We therefore include the Hermes\-3 row as*Forced\-only*in[Table˜8](https://arxiv.org/html/2606.09916#A7.T8)to signal that any tool\-call activity observed required prompt\-level coercion, and we read this collectively as evidence that the cross\-architecture gap is not a prompt\-engineering artifact: where forcing prompts work in isolation, they do not produce the consistent multi\-turn retrieval behaviour that BCP measures\.

### G\.5Why This Gap Exists

The common pattern across F1–F4 is that BCP demands a behavior profile—autonomous multi\-step retrieval over multi\-hop trivia questions—which few open\-source post\-training recipes explicitly target\. Existing tool\-call benchmarks such as BFCL are dominated by single\-turn and two\-turn function\-selection tasks where the system prompt already implies which tool to call; they do not stress the agent’s ability to*initiate*retrieval*spontaneously*when faced with a knowledge gap\. The Qwen team’s post\-training, by contrast, explicitly includes ReAct\-style multi\-turn traces, ToolLLaMA\-derived agentic data, and BrowseComp\-adjacent research\-style trajectories\.

Notably, this gap is*not*a deficiency in the base models’ language understanding: every failed candidate passes BFCL or similar benchmarks with reasonable scores\. The gap is indistribution\-conditional agentic behavior, which appears to require explicit training\-data investment that, among the open\-source families we tested, only Qwen has made\.

### G\.6Caveats

We emphasize three points\.

- •The claim is restricted to our sample\.We tested seven non\-Qwen models; this is not an exhaustive evaluation of the open\-source ecosystem\. Closed\-source models \(GPT\-4\.1, Claude, Gemini\) likely exhibit different behavior, but their closed nature prevents the KV\-cache\-level access required by our method\.
- •Prompt\-engineering mitigations are orthogonal and, in our experience, largely ineffective\.We empirically attempted aggressive system prompts \(“You MUST call retrieval at least three times before answering”; the WebSailor agent persona; tool\-choice\-style directives\) on the F1 family and found that they do not reliably recover BCP behavior in the failing models—Mistral\-Small\-3\.2\-24B remained at zero tool calls across all forcing variants, and Hermes\-3 reached at most a single call on11of33smoke queries \([Section˜G\.4](https://arxiv.org/html/2606.09916#A7.SS4)\)\. In principle, even sufficiently aggressive forcing could elicit some tool use; however, the resulting traces would no longer be comparable across models—they would reflect the strength of the forcing prompt rather than the model’s intrinsic agentic capability, conflating prompt\-engineering effects with IntentKV’s intrinsic benefit\. We therefore exclude both natural and forced runs of the failing models from the main evaluation\.
- •This is not a critique of the failing models\.Llama\-3\.1, Mistral\-Nemo, GLM\-4\-9B\-Chat, and others remain capable general\-purpose assistants; their training simply does not target the specific agentic behaviour BCP requires\. Closing this gap in future open\-source releases would allow a more thorough test of IntentKV’s backbone\-agnostic claim\.

## Appendix HExtended Algorithmic Details

#### Compression event\.

Given requestrr, slot map𝐒r\[0:N\)\\mathbf\{S\}\_\{r\}\[0\{:\}N\), budgetCCand dead slots†s^\{\\dagger\}, IntentKV skips compression whenN≤CN\\leq C\. Otherwise it forms the live\-position setℒ=\{j:𝐒r​\[j\]≠s†\}\\mathcal\{L\}=\\\{j:\\mathbf\{S\}\_\{r\}\[j\]\\neq s^\{\\dagger\}\\\}, resolves the decision query span, and splitsℒ\\mathcal\{L\}into forced positionsℱ\\mathcal\{F\}and candidate positions𝒜\\mathcal\{A\}\. Forquery, the scorer uses only the currentEnc​\(𝐪t\)\\mathrm\{Enc\}\(\\mathbf\{q\}\_\{t\}\)\. Formemoryandlearnable, the scorer first derives a session id, updates𝐌t\\mathbf\{M\}\_\{t\}, computes[Equation˜3](https://arxiv.org/html/2606.09916#S3.E3), and optionally adds the residual[Equation˜4](https://arxiv.org/html/2606.09916#S3.E4)\. Tensor\-parallel ranks all\-reduce the resulting score vector\. The kept set is𝒦=ℱ∪topk​\(𝒜,max⁡\(0,C−\|ℱ\|\)\)\\mathcal\{K\}=\\mathcal\{F\}\\cup\\mathrm\{topk\}\(\\mathcal\{A\},\\max\(0,C\-\|\\mathcal\{F\}\|\)\)\. The implementation persists a boolean valid mask, redirects newly dropped positions tos†s^\{\\dagger\}, and leaves all kept slot ids unchanged\.

#### Prefix\-safe free\.

For each newly dropped positionjj, the old slotu=𝐒r​\[j\]u=\\mathbf\{S\}\_\{r\}\[j\]is returned to the allocator only ifjjis outside the radix\-protected prefix,u≠s†u\\neq s^\{\\dagger\}, anduuis absent from the final kept\-slot set\{𝐒r​\[i\]:i∈𝒦\}\\\{\\mathbf\{S\}\_\{r\}\[i\]:i\\in\\mathcal\{K\}\\\}\. This alias check is required because a request may already contain redirected positions or shared prefix slots\.

#### Session id derivation\.

The session key is chosen by the first available source: an explicitsession\_id, then the token tuple in a designated session span, then a hash of the firstPPinput tokens\. If the request has no token ids, the implementation uses a request\-local key; this preserves correctness by avoiding accidental memory sharing across unrelated embedding\-input requests\.

#### Phase\-2 training step\.

Per training row, the trainer extracts post\-RoPE K and query\-span Q from the frozen model, averages the Q rows to formEnc​\(𝐪t\)\\mathrm\{Enc\}\(\\mathbf\{q\}\_\{t\}\), and updates the sameQueryMemoryused at inference\. It then computes the rule score and residual features over the remaining candidate positions, builds the cross\-attention memory from query\-span K\-vectors, obtains MLP logits, and applies[Equation˜8](https://arxiv.org/html/2606.09916#S3.E8)\(session replay order matches[Section˜3\.5](https://arxiv.org/html/2606.09916#S3.SS5)\)\.

## Appendix IArtifact Licenses and Intended Use

We summarise the licenses, intended use, and our use of every external artifact referenced in the main text and appendices\. All artifacts are used in accordance with their original licenses and for research purposes consistent with their stated intended use\.

#### Language models\.

Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib16)\)andQwen2\.5\-14B\(Qwenet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib17)\), used as the primary backbones, are released by Alibaba under the Apache License 2\.0;Qwen3\-32B, used as the LLM\-as\-a\-judge grader, is released under the same license\. The smoke\-test models in[Appendix˜G](https://arxiv.org/html/2606.09916#A7)carry the following licenses:Llama\-3\.1\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib30)\)under the Llama 3\.1 Community License Agreement;Hermes\-3\-Llama\-3\.1\-8B\(NousResearch\) inherits the same Llama 3\.1 Community License;Mistral\-Nemo\-Instruct\-2407andMistral\-Small\-3\.2\-24B\-Instruct\-2506under the Apache License 2\.0;watt\-tool\-8Bunder the Apache License 2\.0 \(its model card declares Apache\-2\.0; the upstream base model is Llama\-3\.1\-8B\-Instruct, which itself carries the Llama 3\.1 Community License\);gpt\-oss\-20bunder the Apache License 2\.0;GLM\-4\-9B\-ChatunderThe glm\-4\-9b License\(HF license tagglm\-4\)\. All models are used for inference\-only evaluation; the base weights are never updated\. Our use is consistent with each model’s release statement of academic/research use\.

#### Datasets and benchmarks\.

BrowseComp\-Plus\(Chenet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib18)\), the primary evaluation suite, is released under the MIT License; we use the public 830\-query split and the released document corpus without modification beyond the per\-query retrieval cache described in[Section˜4\.1](https://arxiv.org/html/2606.09916#S4.SS1)\.FRAMES\(Krishnaet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib32)\)\(Google\) is released under the Apache License 2\.0; we use its 824\-query test split, gold answers, andwikipedia\_linkURLs unchanged \([Section˜C\.1](https://arxiv.org/html/2606.09916#A3.SS1)\)\.ToolBench\(Qinet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib21)\)and its strict\-cleaned re\-releaseStableToolBench\(Guoet al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib22)\)are released under the Apache License 2\.0 and used only for training the residual head \([Section˜3\.5](https://arxiv.org/html/2606.09916#S3.SS5)\); no ToolBench content appears in the evaluation distribution\. The Wikipedia corpus constructed in[Section˜C\.1](https://arxiv.org/html/2606.09916#A3.SS1)is retrieved from the official MediaWiki API and is governed by theCC BY\-SA 4\.0license; we redistribute only the URL list and SHA\-256 hashes of the cached extracts rather than the article text itself\.

#### Software and serving stacks\.

SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2606.09916#bib.bib24)\)andvLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.09916#bib.bib15)\), on which our compression substrate is implemented and evaluated, are released under the Apache License 2\.0\. The referenceTRIM\-KVcheckpointngocbh/TrimKV\-Qwen3\-8B\-Math\(Buiet al\.,[2025](https://arxiv.org/html/2606.09916#bib.bib4)\)used in[Appendix˜D](https://arxiv.org/html/2606.09916#A4)is distributed by its authors on Hugging Face under the Apache License 2\.0\.

#### Released artifact\.

Our code and trained residual\-head checkpoints will be released under a permissive open\-source license \(MIT\) consistent with our upstream dependencies\. The release contains only model weights for the residual head \(a214,274214\{,\}274\-parameter MLP\) and integration code, with no redistribution of the base LLM weights, ToolBench corpus, or BrowseComp\-Plus content\.

## Appendix JAI Assistant Use Statement

AI assistants were used for language polishing, checklist interpretation, and minor writing assistance\. All technical claims, experimental results, code, and final submission content were reviewed and approved by the authors\.