Learning Agent Execution for KV-Cache Management in Agentic Serving

# Learning Agent Execution for KV-Cache Management in Agentic Serving
Source: [https://arxiv.org/html/2608.14624](https://arxiv.org/html/2608.14624)
Research Full

Rui Zhang1, Chaeeun Kim1, Shaoting Feng2, Kuntai Du3, Yuhan Liu3, Yi Zhong1, Cheng\-Wei Ching1, Junchen Jiang3, Liting Hu1

###### Abstract\.

Multi\-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents\. Across these workflows, every agent repeatedly executes a fixed context consisting of system prompts, tool definitions, and few\-shot examples, creating substantial opportunities for KV\-cache reuse\. Existing LLM serving systems, however, manage KV\-cache reactively using prefix caching and recency\-based replacement, causing reusable agent contexts to be evicted before their next invocation and forcing repeated recomputation\. We presentCacheScout, an agent\-aware KV\-cache runtime layer for multi\-agent LLM serving\. The key insight is that future KV\-cache reuse is governed by agent execution semantics rather than cache recency alone\.CacheScoutcaptures these semantics by learning agent execution transitions online, without requiring predefined workflow graphs or offline training, and uses the learned execution model to guide both cache eviction and proactive prefetching while leaving the serving critical path unchanged\. We implementCacheScouton top of vLLM\. Across representative real\-world multi\-agent workloads,CacheScoutimproves KV\-cache hit rate by 10–18 percentage points, reduces mean TTFT by 18–45%, lowers mean per\-turn latency by 29–38%, and increases peak throughput by up to 57%\. These benefits also generalize to larger models, reducing TTFT by up to 54% while sustaining 37% higher throughput\.

LLM serving, multi\-agent systems, KV cache, cache management, prefix caching, machine learning systems

## 1\.Introduction

![Refer to caption](https://arxiv.org/html/2608.14624v1/x1.png)Figure 1\.Multi\-agent travel planning use case\.Multi\-agent LLM systems have rapidly become the dominant deployment paradigm for production AI services, powering applications such as software engineering assistants\(Yanget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib29); Jimenezet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib36); Zhanget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib37)\), enterprise knowledge assistants\(Lewiset al\.,[2020](https://arxiv.org/html/2608.14624#bib.bib38); Gaoet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib30); Syarubany and Yoo,[2025](https://arxiv.org/html/2608.14624#bib.bib39)\), and deep research agents\(Zhanget al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib31); Huanget al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib40); Duet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib41)\)\. Instead of relying on a single LLM call, these applications decompose each user request into subtasks handled by a sequence of specialized agents that collaborate to complete complex tasks\. A planner coordinates the workflow, domain\-specific agents perform specialized reasoning or tool use, and their results are integrated into a final response\.

As shown in Figure[1](https://arxiv.org/html/2608.14624#S1.F1), consider a user asking an AI assistant to plan a trip to Japan\. The planner first asks a Travel Agent to search for flights, then a Hotel Agent to recommend places to stay, and finally a Restaurant Agent to find local restaurants before putting everything together\. Every time the Travel Agent is invoked, it begins with the same fixed context, including a system prompt such asYou are a travel expert\. Find the best flight options based on the user’s budget and schedule\.together with the same flight\-search tool definitions\. The Hotel Agent likewise begins withYou are a hotel recommendation expert\.\.\.together with the same hotel\-search tools\. The task\-specific user request is appended after this fixed context\. Since this fixed context rarely changes, the KV\-cache generated for it can be reused whenever the same agent is invoked, even across different user sessions\. As shown in Figure[2](https://arxiv.org/html/2608.14624#S1.F2), this recurring fixed context accounts for 53–62% of all prompt tokens across the four multi\-agent workloads evaluated in this paper, making it a substantial opportunity for KV\-cache reuse\.

However, today’s LLM serving systems, such as vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib2)\), SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib3)\), and TensorRT\-LLM\(NVIDIA,[2023](https://arxiv.org/html/2608.14624#bib.bib35)\), do not fully exploit this opportunity because their cache management is largely reactive, relying on prefix caching and recency\-based cache replacement\. Returning to our example, after the Travel Agent finishes searching for flights, the planner continues with the Hotel Agent and the Restaurant Agent\. As their KV blocks gradually fill the GPU cache, the KV\-cache corresponding to the Travel Agent’s fixed context is eventually evicted because it has not been accessed recently\. Later, after reviewing the hotel options, the planner decides to compare another flight itinerary and invokes the Travel Agent again\. Since its reusable fixed context has already been evicted, the serving system must recompute the same prefix from scratch\. This pattern repeats throughout a multi\-agent workflow\. Although a reusable agent anchor may remain idle while other agents execute, it is likely to be reused when the same agent is invoked again\. Recency\-based cache policies cannot distinguish this temporary idle period from genuinely cold data, leading them to evict precisely the KV blocks that are most valuable to future execution\.

\\noindentparagraph

The problem\.Closing the gap requires the cache to retain KV blocks that will be used again, instead of simply keeping the blocks that were used most recently\. This is much harder than it appears\. Today’s serving engines, such as vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib2)\)and SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib3)\), manage KV\-caches as anonymous blocks indexed by content hashes\. They know which block was accessed, but they do not knowwhoused it\. For example, the engine cannot tell whether a block belongs to the Travel Agent, the Hotel Agent, or the Restaurant Agent\. As a result, it cannot tell whether a block is likely to be reused in the near future or can be safely evicted\.

Even if the cache knew which agent produced each block, deciding what comes next is still difficult\. Modern agent frameworks, such as AutoGen\(Wuet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib23)\)and LangGraph\(LangChain,[2024](https://arxiv.org/html/2608.14624#bib.bib33)\), do not follow a fixed workflow\. Instead, the next agent is chosen by the LLM at runtime\. The planner may call the Travel Agent again, switch to another agent, or finish the task altogether\. Therefore, future reuse cannot be determined from a predefined workflow\. Recent systems sidestep this difficulty by requiring the workflow to be known in advance, either as a declared execution graph or developer\-provided annotations\(Panet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib4); Linet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib6); Guoet al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib7)\), or by pinning the current session’s KV\-cache across tool calls\(Liet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib5); Abhyankaret al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib8)\)\. Neither approach helps when the next agent is decided dynamically at runtime\.

Finally, any prediction must be extremely fast\. Cache eviction happens whenever GPU memory becomes full and is part of the serving hot path\. Learning\-based cache replacement policies from the storage literature\(Songet al\.,[2020](https://arxiv.org/html/2608.14624#bib.bib42); Vietriet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib43); Yanget al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib44); Beckmannet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib45)\)improve prediction quality through feature extraction and model inference, but spending milliseconds to make a better cache decision would easily cost more than the prefill computation it saves\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x2.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x3.png)

Figure 2\.Share of prompt tokens occupied by recurring fixed context \(system prompts, tool definitions, skills, and examples\) in four representative multi\-agent workloads: GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.14624#bib.bib26)\), MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib24)\), GAIA\(Mialonet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib25)\), and SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib36)\)\.\\noindentparagraph

Our solution\.We address this problem by introducingCacheScout, a lightweight KV\-cache runtime layer that bridges the gap between agent frameworks and serving engines\. Instead of managing KV blocks solely based on past accesses,CacheScoutobserves agent execution, learns how agents are dispatched at runtime, predicts which agent is likely to execute next, and uses this information to guide cache eviction and prefetching\. In this way, cache management becomes aware of future agent reuse rather than merely reacting to previous accesses\.

CacheScoutrealizes this design through three techniques\. \(1\)Transition learner\.It models agent execution as an online first\-order Markov chain\. For each request, it identifies the agent from the prompt\-prefix fingerprint and updates a single transition counter\. No offline training or predefined workflow graph is required \(Sec\.[3\.2](https://arxiv.org/html/2608.14624#S3.SS2)\)\. \(2\)Survival\-probability scorer\.It estimates the probability that each cached anchor will be reused from the learned transition matrix and combines this probability with recency and reconstruction cost to rank eviction candidates\. As a result, reusable anchors are retained while execution remains predictable, and the policy naturally falls back toward LRU when it is not \(Sec\.[3\.3](https://arxiv.org/html/2608.14624#S3.SS3)\)\. \(3\)Background prefetch\.It predicts the next agent from the same transition matrix and warms the corresponding anchor between requests, leaving the serving critical path unchanged \(Sec\.[3\.4](https://arxiv.org/html/2608.14624#S3.SS4)\)\. Together, these techniques transform the KV\-cache from a passive, LRU\-based component into an active, agent\-aware runtime\.

\\noindentparagraph

Summary of results\.We implementCacheScoutas a lightweight KV\-cache runtime layer on top of vLLM and compare it against vanilla vLLM and Continuum\. We evaluateCacheScouton four representative real\-world multi\-agent workloads using a six\-agent supervisor framework with Llama\-3\.1\-8B\-Instruct\.CacheScoutimproves KV\-cache hit rate by 10–18 percentage points, reaching 81–85% across all workloads, reduces mean TTFT by 18–45% and mean per\-turn latency by 29–38%, and delivers 19–57% higher peak throughput\. Under the same latency budget, it sustains1\.71\.7–12×12\\timesthe load of vanilla vLLM\. These gains also extend to larger models\. On Qwen3\-235B\-A22B, a 235B\-parameter mixture\-of\-experts model,CacheScoutreduces mean TTFT by 33–54% on SWE\-bench and delivers 37% higher throughput\. We also show thatCacheScoutremains robust across different cache capacities and workload conditions while incurring negligible runtime overhead\.

\\noindentparagraph

Contributions\.This paper makes the following contributions\.

- •We identify a new challenge in multi\-agent LLM serving: existing prefix caches are reactive and cannot exploit future agent reuse\.
- •We proposeCacheScout, a lightweight KV\-cache runtime layer that bridges agent frameworks and serving engines, enabling proactive, agent\-aware cache management\.
- •We design and implementCacheScoutwith three techniques: online transition learning, survival\-probability\-based eviction, and asynchronous predictive prefetching, while keeping runtime state and overhead low\.
- •We implementCacheScouton top of vLLM and show significant improvements in cache hit rate, TTFT, and throughput across representative multi\-agent workloads\.

## 2\.Motivation and Challenges

### 2\.1\.Background

\\noindentparagraph

Multi\-agent workflows\.Modern LLM applications increasingly organize complex tasks as multi\-agent workflows\. Instead of relying on a single model invocation, a planner decomposes a user request into a sequence of specialized agents, each responsible for a particular task such as planning, retrieval, coding, or tool use\. These agents may follow a predefined execution graph or be dynamically selected by an LLM at runtime\. Frameworks such as AutoGen\(Wuet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib23)\), LangGraph\(LangChain,[2024](https://arxiv.org/html/2608.14624#bib.bib33)\), CrewAI\(CrewAI,[2024](https://arxiv.org/html/2608.14624#bib.bib34)\), and OpenAI Swarm\(OpenAI,[2024](https://arxiv.org/html/2608.14624#bib.bib28)\)have made this execution model increasingly common in production systems\.

\\noindentparagraph

Agent anchor\.Although different agents perform different tasks, every invocation of the same agent begins with nearly identical prompt prefixes\. These prefixes typically consist of a system prompt describing the agent’s role, tool definitions specifying available APIs, skill descriptions, and optionally few\-shot examples\. Throughout this paper, we refer to this reusable prompt prefix as the agent anchor\. Since an agent anchor rarely changes across invocations or user sessions, the KV\-cache generated for it can be reused whenever the same agent is executed again\.

\\noindentparagraph

Prefix caching\.Modern serving systems, including vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib2)\), SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib3)\), TensorRT\-LLM\(NVIDIA,[2023](https://arxiv.org/html/2608.14624#bib.bib35)\), LMCache\(Liuet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib1)\), and NanoFlow\(Zhuet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib32)\), reduce redundant prefill computation through prefix caching\. During the prefill stage, the model computes KV tensors for every prompt token\. If a later request shares an identical prompt prefix, these KV tensors can be reused instead of recomputing the entire prefix\. Existing systems differ mainly in how they identify identical prefixes\. For example, vLLM partitions prompts into fixed\-size blocks \(16 tokens by default\) and identifies reusable KV blocks through block hashing, whereas SGLang uses RadixAttention, which organizes prompt prefixes in a radix tree to enable token\-level prefix matching\. Despite these implementation differences, all existing systems identify reusable KV\-caches solely according to prompt content\.

\\noindentparagraph

Reactive cache management\.Existing prefix caches are fundamentally reactive\. A KV block becomes reusable only after it has already been requested and inserted into the cache\. Likewise, cache replacement policies, such as LRU\(Mattsonet al\.,[1970](https://arxiv.org/html/2608.14624#bib.bib46)\), LFU\(Ahoet al\.,[1971](https://arxiv.org/html/2608.14624#bib.bib47)\), ARC\(Megiddo and Modha,[2003](https://arxiv.org/html/2608.14624#bib.bib27)\), and LIRS\(Jiang and Zhang,[2002](https://arxiv.org/html/2608.14624#bib.bib48)\), retain KV blocks solely according to past accesses\. The serving engine operates only on content\-addressed KV blocks\. It neither knows which agent produced a cached block nor whether that agent is likely to be invoked again\. As multi\-agent workflows become increasingly dynamic, this reactive design leaves many reuse opportunities unexploited\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x4.png)\(a\)Block reuse counts\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x5.png)\(b\)Anchor ratio vs depth\.

Figure 3\.Reuse concentration in representative multi\-agent workloads\. Agent\-anchor blocks exhibit substantially higher reuse than session\-history blocks, while recurring fixed context remains a significant fraction of each prompt throughout multi\-turn interactions\.
### 2\.2\.New Insights

The reactive design above leaves the reuse opportunities of multi\-agent workloads unexploited\. Recovering them requires the cache to act before reuse happens, and whether that is possible depends on the structure of multi\-agent execution itself\. We therefore ask three questions\. First, which cached KV blocks are most valuable to preserve during eviction? Second, is future agent execution predictable enough to act on? Third, can the execution structure be specified before deployment, or must it be learned at runtime? We answer these questions through a measurement study over representative multi\-agent workloads, and the resulting insights directly shape the design ofCacheScout\.

#### 2\.2\.1\.Which KV blocks should be protected?

A natural question is whether all cached KV blocks are equally valuable\. If future reuse is uniformly distributed across all KV blocks, conventional cache management policies may already be sufficient\. On the other hand, if a small subset of KV blocks contributes disproportionately to future reuse, selectively protecting those blocks could significantly improve cache efficiency\.

To answer this question, we first measure how often different types of KV blocks are reused\. Figure[3\(a\)](https://arxiv.org/html/2608.14624#S2.F3.sf1)reports the average number of future reuses per KV block across four representative multi\-agent workloads\. Agent\-anchor blocks are reused 49–173 times on average because every invocation of the same agent accesses the same fixed context, both within and across user sessions\. In contrast, session\-history blocks belong to a single conversation and are reused only 12–15 times on average\. Overall, agent anchors receive 4–13×\\timesmore reuse than session\-specific history\.

Reuse frequency alone, however, does not determine cache value\. We therefore quantify how much of each prompt is occupied by reusable agent anchors\. We define theagent anchor ratio\(ϕ\\phi\) as

\(1\)ϕ=system prompts\+tool definitions\+skills\+few\-shottotal prompt tokens\.\\phi=\\frac\{\\text\{system prompts\}\+\\text\{tool definitions\}\+\\text\{skills\}\+\\text\{few\-shot\}\}\{\\text\{total prompt tokens\}\}\.
Intuitively,ϕ\\phimeasures the fraction of each prompt occupied by the reusable agent anchor rather than session\-specific conversation history\.

Figure[3\(b\)](https://arxiv.org/html/2608.14624#S2.F3.sf2)plotsϕ\\phiover the first half of a median session\. Although the reusable fraction gradually decreases as conversation history accumulates, agent anchors consistently occupy a substantial portion of every prompt: even a dozen turns into a session, recurring anchor content still accounts for 43–60% of every prompt\.

\\noindentparagraph

Takeaway\. Agent anchors dominate KV\-cache reuse for two reasons\. They are reused substantially more frequently than session\-specific history and occupy a large fraction of every prompt\. Protecting anchor blocks therefore provides the greatest opportunity for improving KV\-cache reuse\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x6.png)\(a\)Entropy reductionRR\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x7.png)\(b\)Online prediction accuracy\.

Figure 4\.Execution predictability in representative multi\-agent workloads\. Execution remains structured across diverse coordination topologies, enabling the online transition learner to quickly achieve high next\-agent prediction accuracy\.
#### 2\.2\.2\.Is future agent execution predictable?

Having identified which KV blocks are most valuable to preserve, we next ask whether future agent execution is predictable\.

Since proactive cache management only requires predicting the next agent invocation, we quantify predictability using the reduction in uncertainty of the next executed agent\. LetAtA\_\{t\}denote the agent executed at steptt\. We measure the relative reduction in conditional entropy,

\(2\)R=1−H​\(At\+1∣At\)H​\(At\+1\),R\\;=\\;1\-\\frac\{H\(A\_\{t\+1\}\\mid A\_\{t\}\)\}\{H\(A\_\{t\+1\}\)\},whereR=0R=0indicates that the next agent is independent of the current agent, whileR=1R=1indicates that the next agent is completely determined by the current one\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x8.png)Figure 5\.Transition matricesP​\(next∣current\)P\(\\mathrm\{next\}\\mid\\mathrm\{current\}\)learned from four coordination topologies built on the same six\-agent framework\. The same agents produce dramatically different execution structures under different coordination behaviors, so transition models cannot be specified offline and must be learned at runtime\.Figure[4\(a\)](https://arxiv.org/html/2608.14624#S2.F4.sf1)reportsRRfor four representative coordination topologies\. Pipeline exhibits deterministic execution \(R=1\.0R=1\.0\) and the moderated Debate topology remains highly structured \(R=0\.78R=0\.78\), while SelectorGroupChat, where the next agent is dynamically selected by an LLM, still achievesR=0\.57R=0\.57\. In other words, conditioning on the current agent eliminates 57% of the uncertainty in the next agent\. Only the Random topology shows little learnable structure \(R=0\.12R=0\.12\)\. Overall, the three practical coordination patterns all exhibit substantial predictability despite making runtime dispatch decisions\.

This predictability is also actionable at runtime\. Figure[4\(b\)](https://arxiv.org/html/2608.14624#S2.F4.sf2)replays each workload’s execution trace through an online first\-order transition counter, the same statisticCacheScoutmaintains: predicting the next agent as the most frequent observed successor of the current agent reaches 76–86% top\-1 accuracy within 50 observed dispatches, without any offline training\.

\\noindentparagraph

Takeaway\. Future agent execution is not random\. Although multi\-agent workflows are dynamically determined at runtime, they retain strong local transition structure that a lightweight online model can capture within tens of dispatches, making future agent execution predictable enough to drive proactive cache management\.

#### 2\.2\.3\.Can execution structure be specified offline?

The previous insight shows that future agent execution is predictable\. We finally ask whether this execution structure can be determined before deployment\.

To answer this question, we compare the transition matrices of four representative coordination topologies built on the same six\-agent framework\. Figure[5](https://arxiv.org/html/2608.14624#S2.F5)shows that execution structure varies dramatically across coordination topologies despite using the same set of agent roles\.

Pipeline and Debate are both highly predictable \(R=1\.0R=1\.0andR=0\.78R=0\.78\), yet their transition matrices are fundamentally different\. Pipeline follows a fixed chain through all six agents, whereas Debate concentrates execution on a proposer–challenger exchange between Coder and Reviewer that is periodically adjudicated by a judge\. SelectorGroupChat represents the production setting, where the next agent is dynamically selected by an LLM\. Although its execution is no longer deterministic \(R=0\.57R=0\.57\), transition probability remains concentrated on a few high\-probability edges\. In contrast, the Random topology distributes probability almost uniformly across all agent pairs \(R=0\.12R=0\.12\), leaving little exploitable execution structure\.

\\noindentparagraph

Takeaway\. Execution structure is not fixed by the agent framework itself\. Instead, it is determined by runtime coordination behavior\. Consequently, transition models cannot be specified offline and must instead be learned online from runtime execution history\.

Table 1\.Comparison of representative approaches to KV\-cache management for multi\-agent serving\. ✓ supported; ✗ not supported\.Approach \(Representative Systems\)Cache AbstractionReuse PolicyAssumptionAgent\-awareLearnableRuntimeLimitations for Multi\-Agent ServingReactive serving engines\(vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib2)\), SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib3)\), TensorRT\-LLM\(NVIDIA,[2023](https://arxiv.org/html/2608.14624#bib.bib35)\)\)Content\-addressed KV blocksPrefix matching \+ recency evictionRecent use predicts reuse✗✗Blind to agent identity; recency\-based eviction discards anchors with long reuse distances despite their near\-certain reuse\.Static agent\-aware KV management\(KVFlow\(Panet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib4)\), Tokencake\(Bianet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib12)\), Continuum\(Liet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib5)\), KVCOMM\(Yeet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib13)\)\)KV\-cache \+ static workflow stateScheduled prefetch, pinning, or KV adaptationWorkflow structure known in advance✓✗Static assumptions break under LLM\-selected routing; Continuum protects only the current session, and KVCOMM’s approximation degrades output quality\.Learning\-based cache replacement\(LRB\(Songet al\.,[2020](https://arxiv.org/html/2608.14624#bib.bib42)\), LeCaR\(Vietriet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib43)\), GL\-Cache\(Yanget al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib44)\), LHD\(Beckmannet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib45)\)\)Opaque cache objectsLearned reuse predictionAccess history predicts reuse✗✓Feature extraction and model inference exceed the serving hot\-path budget; no execution semantics\.Agent workflow frameworks\(AutoGen\(Wuet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib23)\), LangGraph\(LangChain,[2024](https://arxiv.org/html/2608.14624#bib.bib33)\), CrewAI\(CrewAI,[2024](https://arxiv.org/html/2608.14624#bib.bib34)\)\)N/AN/A; delegated to the engineExecution semantics are visible✓✗Sees agent execution but has no visibility into or control over the engine’s KV\-cache\.CacheScout\(ours\)KV\-cache \+ online transition matrixTransition model guided eviction and prefetchExecution structure is learnable✓✓Gains diminish when execution approaches random routing\.

### 2\.3\.Challenges

Building an agent\-aware KV\-cache runtime introduces fundamental challenges\. Table[1](https://arxiv.org/html/2608.14624#S2.T1)summarizes the assumptions underlying existing approaches related to KV\-cache management and their limitations for multi\-agent serving\.

\\noindentparagraph

Challenge \#1: Semantic gap between agents and KV blocks\.Existing prefix caches operate on prompt content, not agent semantics\. vLLM identifies reusable state through block hashes, while SGLang uses radix\-tree prefix matching\. These mechanisms can determine whether two prompt prefixes are identical, but they cannot tell which agent produced a KV block, which session it belongs to, or whether the same agent is likely to be invoked again\. Prior cache replacement policies, from heuristic approaches such as LRU\(Mattsonet al\.,[1970](https://arxiv.org/html/2608.14624#bib.bib46)\), LFU\(Ahoet al\.,[1971](https://arxiv.org/html/2608.14624#bib.bib47)\), ARC\(Megiddo and Modha,[2003](https://arxiv.org/html/2608.14624#bib.bib27)\), and LIRS\(Jiang and Zhang,[2002](https://arxiv.org/html/2608.14624#bib.bib48)\)to learning\-based methods such as LRB\(Songet al\.,[2020](https://arxiv.org/html/2608.14624#bib.bib42)\), LeCaR\(Vietriet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib43)\), GL\-Cache\(Yanget al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib44)\), and LHD\(Beckmannet al\.,[2018](https://arxiv.org/html/2608.14624#bib.bib45)\), still make decisions over cache objects\. They do not expose the agent\-level identity needed to protect reusable agent anchors\.

\\noindentparagraph

Challenge \#2: Dynamic execution without a static graph\.Existing workflow\-level optimizations assume that future execution can be described by a declared graph, a fixed pipeline, or developer\-provided hints\. This assumption breaks for modern agent frameworks such as AutoGen SelectorGroupChat and OpenAI Swarm, where the next agent is often selected by an LLM at runtime\. As we show in Sec\.[2\.2](https://arxiv.org/html/2608.14624#S2.SS2)\(Figure[5](https://arxiv.org/html/2608.14624#S2.F5)\), even the same six\-agent framework can produce very different transition matrices under different coordination topologies\. A cache policy cannot rely on a static workflow graph; it must learn the execution structure from live dispatches\.

\\noindentparagraph

Challenge \#3: Prediction under serving\-system constraints\.Even when future agent execution is predictable, prediction must operate within the stringent latency budget of a production serving engine\. Cache eviction is triggered whenever GPU memory becomes constrained and therefore lies directly on the serving critical path\. Existing learning\-based cache management techniques often improve prediction quality through complex feature extraction, model inference, or periodic retraining\. While effective for traditional storage caches, these techniques introduce overheads that are difficult to justify in LLM serving, where even microseconds of additional latency accumulate across millions of cache operations\. The challenge is therefore not simply to predict future reuse accurately, but to do so using lightweight state, microsecond\-scale decision latency, and graceful degradation when execution exhibits little predictable structure\.

## 3\.Design

![Refer to caption](https://arxiv.org/html/2608.14624v1/x9.png)Figure 6\.CacheScoutarchitecture\.CacheScoutsits between the agent framework and the serving engine as a lightweight runtime layer\.①Each agent dispatch is observed by the Transition Learner, which updates the online transition matrix\. The learned matrix feeds②the Survival Scorer and③the Background Prefetch Coordinator\.④The scorer converts hop distances on the execution graph into survival scores that guide eviction in the GPU KV\-cache pool, and⑤the coordinator issues a background warmup request for the predicted agent’s anchor, reloading it from the external KV pools when present\.### 3\.1\.Runtime Overview

CacheScoutis not another cache replacement policy\. Instead, it rethinks the role of the KV\-cache runtime in agentic serving\. Existing serving engines efficiently manage KV blocks, but all cache decisions are inherently reactive\. They rely solely on past accesses\. Our goal is to answer a different question:Can the serving system prepare for future execution instead of simply reacting to previous execution?

Our key idea is to introduce an independentKV\-cache runtime layerbetween the agent framework and the serving engine\. Rather than embedding agent semantics into the serving engine or pushing cache management into the agent framework,CacheScoutseparates cache intelligence into its own runtime\. This runtime continuously \(1\) learns how agents interact, \(2\) predicts which agents are likely to execute next, and \(3\) translates these predictions into cache management decisions\.

Figure[6](https://arxiv.org/html/2608.14624#S3.F6)illustrates the overall architecture ofCacheScout\. The runtime resides between the agent framework and the serving engine\. The agent framework remains responsible for agent scheduling and workflow execution, while the serving engine continues to perform prefix matching, KV allocation, and GPU memory management\.CacheScoutintroduces no changes to either component\. Instead, it exchanges lightweight runtime metadata with both sides, allowing execution information to influence cache management without modifying the underlying serving stack\.

CacheScoutrealizes this runtime through three cooperating components\. The Transition Learner \(Sec\.[3\.2](https://arxiv.org/html/2608.14624#S3.SS2)\) is responsible for understanding how a workload executes\. As agents interact, it continuously observes runtime dispatches and learns a workload\-specific transition model\. Instead of assuming a predefined workflow graph, it models execution as an online first\-order Markov chain, allowing the runtime to adapt naturally to dynamic routing\. The Survival Scorer \(Sec\.[3\.3](https://arxiv.org/html/2608.14624#S3.SS3)\) answers the next question: which cached state is worth keeping? Rather than asking which block was used most recently, it asks which block is most likely to be reused next\. To do so, it estimates the survival probability of each cached agent anchor from the learned transition model and uses this probability to guide cache replacement\. The Background Prefetch Coordinator \(Sec\.[3\.4](https://arxiv.org/html/2608.14624#S3.SS4)\) turns predictions into actions\. Once the runtime identifies which agent is likely to execute next, it proactively reconstructs the corresponding high\-value KV blocks during idle periods between requests\. By moving this work off the serving critical path,CacheScoutimproves cache readiness without increasing inference latency\.

### 3\.2\.Transition Learner

The first task ofCacheScoutis to infer how a workload executes\. This is challenging because modern agent frameworks do not expose a fixed execution graph\. The next agent is selected dynamically by LLM reasoning, and the resulting execution structure varies across workloads and evolves over time\.

A natural question is how much execution structure needs to be learned\. One option is to model complete execution traces using a sophisticated sequence model\. However, cache management does not need to predict an entire workflow\. It only needs to anticipate the next scheduling decision before cache replacement occurs\. Since Sec\.[2](https://arxiv.org/html/2608.14624#S2)shows that local agent transitions already carry substantial predictive signal,CacheScoutlearns only the local transition structure needed for cache management\. To realize this idea, the Transition Learner models agent execution as a first\-order Markov chain\. LetAtA\_\{t\}denote the agent executed at steptt\. For every observed transitionAt=i→At\+1=jA\_\{t\}=i\\rightarrow A\_\{t\+1\}=j, the runtime updates a transition counterCi​j←Ci​j\+1C\_\{ij\}\\leftarrow C\_\{ij\}\+1\. The transition probability from agentiito agentjjis then estimated as

\(3\)Pi​j=P​\(At\+1=j∣At=i\)=Ci​j\+ϵ∑k\(Ci​k\+ϵ\),P\_\{ij\}=P\(A\_\{t\+1\}=j\\mid A\_\{t\}=i\)=\\frac\{C\_\{ij\}\+\\epsilon\}\{\\sum\_\{k\}\\left\(C\_\{ik\}\+\\epsilon\\right\)\},whereϵ\\epsilonis a small smoothing constant used to avoid zero\-probability transitions early in execution\. Given the current agentAt=iA\_\{t\}=i, the predicted distribution over the next agent is simply theii\-th row of the transition matrix:

\(4\)𝐩t\+1=Pi,:\.\\mathbf\{p\}\_\{t\+1\}=P\_\{i,:\}\.The most likely next agent is

\(5\)A^t\+1=arg⁡maxj⁡Pi​j\.\\hat\{A\}\_\{t\+1\}=\\arg\\max\_\{j\}P\_\{ij\}\.Because each dispatch updates only one counter and one row of the transition matrix, the learner has constant update cost and bounded state\. It requires no offline training, no workflow annotations, and no replay of historical traces\. The result is a lightweight execution model that evolves with the workload and provides the predictive signal needed by the rest ofCacheScout\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x10.png)Figure 7\.Survival score computation\.CacheScoutconverts the learned transition graph into a sparse execution graph by pruning low\-confidence transitions, then computes survival scores from BFS hop distances\.
### 3\.3\.Survival Scorer

The second task ofCacheScoutis not simply to predict which agent will execute next, but to determine which cached agent anchors are worth preserving\. A highly probable agent does not necessarily imply that all of its cached blocks should remain in GPU memory, while an infrequently executed agent may still own reusable anchors that are expensive to recompute\. Consequently, transition probability cannot be used directly as an eviction policy\.

Ideally, we would rank every cached agent anchor according to its probability of being reused within the nextKKscheduling steps\. We define this quantity as the survival probability\.

\(6\)psurv​\(a;a0,K\)=Pr⁡\(∃k∈\[1,K\]:At\+k=a\|At=a0\),p\_\{\\mathrm\{surv\}\}\(a;a\_\{0\},K\)=\\Pr\\left\(\\exists\\,k\\in\[1,K\]:A\_\{t\+k\}=a\\;\\middle\|\\;A\_\{t\}=a\_\{0\}\\right\),wherea0a\_\{0\}is the current agent andaadenotes a candidate agent anchor\. Intuitively, the survival probability measures whether an anchor is likely to remain useful in the near future, making it a natural objective for cache replacement\.

The exact survival probability is prohibitively expensive to compute, as it requires aggregating transition probabilities over all execution paths within the prediction horizon\. Instead,CacheScoutapproximates survival using the learned execution graph\. It first converts the transition matrix into a sparse directed graph by retaining only high\-confidence transitions,

\(7\)\(a,b\)∈E⇔P​\(b∣a\)≥τ,\(a,b\)\\in E\\Leftrightarrow P\(b\\mid a\)\\geq\\tau,whereτ\\tauis a confidence threshold\.

Figure[7](https://arxiv.org/html/2608.14624#S3.F7)illustrates the approximation process\. Starting from the current agent, the runtime performs a single breadth\-first search \(BFS\) over the sparse execution graph to compute the minimum hop distanceE​\[a\]E\[a\]to every reachable agent\. Hop distance serves as an efficient proxy for future reuse: anchors expected to be revisited sooner receive higher survival scores than those farther away in the execution graph\. The survival score is then computed as

\(8\)p~surv​\(a\)=1−min⁡\(E​\[a\],Emax\)Emax,\\tilde\{p\}\_\{\\mathrm\{surv\}\}\(a\)=1\-\\frac\{\\min\\left\(E\[a\],E\_\{\\max\}\\right\)\}\{E\_\{\\max\}\},whereEmaxE\_\{\\max\}bounds the prediction horizon\.

Each cached KV block inherits the survival score of its corresponding agent anchor\. To improve robustness,CacheScoutblends the predicted survival score multiplicatively with a recency signal and weights the result by the block’s reconstruction cost,

\(9\)Score​\(b\)=\(p~surv​\(ab\)\+δ\)⋅\(e−λ⋅age​\(b\)\+δ\)⋅\|b\|,\\mathrm\{Score\}\(b\)=\\left\(\\tilde\{p\}\_\{\\mathrm\{surv\}\}\(a\_\{b\}\)\+\\delta\\right\)\\cdot\\left\(e^\{\-\\lambda\\cdot\\mathrm\{age\}\(b\)\}\+\\delta\\right\)\\cdot\|b\|,whereaba\_\{b\}is the anchor associated with blockbb,age​\(b\)\\mathrm\{age\}\(b\)counts scheduler steps since the block was last accessed,λ\\lambdais a recency decay rate, and\|b\|\|b\|is the number of tokens the block caches\. Blocks with lower scores are evicted first\. The product can be read as the expected prefill work lost by evictingbb: the first two factors approximate the probability that the block is reused soon, and\|b\|\|b\|is the recompute cost paid if that reuse misses\. Between blocks with equal reuse probability,CacheScouttherefore evicts the one that is cheaper to reconstruct\. The multiplicative blend lets either signal compensate when the other is weak: under predictable execution, the survival factor protects high\-value anchors; when execution becomes less predictable, survival scores flatten and the recency factor dominates, so the policy gracefully degrades toward LRU\. The small floorδ\\deltakeeps both factors positive so that newly observed blocks, for which the learner has no signal yet, are not immediately evicted at cold start\.

Algorithm 1CacheScoutruntime workflow\. Numbered steps correspond to the arrows in Figure[6](https://arxiv.org/html/2608.14624#S3.F6)\.1:State:transition counts

n​\(⋅,⋅\)n\(\\cdot,\\cdot\); transition matrix

P^\\widehat\{P\}; current agent

at←⊥a\_\{t\}\\leftarrow\\bot; execution graph

𝒢\\mathcal\{G\}; hop\-count table

E​\[⋅\]E\[\\cdot\]; block\-to\-agent map

agentOf​\[⋅\]\\textsc\{agentOf\}\[\\cdot\]; last\-access table

lastAccess​\[⋅\]\\textsc\{lastAccess\}\[\\cdot\]; decay rate

λ\\lambda, floor

δ\\delta, threshold

RminR\_\{\\min\}\.

2:

3:loop

4:

r←r\\leftarrowframework\.NextRequest\(\)⊳\\trianglerightagent dispatch

5:engine\.Serve\(

rr\)⊳\\trianglerightnormal serving path

6:BetweenStep⊳\\triangleright③off critical path

7:endloop

8:

9:DuringServe, the block pool raises two events:

10:onengine\.pool\.BlockTouch\(b,sb\)\(b,s\_\{b\}\):

11:ObserveTouch\(

b,sbb,s\_\{b\}\)⊳\\triangleright①learn transition

12:onengine\.pool\.Evict:

13:return

arg⁡minb∈ℬevict⁡ScoreBlock​\(b\)\\displaystyle\\arg\\min\_\{b\\in\\mathcal\{B\}\_\{\\mathrm\{evict\}\}\}\\textsc\{ScoreBlock\}\(b\)⊳\\triangleright④survival\-guided eviction

14:

15:procedureObserveTouch\(

b,sbb,s\_\{b\}\)

16:

a←AgentId​\(sb\)a\\leftarrow\\textsc\{AgentId\}\(s\_\{b\}\)
17:

agentOf​\[b\]←a\\textsc\{agentOf\}\[b\]\\leftarrow a;

lastAccess​\[b\]←now​\(\)\\textsc\{lastAccess\}\[b\]\\leftarrow\\textsc\{now\}\(\)
18:if

a≠ata\\neq a\_\{t\}then

19:if

at≠⊥a\_\{t\}\\neq\\botthen

20:

n​\(at,a\)←n​\(at,a\)\+1n\(a\_\{t\},a\)\\leftarrow n\(a\_\{t\},a\)\+1
21:update

P^\\widehat\{P\}from

n​\(⋅,⋅\)n\(\\cdot,\\cdot\)
22:endif

23:

𝒢←ThresholdGraph​\(P^,τ\)\\mathcal\{G\}\\leftarrow\\textsc\{ThresholdGraph\}\(\\widehat\{P\},\\tau\)⊳\\triangleright②refresh scorer state

24:

E​\[⋅\]←Bfs​\(𝒢,a\)E\[\\cdot\]\\leftarrow\\textsc\{Bfs\}\(\\mathcal\{G\},a\)
25:

at←aa\_\{t\}\\leftarrow a
26:endif

27:endprocedure

28:

29:functionScoreBlock\(

bb\)

30:

a←agentOf​\[b\]a\\leftarrow\\textsc\{agentOf\}\[b\]
31:

age←now​\(\)−lastAccess​\[b\]\\mathrm\{age\}\\leftarrow\\textsc\{now\}\(\)\-\\textsc\{lastAccess\}\[b\]
32:return

\(p~surv​\(a\)\+δ\)⋅\(e−λ⋅age\+δ\)⋅\|b\|\(\\tilde\{p\}\_\{\\mathrm\{surv\}\}\(a\)\+\\delta\)\\cdot\(e^\{\-\\lambda\\cdot\\mathrm\{age\}\}\+\\delta\)\\cdot\|b\|
33:endfunction

34:

35:procedureBetweenStep

36:if

R​\(P^\)≥RminR\(\\widehat\{P\}\)\\geq R\_\{\\min\}then

37:

a∗←arg⁡maxa⁡P^​\(a∣at\)a^\{\*\}\\leftarrow\\arg\\max\_\{a\}\\widehat\{P\}\(a\\mid a\_\{t\}\)
38:IssueWarmupRequest\(

a∗a^\{\*\}\)⊳\\triangleright⑤background warmup

39:endif

40:endprocedure

### 3\.4\.Background Prefetch Coordinator

The final task ofCacheScoutis to proactively prepare reusable KV\-cache before it is requested\. While the Transition Learner discovers execution structure and the Survival Scorer determines which cached agent anchors should be retained, neither helps when a future anchor has already been evicted or has not yet been constructed\. The Background Prefetch Coordinator closes this loop by proactively warming up high\-value agent anchors during idle periods\. Algorithm[1](https://arxiv.org/html/2608.14624#alg1)summarizes the overall runtime workflow\. A straightforward solution would be to directly insert predicted KV blocks into the cache\. However, such a design requires intrusive modifications to the serving engine’s memory manager and cache implementation, reducing portability across different inference engines\.CacheScoutinstead adopts a lightweight approach that reuses the engine’s existing prefill pipeline\. Rather than manipulating KV blocks explicitly, the coordinator issues a lightweight warmup request containing only the predicted agent’s reusable anchor, including its system prompt, tool definitions, and a minimal user prompt\. Since this warmup follows the engine’s normal execution path, the resulting KV blocks are constructed exactly as they would be during a real agent invocation, requiring no modifications to the attention kernels, memory allocator, or cache implementation\. Given the current agentata\_\{t\}, the coordinator first selects the most likely next agent,

\(10\)a∗=arg⁡maxa⁡P​\(a∣at\),a^\{\*\}=\\arg\\max\_\{a\}P\(a\\mid a\_\{t\}\),and triggers warmup only for the corresponding reusable anchor\. Because warmup executes during idle periods between requests, all KV construction occurs off the serving critical path, allowing otherwise idle GPU cycles to be converted into future cache hits\. Proactive prefetching is beneficial only when execution is sufficiently predictable\.CacheScouttherefore continuously evaluates the execution predictability metricRR\(Eq\.[2](https://arxiv.org/html/2608.14624#S2.E2)\)\. When

\(11\)R≥Rmin,R\\geq R\_\{\\min\},the coordinator enables background warmup; otherwise, it disables prefetching and falls back to survival\-guided cache replacement alone\. This adaptive gating prevents unnecessary GPU work under highly dynamic execution while preserving the benefits of proactive cache management whenever meaningful execution structure exists\.

## 4\.Implementation

We implementCacheScoutas a lightweight runtime layer on top of vLLM v0\.11 \(V1 engine\), consisting of approximately 2,300 lines of runtime logic and an 800\-line patch\. The runtime exposes four primitives, ObserveTouch\(\), Predict\(\), ScoreBlock\(\), and Warmup\(\), that connect execution\-aware cache management to the serving engine\.

The patch modifies five vLLM source files \(the block pool, the engine core, the scheduler, and the scheduler\- and worker\-side halves of the CPU\-offload connector\) and introduces three runtime hooks\. First, the block pool invokes ObserveTouch\(\) at prefix\-matching time: each request is fingerprinted from its own prefix block hashes, so a fingerprint change signals an agent dispatch and updates transition statistics and the block\-to\-agent mapping without any framework cooperation \(step①in Algorithm[1](https://arxiv.org/html/2608.14624#alg1)\)\. Second, the eviction path invokes ScoreBlock\(\) \(Eq\.[9](https://arxiv.org/html/2608.14624#S3.E9)\) to replace the default LRU ordering with survival\-guided ranking, additionally preferring blocks whose contents are already resident in the CPU tier; survival probabilities are computed once per scheduler step and cached\. Third, Warmup\(\) is executed between agent turns through the standard serving API: after applying theR≥RminR\\geq R\_\{\\min\}gate and a rate limit, it issues an inference request containing only the predicted agent’s anchor withmax\_tokens=1, which the engine recognizes and excludes from transition learning\.

CacheScoutmaintains four runtime data structures: an online transition matrix, a sparse execution graph with shortest\-path distances, a hop\-distance table, and a block\-to\-agent mapping, totaling under 25 KB \(Sec\.[5](https://arxiv.org/html/2608.14624#S5)\)\. Transition counts are updated incrementally on each agent dispatch; the execution graph and hop\-distance table are refreshed only when the current agent changes\. The eviction hot path performs only a single cached score evaluation \(Eq\.[9](https://arxiv.org/html/2608.14624#S3.E9)\); graph construction and background warmup execute entirely off the critical path\.

The implementation reuses vLLM’s existing execution pipeline without modifying attention kernels, memory allocators, schedulers, or block hashing\. Background warmup is issued as a standard inference request, producing the same KV blocks as a normal invocation\.CacheScoutcan be enabled or disabled through a single environment variable, and reuses vLLM’s existing LMCache and CPU offloading infrastructure without modification\.

## 5\.Evaluation

This section evaluates the effectiveness and efficiency ofCacheScouton representative multi\-agent workloads\. Specifically, we seek to answer five questions: \(i\) DoesCacheScoutimprove KV\-cache reuse and end\-to\-end serving performance compared with existing approaches? \(ii\) How much do survival\-guided eviction and proactive prefetch contribute to the overall performance gains? \(iii\) DoesCacheScoutgeneralize to larger models? \(iv\) DoesCacheScoutremain effective under different cache capacities and serving conditions? \(v\) What runtime overhead doesCacheScoutintroduce?

![Refer to caption](https://arxiv.org/html/2608.14624v1/x11.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x12.png)\(a\)KV\-cache hit rate\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x13.png)\(b\)Mean TTFT\.

Figure 8\.Higher KV\-cache hit rates translate into lower inference latency across representative multi\-agent workloads\.![Refer to caption](https://arxiv.org/html/2608.14624v1/x14.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x15.png)\(a\)GSM8K
![Refer to caption](https://arxiv.org/html/2608.14624v1/x16.png)\(b\)MT\-Bench
![Refer to caption](https://arxiv.org/html/2608.14624v1/x17.png)\(c\)GAIA
![Refer to caption](https://arxiv.org/html/2608.14624v1/x18.png)\(d\)SWE\-bench

Figure 9\.TTFT cumulative distributions across representative multi\-agent workloads\.CacheScoutconsistently shifts the distribution toward lower latency\.### 5\.1\.Experimental Setup

\\noindentparagraph

Testbed\.We run experiments on a server with eight NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of memory\. We use Llama\-3\.1\-8B\-Instruct as the main model and run all systems under the same vLLM configuration, GPU memory budget, decoding parameters, and prefix cache settings\. For the model\-scale study \(Sec\.[5\.4](https://arxiv.org/html/2608.14624#S5.SS4)\) we use Qwen3\-235B\-A22B\-FP8, a 235B\-parameter mixture\-of\-experts model, with tensor parallelism across four NVIDIA H200 GPUs each with 141 GB memory and connected via NVLink\.

\\noindentparagraph

Workloads\.We evaluateCacheScouton four representative multi\-agent workloads: GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.14624#bib.bib26)\), MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib24)\), GAIA\(Mialonet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib25)\), and SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib36)\)\. All four run on the same six\-agent supervisor framework with tools and AutoGen’s SelectorGroupChat for dynamic LLM\-selected routing\. Together, these workloads cover a broad spectrum of execution characteristics, from short mathematical exchanges \(GSM8K\) and multi\-turn conversation \(MT\-Bench\) to tool\-heavy web reasoning \(GAIA\) and long, prefill\-heavy software\-engineering sessions \(SWE\-bench\)\.

\\noindentparagraph

Baselines\.We compareCacheScoutagainst two representative KV\-cache management systems\. vLLM serves as the production baseline using reactive prefix caching with LRU eviction over block\-aligned KV\-caches\. Continuum represents cross\-session KV reuse through TTL\-based pinning around tool\-call boundaries\. Following the observed inter\-turn latency in AutoGen, we set the TTL to 0\.3 s\. The model\-scale study \(Sec\.[5\.4](https://arxiv.org/html/2608.14624#S5.SS4)\) compares against vanilla vLLM only, as Continuum’s vLLM fork does not support this model\.

\\noindentparagraph

Metrics\.Our evaluation focuses on the following metrics:

- •KV\-cache hit rate:The fraction of prompt tokens served from the KV\-cache, computed as total\_cached\_tokens / total\_prompt\_tokens\.
- •Time\-to\-first\-token \(TTFT\):The latency from request arrival to the first generated token, measuring user\-perceived responsiveness\.
- •Per\-turn latency:The end\-to\-end latency of an individual agent invocation, including both prefill and decoding\.
- •Throughput:The number of completed agent turns per second under a fixed workload and hardware configuration\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x19.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x20.png)\(a\)Mean per\-turn latency\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x21.png)\(b\)Peak throughput\.

Figure 10\.CacheScoutconsistently achieves lower mean per\-turn latency and higher peak throughput across representative multi\-agent workloads\.
### 5\.2\.Main Results:CacheScoutimproves KV\-cache reuse and serving performance

CacheScoutaims to improve both KV\-cache reuse and end\-to\-end serving performance for dynamic multi\-agent workloads\. We first examine cache effectiveness through KV\-cache hit rate, and then evaluate its impact on serving performance\.

\\noindentparagraph

KV\-cache hit rate and Time\-to\-first\-token\.Figure[8\(a\)](https://arxiv.org/html/2608.14624#S5.F8.sf1)shows thatCacheScoutconsistently achieves the highest KV\-cache hit rate across all four workloads, improving over vanilla vLLM by 10–18 percentage points despite dynamic LLM\-selected routing\. The higher hit rate translates directly into faster responses: Figure[8\(b\)](https://arxiv.org/html/2608.14624#S5.F8.sf2)shows thatCacheScoutreduces mean TTFT by 18–45% across all four workloads\. Figure[9](https://arxiv.org/html/2608.14624#S5.F9)further shows that the improvement holds across the latency distribution: median TTFT drops by 14–52% \(e\.g\., 231→\{\\to\}114 ms on GAIA and 239→\{\\to\}115 ms on GSM8K\), and P99 TTFT drops by 21–52% on three of the four workloads \(711→\{\\to\}342 ms on SWE\-bench\), soCacheScoutis below vanilla vLLM at the median on every workload\. This indicates that the benefit is not limited to a small fraction of requests but consistently improves user\-perceived responsiveness\.

\\noindentparagraph

End\-to\-end latency and throughput\.The higher KV\-cache hit rate translates directly into better serving performance\. As shown in Figure[10](https://arxiv.org/html/2608.14624#S5.F10),CacheScoutreduces mean per\-turn latency by 29–38% while delivering 19–57% higher peak throughput across all four workloads\. Figure[11](https://arxiv.org/html/2608.14624#S5.F11)shows how this advantage compounds with load: vanilla vLLM’s throughput plateaus once its KV\-cache saturates, whileCacheScoutcontinues to scale, and within the same mean\-latency budgetCacheScoutsustains1\.71\.7–12×12\\timesthe arrival rate of vanilla vLLM and4\.24\.2–16×16\\timesthat of Continuum\. Together, these results demonstrate that improving KV\-cache reuse benefits not only request latency but also the overall efficiency of multi\-agent serving\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x22.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x23.png)

Figure 11\.Latency and throughput versus session arrival rate\.CacheScoutdelays cache saturation, sustaining lower latency and higher throughput under increasing load\.
### 5\.3\.Design Breakdown: Predictive eviction provides the primary performance gains

![Refer to caption](https://arxiv.org/html/2608.14624v1/x24.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x25.png)\(a\)KV\-cache hit rate\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x26.png)\(b\)Mean TTFT\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x27.png)\(c\)Mean per\-turn latency\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x28.png)\(d\)Peak throughput\.

Figure 12\.Mechanism ablation\. Survival\-guided eviction accounts for most of the cache hit rate improvement, while background prefetch further reduces latency and improves throughput\.CacheScoutcombines two predictive runtime mechanisms: predictive eviction and background prefetch\. To understand their individual contributions, we evaluate three configurations: vanilla vLLM, eviction only, and the completeCacheScoutruntime\. Figure[12](https://arxiv.org/html/2608.14624#S5.F12)summarizes their impact on KV\-cache hit rate, mean TTFT, per\-turn latency, and throughput\.

\\noindentparagraph

Predictive eviction\.Figure[12](https://arxiv.org/html/2608.14624#S5.F12)shows that predictive eviction provides the dominant source of improvement on the four real\-world workloads\. Enabling predictive eviction alone increases KV\-cache hit rate by 18–22 percentage points over vanilla vLLM, whereas background prefetch alone adds at most one percentage point\. The higher cache hit rate directly translates into lower per\-turn latency and higher throughput \(Figures[12\(c\)](https://arxiv.org/html/2608.14624#S5.F12.sf3)and[12\(d\)](https://arxiv.org/html/2608.14624#S5.F12.sf4)\), demonstrating that execution\-aware cache replacement is the primary contributor toCacheScout’s performance under realistic multi\-agent workloads\.

\\noindentparagraph

Background prefetch\.The effectiveness of background prefetch depends on what standalone warmup can preserve\. On its own, prefetch is inert on these workloads: the median turn carries no warmable agent prefix, and the blocks a warmup recomputes are evicted by LRU before the predicted agent arrives\. Combined with predictive eviction, however, warmup re\-activates agent anchors that eviction then protects: the complete runtime reduces per\-turn latency by a further 28% over eviction alone on GAIA \(251 vs\. 347 ms\) and achieves the best overall serving performance on both serving metrics \(Figures[12\(c\)](https://arxiv.org/html/2608.14624#S5.F12.sf3)and[12\(d\)](https://arxiv.org/html/2608.14624#S5.F12.sf4)\)\.

These results validate the design ofCacheScout\. Predictive eviction serves as the primary optimization for dynamic multi\-agent workloads, while background prefetch provides additional gains when execution exhibits strong temporal regularity\. This complementary behavior motivatesCacheScout’s adaptive prefetch policy\.

### 5\.4\.Scalability:CacheScoutremains effective under varying deployment settings and model scale

We evaluate the robustness ofCacheScoutunder varying model scale, GPU cache capacities, and offered load\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x29.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x30.png)\(a\)KV\-cache hit rate\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x31.png)\(b\)Mean TTFT\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x32.png)\(c\)Mean per\-turn latency\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x33.png)\(d\)Throughput\.

Figure 13\.CacheScouton Qwen3\-235B\-A22B\-FP8 \(tensor parallelism over four H200 GPUs\)\. KV\-cache hit rates are reported across all workloads, while latency and throughput are evaluated on SWE\-bench\.CacheScoutconsistently improves cache reuse, reduces inference latency, and sustains higher throughput under increasing load\.![Refer to caption](https://arxiv.org/html/2608.14624v1/x34.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x35.png)\(a\)KV\-cache hit rate versus GPU cache budget\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x36.png)\(b\)Per\-turn latency with adaptive prefetch gating\.

Figure 14\.Sensitivity to GPU cache budget\.CacheScoutmaintains high KV\-cache hit rates under tight GPU memory budgets, while adaptive prefetch reduces latency without introducing unnecessary warmup overhead\.\\noindentparagraph

Model scale\.To validate thatCacheScout’s mechanisms are not specific to small models, we repeat the serving comparison with Qwen3\-235B\-A22B\-FP8, a 235B\-parameter mixture\-of\-experts model served with tensor parallelism across four H200 GPUs\. As shown in Figure[13\(a\)](https://arxiv.org/html/2608.14624#S5.F13.sf1),CacheScout’s hit rate advantage transfers unchanged: it improves KV\-cache hit rate by 7–13 percentage points across all four workloads\. On SWE\-bench, whose long prompts keep prefill on the costly path, this hit rate advantage compounds across the entire load range: sweeping the arrival rate from 0\.2 to 2\.0 sessions/s \(Figure[13](https://arxiv.org/html/2608.14624#S5.F13)b–d\),CacheScoutreduces mean TTFT by 33–54% and mean per\-turn latency by 26–36% at every rate, and where vanilla vLLM’s throughput saturates at 5\.2 turns/s,CacheScoutcontinues scaling to 7\.2 turns/s \(\+\+37%\)\. These results indicate thatCacheScoutgeneralizes across model scale\.

\\noindentparagraph

GPU cache capacity and load\.As shown in Figure[14\(a\)](https://arxiv.org/html/2608.14624#S5.F14.sf1),CacheScoutmaintains a nearly constant KV\-cache hit rate of 86–87% as the GPU cache budget varies from 100 to 200 blocks \(budgets below the workload’s∼\{\\sim\}93\-block maximum request footprint cannot admit every request and are excluded\)\. In contrast, vanilla vLLM improves gradually from 64\.4% to 76\.6%, requiring substantially more GPU memory to approach the hit rate achieved byCacheScout\.CacheScoutis equally robust to load: across the full arrival\-rate sweep of Figure[11](https://arxiv.org/html/2608.14624#S5.F11)\(0\.2–50 sessions/s\), its KV\-cache hit rate holds at 86–87% while vanilla vLLM stays near 65–68%\. These results demonstrate thatCacheScoutremains effective under both memory pressure and increasing load\.

\\noindentparagraph

Adaptive prefetch gate\.Figure[14\(b\)](https://arxiv.org/html/2608.14624#S5.F14.sf2)evaluates the adaptive prefetch gate introduced in Sec\.[3\.4](https://arxiv.org/html/2608.14624#S3.SS4)\. With the gate enabled,CacheScoutmaintains stable per\-turn latency once sufficient cache capacity is available and degrades gracefully as GPU memory becomes more constrained\. In contrast, disabling the gate increases per\-turn latency by up to 22% at small cache sizes because unnecessary background warmup competes with foreground inference for GPU resources\.

### 5\.5\.Runtime Overhead:CacheScoutincurs negligible runtime overhead

We evaluate the runtime overhead ofCacheScoutthrough in\-process microbenchmarks measuring coordinator memory usage and hot\-path latency\.

![Refer to caption](https://arxiv.org/html/2608.14624v1/x37.png)

![Refer to caption](https://arxiv.org/html/2608.14624v1/x38.png)\(a\)Coordinator state size\.
![Refer to caption](https://arxiv.org/html/2608.14624v1/x39.png)\(b\)Runtime hot\-path latency\.

Figure 15\.Runtime overhead\.CacheScoutmaintains a compact runtime state and introduces negligible hot\-path overhead, even as the number of agents increases\.\\noindentparagraph

Memory footprint\.As shown in Figure[15\(a\)](https://arxiv.org/html/2608.14624#S5.F15.sf1), the coordinator state converges quickly as execution history accumulates and remains below 25 KB even with 24 agents, indicating negligible memory overhead\.

\\noindentparagraph

Runtime latency\.Figure[15\(b\)](https://arxiv.org/html/2608.14624#S5.F15.sf2)shows that both runtime operations incur only microsecond\-level overhead\. ObserveTouch requires approximately\\qty1 per block touch, while PredictSurvival completes within\\qty6 even for 24 agents\. These costs are negligible compared with LLM serving latency\.

## 6\.Related Work

\\noindentparagraph

DAG\-aware KV\-cache management\.Existing LLM serving engines such as vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14624#bib.bib2)\)and SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib3)\)provide efficient prefix caching, but their cache management remains largely reactive\. In agentic workflows, this triggers premature eviction of the agent anchor between steps and forces frequent prefill recompute on bytes the engine just processed\. To solve this, a line of work models the agent calling stack as a DAG known a priori, which lets the runtime anticipate future memory accesses along the declared workflow\. For example, KVFlow\(Panet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib4)\)declares the agent step graph via SGLangsgl\.functionannotations and scores eviction by step\-distance on the declared graph\. SAGA\(Guoet al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib7)\)schedules agent workflows atomically using Agent Execution Graphs supplied by the agent framework, and falls back to runtime pattern inference at a15\.6%15\.6\\%TCT penalty when framework hints are unavailable\. Parrot\(Linet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib6)\)constructs a request DAG from developer\-supplied Semantic Variables that annotate input/output placeholders in prompts, and uses the DAG to colocate dependent requests and share common prompt prefixes across them\. PBKV\(Zhenget al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib49)\)also predicts agent invocations to guide eviction and prefetching, but its neural predictor is trained offline per workload and reuses KV only within a single workflow execution\. These systems rely on developer annotations or framework\-provided execution graphs, but in real\-world agentic workflows such a static DAG often cannot be constructed beforehand\.CacheScoutinstead learns the per\-workload transition matrix online without any deploy\-time\-declared graph\.

\\noindentparagraph

KV\-cache retention across execution stalls\.Agentic workloads interleave LLM generation with external function calls; while a tool runs, the GPU pauses, the session’s KV is evicted under concurrent traffic, and the next step pays a costly re\-prefill when the tool returns\. One line of work prevents the session’s KV from being evicted during the tool call, either based on the predicted call duration or on the request’s memory footprint over the call’s lifetime\(Liet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib5); Abhyankaret al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib8); Shahoutet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib9)\)\. Another line of work fills the tool pause with other agents’ work and reloads the paused agent’s KV from CPU just before the tool returns\(Bianet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib12); Luoet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib10); Weiet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib11)\)\. These approaches retain only the current agent’s KV state within a session\.CacheScoutaddresses a different and broader problem: it prefetches the next agent’s anchor at the runtime layer before the call arrives, hitting the cache even when the next call dispatches to a different agent or a different session\.

\\noindentparagraph

Faster agent serving\.The latency of individual agents is constrained by the LLM’s sequential autoregressive decoding\. Unlike chatbot interactions, agentic workflows are inherently structured, and a line of work accelerates serving by exploiting that structure to bypass or parallelize token\-by\-token generation\. Token\-level methods bypass autoregression via structured generation, constraining output to a grammar so that deterministic spans need not be sampled token\-by\-token\(Donget al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib14); Willard and Louf,[2023](https://arxiv.org/html/2608.14624#bib.bib15)\)\. Sequence\-level methods speculatively execute the agent’s predicted next step in parallel with the LLM’s deliberation\(Suiet al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib16); Roet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib17)\)\. At the framework level, Nalar\(Lajuet al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib54)\)decouples workflow specification from execution and schedules agent invocations through dependency\-carrying futures\. Runtime layers have also been proposed to enforce serving policies for agentic workloads\(Zhanget al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib52); Zhang and Hu,[2025](https://arxiv.org/html/2608.14624#bib.bib51)\)\. This line of work is complementary and orthogonal toCacheScout, and can be combined withCacheScoutto further increase the efficiency of agent inference\.

\\noindentparagraph

Other KV\-cache optimizations\.Other work optimizes KV\-cache management along complementary dimensions, including non\-prefix KV\-cache sharing with approximation\(Yaoet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib18); Gimet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib19)\), lossy KV\-cache reuse with lossless output guarantees\(Yaoet al\.,[2026](https://arxiv.org/html/2608.14624#bib.bib53)\), joint KV\-cache compression and eviction\(Fenget al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib50)\), cross\-LLM KV\-cache reusing with partial layer recomputation\(Liuet al\.,[2024](https://arxiv.org/html/2608.14624#bib.bib20)\), request routing that is aware of KV\-cache location\(Srivatsaet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib21)\), and improving paged attention for multi\-modality models with different shapes of KV\-cache\(Tuet al\.,[2025](https://arxiv.org/html/2608.14624#bib.bib22)\)\. While this line of work also makes KV\-cache in LLM inference more practical in multiple use cases,CacheScoutis complementary to them as it targets agent\-level reuse across session boundaries, an axis orthogonal to these per\-block optimizations\.

## 7\.Conclusion

CacheScoutis an agent\-aware KV\-cache runtime layer for multi\-agent LLM serving that improves cache reuse by leveraging agent execution semantics\. By learning execution transitions online, it enables predictive cache management without requiring predefined workflow graphs, offline training, or modifications to the serving critical path\. Experiments on representative real\-world multi\-agent workloads show thatCacheScoutsubstantially improves KV\-cache hit rate, reduces TTFT and end\-to\-end latency, and increases serving throughput\.CacheScouthas been implemented on top of vLLM and will be open\-sourced along with the benchmark suite\.CacheScoutdemonstrates that incorporating agent execution semantics into KV\-cache management is an effective and practical approach for efficient multi\-agent LLM serving\.

## References

- R\. Abhyankar, Z\. He, V\. Srivatsa, H\. Zhang, and Y\. Zhang \(2024\)Infercept: efficient intercept support for augmented large language model inference\.arXiv preprint arXiv:2402\.01869\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- A\. V\. Aho, P\. J\. Denning, and J\. D\. Ullman \(1971\)Principles of optimal page replacement\.Journal of the ACM18\(1\),pp\. 80–93\.Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p4.2),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2)\.
- N\. Beckmann, H\. Chen, and A\. Cidon \(2018\)LHD: improving cache hit rate by maximizing hit density\.In15th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 18\),pp\. 389–403\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p6.1),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.5.1.1.1.3)\.
- Z\. Bian, F\. Wu, T\. Ma, and Y\. Zhuo \(2025\)Tokencake: a kv\-cache\-centric serving framework for llm\-based multi\-agent applications\.arXiv preprint arXiv:2510\.18586\.Cited by:[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.4.1.1.1.3),[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Figure 2](https://arxiv.org/html/2608.14624#S1.F2),[Figure 2](https://arxiv.org/html/2608.14624#S1.F2.4.2),[§5\.1](https://arxiv.org/html/2608.14624#S5.SS1.p2.2)\.
- CrewAI \(2024\)CrewAI: framework for orchestrating role\-playing, autonomous ai agents\.Note:[https://github\.com/crewAIInc/crewAI](https://github.com/crewAIInc/crewAI)Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p1.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.6.1.1.1.3)\.
- Y\. Dong, C\. F\. Ruan, Y\. Cai, Z\. Xu, Y\. Zhao, R\. Lai, and T\. Chen \(2025\)Xgrammar: flexible and efficient structured generation engine for large language models\.Proceedings of Machine Learning and Systems7\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. Mao \(2025\)Deepresearch bench: a comprehensive benchmark for deep research agents\.arXiv preprint arXiv:2506\.11763\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- S\. Feng, Y\. Liu, H\. Li, X\. Chen, S\. Shen, K\. Du, Z\. Gu, R\. Zhang, Y\. Huang, Y\. Cheng,et al\.\(2025\)EVICPRESS: joint kv\-cache compression and eviction for efficient llm serving\.arXiv preprint arXiv:2512\.14946\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. Wang \(2023\)Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. Zhong \(2024\)Prompt cache: modular attention reuse for low\-latency inference\.Proceedings of Machine Learning and Systems6,pp\. 325–338\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- D\. Guo, J\. Wu, and S\. M\. Yiu \(2026\)SAGA: workflow\-atomic scheduling for ai agent inference on gpu clusters\.arXiv preprint arXiv:2605\.00528\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- Y\. Huang, Y\. Chen, H\. Zhang, K\. Li, H\. Zhou, M\. Fang, L\. Yang, X\. Li, L\. Shang, S\. Xu,et al\.\(2025\)Deep research agents: a systematic examination and roadmap\.arXiv preprint arXiv:2506\.18096\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- S\. Jiang and X\. Zhang \(2002\)LIRS: an efficient low inter\-reference recency set replacement policy to improve buffer cache performance\.InProceedings of the 2002 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems,pp\. 31–42\.Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p4.2),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Cited by:[Figure 2](https://arxiv.org/html/2608.14624#S1.F2),[Figure 2](https://arxiv.org/html/2608.14624#S1.F2.4.2),[§1](https://arxiv.org/html/2608.14624#S1.p1.1),[§5\.1](https://arxiv.org/html/2608.14624#S5.SS1.p2.2)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p3.1),[§1](https://arxiv.org/html/2608.14624#S1.p4.2),[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p3.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.3.1.1.1.3),[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- M\. Laju, D\. Son, S\. Agarwal, N\. Kedia, M\. Lee, J\. Srinivasa, and A\. Akella \(2026\)Nalar: an agent serving framework\.arXiv preprint arXiv:2601\.05109\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- LangChain \(2024\)LangGraph\.Note:[https://github\.com/langchain\-ai/langgraph](https://github.com/langchain-ai/langgraph)Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p1.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.6.1.1.1.3)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- H\. Li, R\. He, Q\. Mang, Q\. Zhang, H\. Mao, X\. Chen, H\. Zhou, A\. Cheung, J\. Gonzalez, and I\. Stoica \(2025\)Continuum: efficient and robust multi\-turn llm agent scheduling with kv cache time\-to\-live\.arXiv preprint arXiv:2511\.02230\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.4.1.1.1.3),[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- C\. Lin, Z\. Han, C\. Zhang, Y\. Yang, F\. Yang, C\. Chen, and L\. Qiu \(2024\)Parrot: efficient serving of\{\\\{llm\-based\}\\\}applications with semantic variable\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 929–945\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- Y\. Liu, Y\. Cheng, J\. Yao, Y\. An, X\. Chen, S\. Feng, Y\. Huang, S\. Shen, R\. Zhang, K\. Du,et al\.\(2025\)Lmcache: an efficient kv cache layer for enterprise\-scale llm inference\.arXiv preprint arXiv:2510\.09665\.Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p3.2)\.
- Y\. Liu, Y\. Huang, J\. Yao, S\. Feng, Z\. Gu, K\. Du, H\. Li, Y\. Cheng, J\. Jiang, S\. Lu,et al\.\(2024\)DroidSpeak: kv cache sharing for cross\-llm communication and multi\-llm serving\.arXiv preprint arXiv:2411\.02820\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- M\. Luo, X\. Shi, C\. Cai, T\. Zhang, J\. Wong, Y\. Wang, C\. Wang, Y\. Huang, Z\. Chen, J\. E\. Gonzalez,et al\.\(2025\)Autellix: an efficient serving engine for llm agents as general programs\.arXiv preprint arXiv:2502\.13965\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- R\. L\. Mattson, J\. Gecsei, D\. R\. Slutz, and I\. L\. Traiger \(1970\)Evaluation techniques for storage hierarchies\.IBM Systems journal9\(2\),pp\. 78–117\.Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p4.2),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2)\.
- N\. Megiddo and D\. S\. Modha \(2003\)\{\\\{arc\}\\\}: A\{\\\{self\-tuning\}\\\}, low overhead replacement cache\.In2nd USENIX Conference on File and Storage Technologies \(FAST 03\),Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p4.2),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2)\.
- G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)Gaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[Figure 2](https://arxiv.org/html/2608.14624#S1.F2),[Figure 2](https://arxiv.org/html/2608.14624#S1.F2.4.2),[§5\.1](https://arxiv.org/html/2608.14624#S5.SS1.p2.2)\.
- NVIDIA \(2023\)TensorRT\-LLM\.Note:[https://github\.com/NVIDIA/TensorRT\-LLM](https://github.com/NVIDIA/TensorRT-LLM)Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p3.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.3.1.1.1.3)\.
- OpenAI \(2024\)Swarm: educational framework exploring ergonomic, lightweight multi\-agent orchestration\.Note:[https://github\.com/openai/swarm](https://github.com/openai/swarm)Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p1.2)\.
- Z\. Pan, A\. D\. PATEL, Y\. Shen, Z\. Hu, Y\. Guan, W\. Li, L\. Qin, Y\. Wang, and Y\. Ding \(2025\)KVFlow: efficient prefix caching for accelerating llm\-based multi\-agent workflows\.Advances in Neural Information Processing Systems38,pp\. 126246–126265\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.4.1.1.1.3),[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- Y\. Ro, H\. Qiu, Í\. Goiri, R\. Fonseca, R\. Bianchini, A\. Akella, Z\. Wang, M\. Erez, and E\. Choukse \(2025\)Sherlock: reliable and efficient agentic workflow execution\.arXiv preprint arXiv:2511\.00330\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- R\. Shahout, C\. Liang, S\. Xin, Q\. Lao, Y\. Cui, M\. Yu, and M\. Mitzenmacher \(2025\)Fast inference for augmented large language models\.Advances in Neural Information Processing Systems38,pp\. 71562–71591\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- Z\. Song, D\. S\. Berger, K\. Li, and W\. Lloyd \(2020\)Learning relaxed belady for content distribution network caching\.In17th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 20\),pp\. 529–544\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p6.1),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.5.1.1.1.3)\.
- V\. Srivatsa, Z\. He, R\. Abhyankar, D\. Li, and Y\. Zhang \(2025\)Preble: efficient distributed prompt scheduling for llm serving\.InInternational conference on learning representations,Vol\.2025,pp\. 37057–37082\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- Y\. Sui, H\. Zhao, R\. Ma, Z\. He, H\. Wang, J\. Li, and Y\. Yang \(2026\)Parallelizing tool execution and LLM generation for low\-latency agent serving\.arXiv preprint arXiv:2603\.18897\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- A\. H\. M\. Syarubany and C\. D\. Yoo \(2025\)PentaRAG: large\-scale intelligent knowledge retrieval for enterprise llm applications\.arXiv preprint arXiv:2506\.21593\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- D\. Tu, D\. Vashchilenko, Y\. Lu, and P\. Xu \(2025\)VL\-cache: sparsity and modality\-aware kv cache compression for vision\-language model inference acceleration\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 219–239\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- G\. Vietri, L\. V\. Rodriguez, W\. A\. Martinez, S\. Lyons, J\. Liu, R\. Rangaswami, M\. Zhao, and G\. Narasimhan \(2018\)Driving cache replacement with ml\-based lecar\.In10th USENIX Workshop on Hot Topics in Storage and File Systems \(HotStorage 18\),Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p6.1),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.5.1.1.1.3)\.
- X\. Wei, J\. Zhang, H\. Li, J\. Chen, H\. Guan, R\. Qu, M\. Li, X\. Chen, and G\. Luo \(2025\)Agent\. xpu: efficient scheduling of agentic llm workloads on heterogeneous soc\.arXiv preprint arXiv:2506\.24045\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p2.2)\.
- B\. T\. Willard and R\. Louf \(2023\)Efficient guided generation for large language models\.arXiv preprint arXiv:2307\.09702\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2024\)Autogen: enabling next\-gen llm applications via multi\-agent conversation\.InFirst conference on language modeling,Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p1.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.6.1.1.1.3)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 50528–50652\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- J\. Yang, Z\. Mao, Y\. Yue, and K\. Rashmi \(2023\)GL\-Cache: group\-level learning for efficient and high\-performance caching\.In21st USENIX Conference on File and Storage Technologies \(FAST 23\),pp\. 115–134\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p6.1),[§2\.3](https://arxiv.org/html/2608.14624#S2.SS3.p2.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.5.1.1.1.3)\.
- J\. Yao, H\. Li, Y\. Liu, S\. Ray, Y\. Cheng, Q\. Zhang, K\. Du, S\. Lu, and J\. Jiang \(2025\)Cacheblend: fast large language model serving for rag with cached knowledge fusion\.InProceedings of the twentieth European conference on computer systems,pp\. 94–109\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- J\. Yao, S\. Shen, K\. Du, S\. Feng, D\. Seo, R\. Zhang, Y\. Huang, Y\. Liu, S\. Lu, and J\. Jiang \(2026\)VeriCache: turning lossy kv cache into lossless llm inference\.arXiv preprint arXiv:2605\.17613\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p4.2)\.
- H\. Ye, Z\. Gao, M\. Ma, Q\. Wang, Y\. Fu, M\. Chung, Y\. Lin, Z\. Liu, J\. Zhang, D\. Zhuo,et al\.\(2025\)Kvcomm: online cross\-context kv\-cache communication for efficient llm\-based multi\-agent systems\.Advances in Neural Information Processing Systems38,pp\. 17882–17928\.Cited by:[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.4.1.1.1.3)\.
- R\. Zhang and L\. Hu \(2025\)Enabling fairness across multi\-modal and multi\-agent applications\.In2025 IEEE International Conference on Edge Computing and Communications \(EDGE\),pp\. 90–92\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- R\. Zhang, C\. Kim, and L\. Hu \(2026\)A policy\-driven runtime layer for agentic llm serving\.arXiv preprint arXiv:2605\.27744\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p3.2)\.
- W\. Zhang, X\. Li, Y\. Zhang, P\. Jia, Y\. Wang, H\. Guo, Y\. Liu, and X\. Zhao \(2025\)Deep research: a survey of autonomous research agents\.arXiv preprint arXiv:2508\.12752\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- Y\. Zhang, H\. Ruan, Z\. Fan, and A\. Roychoudhury \(2024\)Autocoderover: autonomous program improvement\.InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,pp\. 1592–1604\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p1.1)\.
- H\. Zheng, F\. Fu, J\. Wu, B\. Yuan, Y\. Zhang, H\. Wang, Y\. Zhu, X\. Yan, and J\. Jiang \(2026\)Efficient serving for dynamic agent workflows with prediction\-based kv\-cache management\.arXiv preprint arXiv:2605\.06472\.Cited by:[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[Figure 2](https://arxiv.org/html/2608.14624#S1.F2),[Figure 2](https://arxiv.org/html/2608.14624#S1.F2.4.2),[§5\.1](https://arxiv.org/html/2608.14624#S5.SS1.p2.2)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.\(2024\)Sglang: efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[§1](https://arxiv.org/html/2608.14624#S1.p3.1),[§1](https://arxiv.org/html/2608.14624#S1.p4.2),[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p3.2),[Table 1](https://arxiv.org/html/2608.14624#S2.T1.2.3.1.1.1.3),[§6](https://arxiv.org/html/2608.14624#S6.p1.1)\.
- K\. Zhu, Y\. Zhao, L\. Zhao, G\. Zuo, Y\. Gu, D\. Xie, Y\. Gao, Q\. Xu, T\. Tang, Z\. Ye, K\. Kamahori, C\. Lin, S\. Wang, A\. Krishnamurthy, and B\. Kasikci \(2024\)NanoFlow: towards optimal large language model serving throughput\.arXiv preprint arXiv:2408\.12757\.Cited by:[§2\.1](https://arxiv.org/html/2608.14624#S2.SS1.p3.2)\.