TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

# TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
Source: [https://arxiv.org/html/2608.25523](https://arxiv.org/html/2608.25523)
Hongqiu Ni1, Han Tian1, Chi Zhang2, Guopeng Li1and Haisheng Tan1Affiliation:1University of Science and Technology of China, ChinaAffiliation:2Hefei University of Technology, ChinaAffiliation:

###### Abstract

Prefix caching introduces a fundamental tradeoff in multi\-agent large language model \(LLM\) serving: retaining a long system\-prompt key–value \(KV\) cache for an agent accelerates future calls, yet it reduces the GPU memory available for batching concurrent requests\. In multi\-stage workflows, existing schedulers tend to prioritize either immediate prefix locality oroverallworkflow progress\. However, under a shared KV cache budget, optimizing either objective in isolation can prolong task\-level job completion time \(JCT\) through downstream delays or frequent prefix replacement\. To strike a balance, we here proposeTopas, a Task\-Oriented Prefix\-Aware Schedulerthat jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution\.Topasscores candidate post\-decision statesby trading off the expected reduction in each task’s longest remaining service path against the near\-term benefit of downstream prefix reuse, accounting for the costs of prefix movement and preemption\. A task‑level aging mechanism is also incorporated to prevent starvation\.We implementTopaswithin the SGLang framework and assess its performance on three synthetic DAGs and two MetaGPT software‑development workflows\. Compared with the best\-performing baseline for each workload and metric,Topasreduces the mean/p99 JCT by up to 39\.8%/49\.4% on the synthetic workloads, while lowering mean JCT by 9\.8% on MetaGPT\-SOP and mean/p99 JCT by 22\.0%/26\.6% on MetaGPT\-TL\.

## IIntroduction

Multi\-agent LLM applications tackle complex tasks via interdependent requests generated by role\-specialized agents\[[32](https://arxiv.org/html/2608.25523#bib.bib12),[20](https://arxiv.org/html/2608.25523#bib.bib38)\]\. Each request contains an agent\-specific static prefix whose KV cache is reusable across requests from the same agent in different tasks\.Serving systems such as SGLang maintain cached KV pairs in a radix tree, where requests sharing the same agent prefix traverse the static path and each appends its own request‑specific dynamic content\.Retaining this cache in GPU memory—referred to as*prefix residency*—eliminates redundant prefill computations, yet it contends with concurrently executing requests for the limited KV budget\. Long system prompts further amplify both the reuse benefit and the residency cost\. Consequently, request scheduling influences not only task progress but also the overall serving capacity of the GPU\.

Mainstream inference frameworks such as SGLang focus on optimizing batching, prefix caching, and KV cache management\[[11](https://arxiv.org/html/2608.25523#bib.bib1),[37](https://arxiv.org/html/2608.25523#bib.bib3)\]\. Their schedulers primarily operate at the request level, ranking ready requests using engine‑local information\. Although well suited for per‑request execution optimization, this request‑level view offers only an indirect reflection of the inter‑request dependencies and program‑level progress that the orchestration layer monitors\.

Recent efforts aim to bridge this abstraction gap through distinct strategies\. Parrot and Autellix incorporate application dataflow or program\-level progress into request scheduling\[[13](https://arxiv.org/html/2608.25523#bib.bib4),[16](https://arxiv.org/html/2608.25523#bib.bib6)\], whereas KVFlow leverages workflow context for prefix\-cache placement\[[18](https://arxiv.org/html/2608.25523#bib.bib14)\]\. These systems primarily optimize application progress, cache availability, or resource placement, while prefix residency remains largely separate from request admission\. This decoupling can leave heterogeneous prefixes co‑residing in the cache—reducing the feasible concurrent batch size—or provoke repeated prefix evictions and reloads as execution shifts across agents\.

An inherent conflict exists between end\-to\-end progress against prefix reuse\.Consolidating requests from the same agent allows fewer resident prefixes to support a larger batch, yet it can stall upstream execution while downstream tasks remain pending\. Conversely, prioritizing workflow progress tends to trigger frequent agent switching, sacrificing reuse and incurring significant prefix‑movement overhead\. Consequently, a scheduler must reconcile prefix locality with holistic progress, rather than optimizing either objective in isolation\.

We therefore pose the following question:How can a serving system coordinate prefix residency and request admission throughout workflow execution to minimize*task\-level*job completion time \(JCT\) under a constrained GPU\-memory budget?We formalize this problem as*online prefix\-state scheduling*\. At each scheduling point, the scheduler jointly decides which agent prefixes to retain and which ready requests to admit, subject to a shared KV cache budget\. This joint decision determines the current batch capacity, immediate task progress, and the set of prefixes most likely to be reused by subsequent workflow steps\.

To solve this problem, we proposeTopas, a Task\-Oriented Prefix\-Aware Scheduler whose key novelty lies in treating prefix residency as an explicit workflow\-level scheduling decision, jointly optimized with request admission, rather than as an indirect byproduct of request ordering and cache eviction\. To our knowledge,Topasis the first multi\-agent LLM scheduler to jointly optimize prefix residency and request admission for task\-level JCT under a shared KV budget\.Topasconducts a hierarchical search to construct post\-decision GPU states, each specifying a resident\-prefix set and a compatible allocation of admitted requests\. It selects the optimal state via a JCT\-oriented utility that trades off anticipated reduction in each task’s longest remaining service path and near\-term prefix reuse, accounting for the costs of prefix movement and preemption\.Our contributions are summarized as:

- •We formalize the online prefix\-state scheduling problem, which jointly selects resident agent prefixes and admitted requests under a shared KV cache budget\. Our diagnostic analysis reveals that heterogeneous prefix co‑residency reduces dynamic batch capacity,while locality‑first and progress‑first policies fail in complementary leading to bad performance\.
- •We introduceTopas, an online scheduler that jointly determines prefix residency and request admission via a JCT‑oriented utility function\. This utility balances expected reductions in tasks’ longest remaining service paths and near\-term prefix reuse\. A task\-level aging mechanism is also incorporated to prevent starvation\.
- •We implementTopasatop SGLang and evaluate it on three synthetic DAG workloads and two MetaGPT workloads\. Compared with the best\-performing baseline per workload and metric,Topasreduces mean/p99 JCT by up to 39\.8%/49\.4% on the synthetic workloads; lowers mean JCT by 9\.8% on MetaGPT\-SOP; and achieves mean and p99 JCT reductions by 22\.0% and 26\.6%, respectively, on MetaGPT\-TL\.

## IIMotivation: A Progress–Reuse Conflict

Task\-level policies can use workflow dependencies to decide which tasks or stages should advance, but their decisions are ultimately executed by a request\-level serving engine\. The engine repeatedly admits ready requests into a continuous batch under a finite KV budget\. Same\-agent requests reuse a resident static\-prefix cache, while each request consumes additional KV capacity for its dynamic suffix and decode tokens\. Admission order and cache eviction therefore determine both which prefixes remain resident and how many requests can run together\. Task\-level priority alone does not control this prefix–batch interaction; the following diagnostics isolate its two consequences\.

To examine how request order affects batching, we use a two\-agent microbenchmark on the native LLM serving runtime\. The workload containsnnindependent requests for each of agentsAAandBB\. The agents have distinct long prefixes, while all requests use the same dynamic suffix and decode length\.Alternatingorders them asA,B,A,B,…A,B,A,B,\\ldots, keeping both prefixes resident for most of the run, whereasGroupedserves each agent’s requests together\. Both orders contain the same requests and use the same KV budget; only their order differs\.

TABLE I:Under FCFS, heterogeneous prefix co\-residency shrinks the running batch, thereby increasing makespan\. Slash\-separated values are Grouped/Alternating\.Table[I](https://arxiv.org/html/2608.25523#S2.T1)shows thatgroupeddoubles the average running batch size;alternatingtakes1\.81\.8–1\.9×1\.9\\timesas long to finish the workload\. Serving same\-agent requests together lets one resident prefix support a larger batch\. Alternating between agents instead divides KV capacity across both long prefixes, leaving less space for dynamic suffixes and decode tokens\.

The batching benefit in Diagnostic 1 suggests a natural response: favor requests that can reuse a resident prefix\. Longest Prefix Match \(LPM\) is a representative locality\-first policy: it prioritizes ready requests with the longest immediately reusable prefix\. Yet it does not actively choose a target prefix state\. It operates in a reactive, arrival\-driven manner: request order and cache eviction determine prefix residency, which then becomes the locality signal for the next decision\. In anA→B→CA\\\!\\to\\\!B\\\!\\to\\\!Cworkflow with continuous arrivals atAA, this feedback keeps service near the workflow entrance and delays stagesBBandCC\. To test whether active control can break this behavior, we augment LPM with a simple downstream gate that prioritizesCConce its backlog accumulates\. Figure[1](https://arxiv.org/html/2608.25523#S2.F1)shows that this coarse intervention achieves lower mean JCT than LPM\.

Progress\-first scheduling lies at the other extreme\. Shortest Remaining Processing Time \(SRPT\) prioritizes tasks with shorter remaining processing time and can move them toward the workflow exit\. Yet it has no explicit notion of prefix residency: repeatedly following the shortest remaining work switches between agents, causing prefix transfers or recomputation and reducing effective service capacity\. At the same operating point, the downstream\-gated policy also achieves lower mean JCT than SRPT\. Thus, neither immediate prefix locality nor task progress alone is sufficient to minimize task\-level JCT\.

Fig\. 1:Chain\-3 at 0\.15 task/s under locality\-first LPM, progress\-first SRPT, and downstream gating\.Figure[2](https://arxiv.org/html/2608.25523#S2.F2)exposes the mechanism behind this conflict\. SRPT pushes one task through the chain, but each transition to the next stage changes the resident prefix and pays another setup cost\. LPM instead batches same\-prefix requests to amortize setup, but this locality\-first order holds service at upstream stages and starves downstream progress\. The Gantt chart therefore shows how prioritizing task progress can sacrifice prefix locality, while preserving locality can delay workflow completion\.

![Refer to caption](https://arxiv.org/html/2608.25523v1/jizhi.png)Fig\. 2:A Gantt chart comparing LPM and SRPT on anA→B→CA\\\!\\to\\\!B\\\!\\to\\\!Cworkflow\. GPU memory can hold the KV cache for only one Agent prefix at a time; each request takes four time units, and each prefix setup takes one time unit\.An effective scheduler should treat prefix residency as an explicit decision\. It should group requests that share a prefix to preserve batching efficiency, but move service to another prefix when continued reuse delays task completion\. The goal is to coordinate prefix reuse and task progress within each scheduling decision\.

## IIIProblem Formulation

All tasks follow the same known DAGG=\(V,E\)G=\(V,E\)\. TaskJiJ\_\{i\}arrives at timeτi\\tau\_\{i\}\. Each nodev∈Vv\\in Vis a stage executed by an agent and becomes ready after all of its predecessors complete\. A stage may issue one or more LLM requests before producing its output\. ReAct iterations and tool delays are not known in advance; they are runtime events that determine when subsequent requests and stages become ready\. Requests produced by agentaashare a static prefix of lengthℓa\\ell\_\{a\}\. LetCi,vC\_\{i,v\}be the completion time of stagevvin taskJiJ\_\{i\}\. The task completion time and task\-level JCT are

C⁡\(Ji\)=maxv∈V⁡Ci,v,JCT⁡\(Ji\)=C⁡\(Ji\)−τi\.C\(J\_\{i\}\)=\\max\_\{v\\in V\}C\_\{i,v\},\\qquad\\mathrm\{JCT\}\(J\_\{i\}\)=C\(J\_\{i\}\)\-\\tau\_\{i\}\.\(1\)The serving objective is to minimize average task\-level JCT under online arrivals\.

At a scheduling pointtt, letSt=\(Rt,Qt\)S\_\{t\}=\(R\_\{t\},Q\_\{t\}\)denote the current GPU state, whereRtR\_\{t\}is the set of resident agent prefixes andQtQ\_\{t\}is the set of running requests\. A scheduling action selects a post\-decision stateS′=\(R′,Q′\)S^\{\\prime\}=\(R^\{\\prime\},Q^\{\\prime\}\)\. Requests inQ′∖QtQ^\{\\prime\}\\setminus Q\_\{t\}are newly admitted, while requests inQt∖Q′Q\_\{t\}\\setminus Q^\{\\prime\}are preempted\. A request can belong toQ′Q^\{\\prime\}only when its agent prefix belongs toR′R^\{\\prime\}\. We exclude idle resident prefixes, soR′=\{a⁡\(r\):r∈Q′\}R^\{\\prime\}=\\\{a\(r\):r\\in Q^\{\\prime\}\\\}\. The selected joint state must satisfy

∑a∈R′ℓa\+∑r∈Q′dr​\(t\)≤Bt,\\sum\_\{a\\in R^\{\\prime\}\}\\ell\_\{a\}\+\\sum\_\{r\\in Q^\{\\prime\}\}d\_\{r\}\(t\)\\leq B\_\{t\},\(2\)wheredr​\(t\)d\_\{r\}\(t\)accounts for the request’s dynamic\-suffix KV, retained output tokens, and decode reservation, andBtB\_\{t\}is the available token\-equivalent KV capacity\. Multiple requests for the same agent pay the static prefix cost once\.

The scheduler observes ready requests, the current GPU state, task arrival times, stage progress, and the workflow topology and agent mapping\. Future task arrivals, future ready times, ReAct iterations, and unexpanded tool outputs are unknown\.

## IVTopas: Workflow\-Aware Prefix\-State Scheduling

### IV\-ADesign Overview

The two diagnostics in Section[II](https://arxiv.org/html/2608.25523#S2)point to a limitation of request\-level scheduling\. When requests are chosen one at a time, prefix residency is determined indirectly by request order and the serving runtime’s cache\-eviction policy\. Different agent prefixes may therefore remain resident together and leave too little KV capacity for dynamic suffixes and decoding\. At the same time, service may stay on a reusable prefix even when another workflow stage matters more to task completion\.Topasinstead schedules a post\-decision prefix–request state, directly choosing both the resident prefixes and the requests served under them\.

For each candidate prefix set,Topasconstructs a compatible request allocation under the joint KV budget\. Its base utility \(Section[IV\-B](https://arxiv.org/html/2608.25523#S4.SS2)\) uses the workflow DAG to estimate the reduction in tasks’ remaining LLM\-service paths, balancing this progress against prefix movement and preemption\. A one\-hop lookahead values prefixes likely to serve newly released downstream work \(Section[IV\-C](https://arxiv.org/html/2608.25523#S4.SS3)\), while task aging prevents new arrivals from repeatedly overtaking older tasks \(Section[IV\-D](https://arxiv.org/html/2608.25523#S4.SS4)\)\. Section[IV\-E](https://arxiv.org/html/2608.25523#S4.SS5)searches these joint states by enumerating small agent pools and using bounded greedy search for larger ones\. The resulting event\-level score uses expected remaining\-path reduction as a heuristic estimate of progress toward task completion\.Topasgreedily commits the highest\-scoring generated next state at each scheduling event and reevaluates the decision as the system state changes\.

### IV\-BBase Transition Utility

For a candidate stateS′S^\{\\prime\}, letA=Q′∖QtA=Q^\{\\prime\}\\setminus Q\_\{t\}andP=Qt∖Q′P=Q\_\{t\}\\setminus Q^\{\\prime\}denote its newly admitted and preempted requests\. Candidate feasibility follows the joint prefix–request capacity constraint in Section[III](https://arxiv.org/html/2608.25523#S3)\.Topasfirst evaluates

Ubase​\(S′∣St\)=Jadmit−Jmove−Jredo−Jrevoke\.U\_\{\\mathrm\{base\}\}\(S^\{\\prime\}\\mid S\_\{t\}\)=J\_\{\\mathrm\{admit\}\}\-J\_\{\\mathrm\{move\}\}\-J\_\{\\mathrm\{redo\}\}\-J\_\{\\mathrm\{revoke\}\}\.\(3\)All terms are measured in seconds\.JadmitJ\_\{\\mathrm\{admit\}\}estimates the reduction in the selected requests’ task\-level remaining paths; the other terms charge transition delay and work invalidated by preemption\.

A request’s service time is not itself task progress\. In a DAG, advancing a non\-bottleneck branch may leave task completion unchanged\.Topastherefore values an admitted request by the expected reduction in the task’s longest remaining LLM\-service path after that request completes\. For a chain, the path is fixed and this value reduces to the expected reduction of the current stage’s residual service\. In a fork/join or a general DAG, the bottleneck branch can change as stages advance, so the value must account for the distributions of their remaining service\.

Under ReAct, letXi,v∼Fa⁡\(v\),vX\_\{i,v\}\\sim F\_\{a\(v\),v\}be the total LLM service of stagevvin taskJiJ\_\{i\}\. Each agent–stage pair has a stage\-specific profile of total LLM service\. Once completed executions are available,Fa⁡\(v\),vF\_\{a\(v\),v\}is their empirical distribution, including all ReAct rounds but excluding tool waits; before then, it is a point mass at the profiled total stage\-service value\. Letei,v​\(t\)e\_\{i,v\}\(t\)be the cumulative service recorded in the stage history\. A started but unfinished stage has residual distribution

Zi,v​\(t\)​=𝑑​Xi,v−ei,v​\(t\)\|Xi,v\>ei,v​\(t\)\.Z\_\{i,v\}\(t\)\\overset\{d\}\{=\}X\_\{i,v\}\-e\_\{i,v\}\(t\)\\;\\bigm\|\\;X\_\{i,v\}\>e\_\{i,v\}\(t\)\.\(4\)Unstarted and completed stages have residualsXi,vX\_\{i,v\}and zero, respectively\. Equation \([4](https://arxiv.org/html/2608.25523#S4.E4)\) conditions repeated ReAct rounds on service already observed without assuming a fixed number of rounds; tool waits affect readiness, not this controllable\-service estimate\.

For requestrr, letD^r​\(t\)\\widehat\{D\}\_\{r\}\(t\)be its expected LLM\-service contribution under the historical service observations for its agent–stage pair\. Its hypothetical completion updates the corresponding residual to\[Zi,v​\(t\)−D^r​\(t\)\]\+\[Z\_\{i,v\}\(t\)\-\\widehat\{D\}\_\{r\}\(t\)\]\_\{\+\}\. For an agent–stage pair without completed observations,D^r​\(t\)\\widehat\{D\}\_\{r\}\(t\)falls back to the profiled estimateTprefill​\(\|inputr\|\)\+Tdecode​max​\_​newrT\_\{\\mathrm\{prefill\}\}\(\|\\mathrm\{input\}\_\{r\}\|\)\+T\_\{\\mathrm\{decode\}\}\\mathrm\{max\\\_new\}\_\{r\}until empirical observations become available\. After completion, the realized service is recorded, andFa⁡\(v\),vF\_\{a\(v\),v\}is updated when the stage finishes\.

Let𝐙i​\(ℋ,t\)\\mathbf\{Z\}\_\{i\}\(\\mathcal\{H\},t\)be the joint residual vector after hypothetical request completionsℋ\\mathcal\{H\}\. Conditioned on the observed history,Topasmodels unfinished\-stage residuals independently and computes

L^i​\(ℋ,t\)=𝔼𝐙i​\(ℋ,t\)​\[LP⁡\(G,𝐙i​\(ℋ,t\)\)\],\\widehat\{L\}\_\{i\}\(\\mathcal\{H\},t\)=\\mathbb\{E\}\_\{\\mathbf\{Z\}\_\{i\}\(\\mathcal\{H\},t\)\}\\\!\\left\[\\operatorname\{LP\}\\\!\\left\(G,\\mathbf\{Z\}\_\{i\}\(\\mathcal\{H\},t\)\\right\)\\right\],\(5\)whereLP\\operatorname\{LP\}is the longest remaining path for a residual realization, evaluated in reverse topological order\. Requestrrfrom taskJiJ\_\{i\}then has conditional progress

Δtask​\(r∣ℋ,t\)=L^i​\(ℋ,t\)−L^i​\(ℋ∪\{r\},t\)\.\\Delta\_\{\\mathrm\{task\}\}\(r\\mid\\mathcal\{H\},t\)=\\widehat\{L\}\_\{i\}\(\\mathcal\{H\},t\)\-\\widehat\{L\}\_\{i\}\(\\mathcal\{H\}\\cup\\\{r\\\},t\)\.\(6\)The expectation is taken over the empirical conditional residual distributions of unfinished stages\. Placing it outsideLP\\operatorname\{LP\}allows the critical path to change across residual realizations and stage updates instead of fixing a path from mean durations\.

Within a candidate, requests follow the greedy order constructed in Section[IV\-E](https://arxiv.org/html/2608.25523#S4.SS5)\. LetAi<rA\_\{i\}^\{<r\}contain the earlier hypothetical request completions from the same task\. The base admission term is

Jadmit=∑r∈AΔtask​\(r∣Ai<r,t\)\.J\_\{\\mathrm\{admit\}\}=\\sum\_\{r\\in A\}\\Delta\_\{\\mathrm\{task\}\}\(r\\mid A\_\{i\}^\{<r\},t\)\.\(7\)Sequential residual updates make these terms sum to each task’s joint expected remaining\-path reduction\.

Topasmoves prefix KV caches between GPU and CPU memory as agents enter and leave the resident set\. LetRin=R′∖RtR\_\{\\mathrm\{in\}\}=R^\{\\prime\}\\setminus R\_\{t\}andRout=Rt∖R′R\_\{\\mathrm\{out\}\}=R\_\{t\}\\setminus R^\{\\prime\}\. Letμ\\mube the KV bytes per prefix token andBswapB\_\{\\mathrm\{swap\}\}the swap bandwidth\. Transfers within a transition are serialized, yielding

Tmove​\(S′∣St\)\\displaystyle T\_\{\\mathrm\{move\}\}\(S^\{\\prime\}\\mid S\_\{t\}\)=μBswap​\(∑a∈Routℓa\+∑a∈Rinℓa\),\\displaystyle=\\frac\{\\mu\}\{B\_\{\\mathrm\{swap\}\}\}\\left\(\\sum\_\{a\\in R\_\{\\mathrm\{out\}\}\}\\ell\_\{a\}\+\\sum\_\{a\\in R\_\{\\mathrm\{in\}\}\}\\ell\_\{a\}\\right\),\(8\)Jmove\\displaystyle J\_\{\\mathrm\{move\}\}=Tmove​\(S′∣St\)​\|𝒯in\|\.\\displaystyle=T\_\{\\mathrm\{move\}\}\(S^\{\\prime\}\\mid S\_\{t\}\)\|\\mathcal\{T\}\_\{\\mathrm\{in\}\}\|\.Here𝒯in=\{task\(r\):r∈A,a\(r\)∈Rin\}\\mathcal\{T\}\_\{\\mathrm\{in\}\}=\\\{\\mathrm\{task\}\(r\):r\\in A,a\(r\)\\in R\_\{\\mathrm\{in\}\}\\\}contains the distinct admitted tasks that wait for an incoming prefix\. Resident\-prefix admissions overlap with movement, and each affected task is charged once\.

SGLang reconstructs the generated\-token KV of a preempted request through autoregressive decoding\. Letyr​\(t\)y\_\{r\}\(t\)be the number of tokens to reconstruct andTdecodeT\_\{\\mathrm\{decode\}\}the profiled decode time per token\. The discarded work is

Jredo=∑r∈PTdecode​yr​\(t\)\.J\_\{\\mathrm\{redo\}\}=\\sum\_\{r\\in P\}T\_\{\\mathrm\{decode\}\}y\_\{r\}\(t\)\.\(9\)
JredoJ\_\{\\mathrm\{redo\}\}charges repeated computation;JrevokeJ\_\{\\mathrm\{revoke\}\}prevents an unfinished attempt from repeatedly collecting admission value\. At admission timetat\_\{a\},Topasstores the selected candidate’s conditional creditλr=Δtask​\(r∣Ai<r,ta\)\\lambda\_\{r\}=\\Delta\_\{\\mathrm\{task\}\}\(r\\mid A\_\{i\}^\{<r\},t\_\{a\}\)\. Preemption revokes it:

Jrevoke=∑r∈Pλr\.J\_\{\\mathrm\{revoke\}\}=\\sum\_\{r\\in P\}\\lambda\_\{r\}\.\(10\)The credit remains fixed as sibling branches progress, is cleared on completion, and is recreated on re\-admission\. Section[IV\-D](https://arxiv.org/html/2608.25523#S4.SS4)adds aging\.

### IV\-CShort\-Horizon Prefix Reuse

Current reuse is already reflected by the capacity constraint: same\-agent requests share one static\-prefix cost\. To value near\-future reuse,Topascounts downstream stages about to become ready\. Letna​\(t\)n\_\{a\}\(t\)be the number of such stages assigned to agentaawhose only unfinished predecessor is running; thus each fork child contributes separately, while a join contributes only when all other predecessors have finished\. A candidate receives

Vreuse=β​∑a∈R′na​\(t\)​μ​ℓaBswap\.V\_\{\\mathrm\{reuse\}\}=\\beta\\sum\_\{a\\in R^\{\\prime\}\}n\_\{a\}\(t\)\\frac\{\\mu\\ell\_\{a\}\}\{B\_\{\\mathrm\{swap\}\}\}\.\(11\)where the reload timeμ​ℓa/Bswap\\mu\\ell\_\{a\}/B\_\{\\mathrm\{swap\}\}values retaining an expensive prefix andβ\\betasets the pressure strength\. Lookahead stops after one workflow transition because farther demand depends on unresolved branches, ReAct rounds, and tool outcomes\. Idle prefixes remain infeasible, so this term ranks active states rather than prefetching inactive agents\.

### IV\-DTask\-Level Aging

To keep new arrivals from repeatedly overtaking older tasks with similar progress,Topasscales only admission progress by task age:

gi​\(t\)=1\+ρ​log⁡\(1\+t−τiHage\),g\_\{i\}\(t\)=1\+\\rho\\log\\\!\\left\(1\+\\frac\{t\-\\tau\_\{i\}\}\{H\_\{\\mathrm\{age\}\}\}\\right\),\(12\)whereτi\\tau\_\{i\}is the arrival time of taskJiJ\_\{i\},HageH\_\{\\mathrm\{age\}\}is the observed service\-time scale used to normalize age, andρ\\rhocontrols its strength; the supplementary material gives the update rule\. The admission term isJadmitage=∑r∈Agtask⁡\(r\)​\(t\)​Δtask​\(r∣Ai<r,t\)J\_\{\\mathrm\{admit\}\}^\{\\mathrm\{age\}\}=\\sum\_\{r\\in A\}g\_\{\\mathrm\{task\}\(r\)\}\(t\)\\Delta\_\{\\mathrm\{task\}\}\(r\\mid A\_\{i\}^\{<r\},t\)\. The stored credit uses the same admission\-time weight, so preemption revokes the value originally granted\. The full transition score is

Score⁡\(S′∣St\)=Jadmitage−Jmove−Jredo−Jrevoke\+Vreuse\.\\mathrm\{Score\}\(S^\{\\prime\}\\mid S\_\{t\}\)=J\_\{\\mathrm\{admit\}\}^\{\\mathrm\{age\}\}\-J\_\{\\mathrm\{move\}\}\-J\_\{\\mathrm\{redo\}\}\-J\_\{\\mathrm\{revoke\}\}\+V\_\{\\mathrm\{reuse\}\}\.\(13\)

### IV\-EHierarchical State Search

Direct joint\-state search is combinatorial, and task\-level utility is not separable across requests\.Topastherefore separates prefix\-set generation from request allocation\. For each setRR,GreedyPackretains running requests covered byRR, derives preemptions, and repeatedly admits the fitting request with the largest age\-weighted conditional progress per additional request\-specific KV reservation\. Same\-task marginals are updated after each choice\. It reserves an admission slot and capacity for every otherwise uncovered prefix inRR, rejects sets that cannot be covered, and emits every feasible intermediate allocation\. Packing continues through zero\-gain choices to expose complementary fork branches, preferring a represented task\. LetΦ⁡\(R\)\\Phi\(R\)be the best full score emitted forRR, or−∞\-\\inftyif none is feasible\.

For the active\-agent pool𝒜t=\{a⁡\(r\):r∈Wt∪Qt\}\\mathcal\{A\}\_\{t\}=\\\{a\(r\):r\\in W\_\{t\}\\cup Q\_\{t\}\\\},Topasenumerates𝒫⁡\(𝒜t\)\\mathcal\{P\}\(\\mathcal\{A\}\_\{t\}\)when\|𝒜t\|≤M\|\\mathcal\{A\}\_\{t\}\|\\leq M\. Otherwise it starts atRg=RtR\_\{g\}=R\_\{t\}and repeatedly takes the bestΦ\\Phi\-improving single\-agent addition\. At the first stall, it tests all remaining\-agent pairs once and, if one improvesΦ\\Phi, adds the pair and resumes single additions\. It also tests single\-agent drop and swap repairs, denotedRepair​\(R\)\\textsc\{Repair\}\(R\), fromRtR\_\{t\}and the finalRgR\_\{g\}, and retains the incumbent\. This preserves full prefix\-set enumeration for small pools and limits large\-pool search toO⁡\(\|𝒜t\|2\)O\(\|\\mathcal\{A\}\_\{t\}\|^\{2\}\)prefix\-set evaluations\.

The capKKapplies separately to admissions and preemptions:\|AR\|≤K\|A\_\{R\}\|\\leq Kand\|PR\|≤K\|P\_\{R\}\|\\leq K;K=0K=0removes both caps\. Natural completions are committed before search and do not enterPRP\_\{R\}\. From the generated feasible states𝒞^t\\widehat\{\\mathcal\{C\}\}\_\{t\},Topasselects

Stsel=arg⁡maxS′∈𝒞^t⁡Score⁡\(S′∣St\)\.S\_\{t\}^\{\\mathrm\{sel\}\}=\\arg\\max\_\{S^\{\\prime\}\\in\\widehat\{\\mathcal\{C\}\}\_\{t\}\}\\mathrm\{Score\}\(S^\{\\prime\}\\mid S\_\{t\}\)\.\(14\)The selection in Eq\. \([14](https://arxiv.org/html/2608.25523#S4.E14)\) defines an event\-level greedy policy:Topasexecutes the highest\-scoring feasible candidate and re\-optimizes at the next scheduling event\. The score uses expected remaining\-path reduction under the empirical conditional residual distributions as a heuristic estimate of progress toward task completion, together with the modeled transition costs and reuse value\.GreedyPackenforces feasibility by construction, and candidate generation bounds the search at every event\. With an explicit event\-level score,Topasevaluates and selects the generated feasible states directly\. An empty incumbent is removed once a nonempty state is available\.

Algorithm 1Hierarchical state searchInput: ready requestsWtW\_\{t\}, incumbent state\(Rt,Qt\)\(R\_\{t\},Q\_\{t\}\), budgetBtB\_\{t\}, and boundsM,KM,K Output: selected stateStselS\_\{t\}^\{\\mathrm\{sel\}\}

1:

𝒜t←\{a⁡\(r\):r∈Wt∪Qt\}\\mathcal\{A\}\_\{t\}\\leftarrow\\\{a\(r\):r\\in W\_\{t\}\\cup Q\_\{t\}\\\}and

𝒞^t←∅\\widehat\{\\mathcal\{C\}\}\_\{t\}\\leftarrow\\emptyset\.

2:if

\|𝒜t\|≤M\|\\mathcal\{A\}\_\{t\}\|\\leq Mthen

3:Add all states from

GreedyPack​\(𝒫​\(𝒜t\)\)\\textsc\{GreedyPack\}\(\\mathcal\{P\}\(\\mathcal\{A\}\_\{t\}\)\)to

𝒞^t\\widehat\{\\mathcal\{C\}\}\_\{t\}\.

4:else

5:

Rg←RtR\_\{g\}\\leftarrow R\_\{t\}and

ppair←falsep\_\{\\mathrm\{pair\}\}\\leftarrow\\mathrm\{false\}; add states from

GreedyPack​\(\{Rt\}∪Repair​\(Rt\)\)\\textsc\{GreedyPack\}\(\\\{R\_\{t\}\\\}\\cup\\textsc\{Repair\}\(R\_\{t\}\)\)\.

6:repeat

7:Form

ℛ1←\{Rg∪\{a\}:a∈𝒜t∖Rg\}\\mathcal\{R\}\_\{1\}\\leftarrow\\\{R\_\{g\}\\cup\\\{a\\\}:a\\in\\mathcal\{A\}\_\{t\}\\setminus R\_\{g\}\\\}and add

GreedyPack​\(ℛ1\)\\textsc\{GreedyPack\}\(\\mathcal\{R\}\_\{1\}\)states\.

8:ifthe best addition improves

Φ⁡\(Rg\)\\Phi\(R\_\{g\}\)then

9:Update

RgR\_\{g\}with the best addition\.

10:elseif

ppair=falsep\_\{\\mathrm\{pair\}\}=\\mathrm\{false\}then

11:Set

ppair←truep\_\{\\mathrm\{pair\}\}\\leftarrow\\mathrm\{true\}; form all

Rg∪\{a,b\}R\_\{g\}\\cup\\\{a,b\\\}once and add theirGreedyPackstates\.

12:ifthe best pair improves

Φ⁡\(Rg\)\\Phi\(R\_\{g\}\)then

13:Update

RgR\_\{g\}with the best pair\.

14:else

15:break\.

16:endif

17:else

18:break\.

19:endif

20:untilthe search terminates

21:Add states from

GreedyPack​\(Repair​\(Rg\)\)\\textsc\{GreedyPack\}\(\\textsc\{Repair\}\(R\_\{g\}\)\)and add incumbent

StS\_\{t\}\.

22:endif

23:Remove the empty incumbent if a nonempty candidate exists\.

24:return

arg⁡maxS′∈𝒞^t⁡Score⁡\(S′∣St\)\\arg\\max\_\{S^\{\\prime\}\\in\\widehat\{\\mathcal\{C\}\}\_\{t\}\}\\mathrm\{Score\}\(S^\{\\prime\}\\mid S\_\{t\}\)with deterministic ties\.

## VExperiments

Fig\. 3:End\-to\-end task completion on the synthetic DAG workloads\.### V\-AExperimental Setup

We implementTopasas a scheduling module in SGLang v0\.5\.3\[[37](https://arxiv.org/html/2608.25523#bib.bib3)\]and evaluate it on a single NVIDIA A100 80GB GPU\. The server allocates the A100 memory remaining after loading the model weights to the KV cache; we impose no workload\-specific KV limit\. In a separate MetaGPT\-SOP overhead run at 0\.15 task/s,Topasaverages 1\.9 ms per scheduling decision, and its cumulative scheduler time accounts for 0\.31% of experiment wall time\.

All workloads use Qwen2\.5\-32B\-Instruct\. At each operating point, tasks arrive according to a Poisson process whose rate is the arrival rate shown in the figures; the sampled arrival trace is fixed and shared across all policies\. We evaluate three synthetic DAG workloads:Chain\-3,DAG\-4, andDAG\-10\-Wide\. They cover linear, fork/join, and wide execution structures with increasing numbers of prefix namespaces and concurrently ready agents\. They use fixed\-length user queries constructed for the evaluation and fixed per\-stage generations, isolating topology and prefix\-state pressure from variations in user content\.

The two MetaGPT workloads use real prompts sampled from the MetaGPT SoftwareDev dataset\[[8](https://arxiv.org/html/2608.25523#bib.bib11)\]\. TheMetaGPT\-SOPworkload follows the static five\-role pipeline of earlier MetaGPT versions, whereasMetaGPT\-TLunrolls a nine\-stage path through the newer star\-shaped organization, alternating a TeamLeader with four specialists to preserve recurrent reuse of the central prefix\.

All policies run on the same SGLang backend\. We compare five baselines:\(1\) FCFSorders requests by arrival time;\(2\) Longest Prefix Match \(LPM\)prioritizes requests with the longest reusable prefix;\(3\) Parrot\-FCFS\[[13](https://arxiv.org/html/2608.25523#bib.bib4)\]orders ready requests by task arrival time, using request arrival time as the tie\-breaker;\(4\) Autellix Least\-Attained Service \(LAS\)\[[16](https://arxiv.org/html/2608.25523#bib.bib6)\]prioritizes the task with the least accumulated LLM service; and\(5\) Shortest\-Path\-First \(SPF\)prioritizes deeper workflow stages, using FCFS within a depth\. On the equal\-cost Chain\-3 workload, this ordering is equivalent to Shortest Remaining Processing Time \(SRPT\)\.

We use mean and p99 task JCT as primary metrics\. First\-task completion time and request throughput characterize initial progress and serving capacity; complete results are in the supplementary material\. Workload\-level comparisons average each metric over the reported operating points; for each JCT metric, the comparator is the baseline with the lowest average\.

### V\-BEnd\-to\-End Task Completion

Figure[3](https://arxiv.org/html/2608.25523#S5.F3)shows that, relative to the best\-performing baseline for each workload and metric,Topasreduces mean JCT by 27\.5% on Chain\-3, 39\.8% on DAG\-4, and 27\.7% on DAG\-10\-Wide\. The corresponding p99 reductions are 31\.7%, 49\.4%, and 30\.8%\.

Fig\. 4:Task completion on the MetaGPT workloads; high\-load Autellix\-LAS values exceed the displayed MetaGPT\-TL range\.Figure[4](https://arxiv.org/html/2608.25523#S5.F4)extends the evaluation to multi\-turn traces with variable call lengths\. On MetaGPT\-SOP, LPM and Autellix\-LAS attain the lowest p99 JCT and highest request throughput, respectively\. Against SPF, the strongest mean\-JCT baseline,Topasimproves all three metrics: it lowers mean/p99 JCT by 9\.8%/4\.5% and raises request throughput by 6\.7%\.

In the TeamLeader trace, a recurrent central prefix competes with heterogeneous specialist prefixes\. At light load, the policies remain close; as contention grows,Topaspulls ahead of both locality\- and progress\-first baselines\. Relative to SPF and Parrot\-FCFS, the best\-performing baselines for the two metrics,Topaslowers mean and p99 JCT by 22\.0% and 26\.6%, respectively\. The difference between SOP and TL follows their prefix dynamics: SOP advances through five roles in a largely one\-pass pipeline, whereas TL repeatedly returns to a central agent, creating more opportunities for explicit residency control\.

### V\-CAblations

Figure[5](https://arxiv.org/html/2608.25523#S5.F5)compares the full policy withTopas\-base and the two single\-component ablations on MetaGPT\-TL\.Topas\-base retains the base transition utility and joint\-state search but omits future reuse and aging\. Relative toTopas\-base, the full policy lowers mean and p99 JCT by 60\.5% and 53\.6%, respectively\. Both components contribute: compared with the two single\-component variants, the full policy further lowers mean JCT by 44\.9–51\.0% and p99 JCT by 44\.2–48\.5%\.

Fig\. 5:MetaGPT\-TL component ablation at 0\.0175 task/s\.

## VIRelated Work

LLM servers improve GPU efficiency with efficient attention kernels, continuous batching, paged KV\-cache allocation, preemption, and chunked prefill\[[5](https://arxiv.org/html/2608.25523#bib.bib28),[36](https://arxiv.org/html/2608.25523#bib.bib2),[11](https://arxiv.org/html/2608.25523#bib.bib1),[31](https://arxiv.org/html/2608.25523#bib.bib23),[2](https://arxiv.org/html/2608.25523#bib.bib24)\]; disaggregated systems separate prefill and decode\[[38](https://arxiv.org/html/2608.25523#bib.bib25),[19](https://arxiv.org/html/2608.25523#bib.bib26)\]\. Prefix\-aware systems reuse shared computation\[[37](https://arxiv.org/html/2608.25523#bib.bib3),[7](https://arxiv.org/html/2608.25523#bib.bib29),[35](https://arxiv.org/html/2608.25523#bib.bib33)\], route requests toward cached prefixes\[[27](https://arxiv.org/html/2608.25523#bib.bib5),[21](https://arxiv.org/html/2608.25523#bib.bib32)\], or manage caches across memory tiers\[[26](https://arxiv.org/html/2608.25523#bib.bib27),[6](https://arxiv.org/html/2608.25523#bib.bib30),[9](https://arxiv.org/html/2608.25523#bib.bib31)\]\. These mechanisms optimize request execution, locality, or data movement without considering how prefix residency advances a workflow under limited GPU memory\.

Agentic applications interleave model calls with tools and coordinate model or agent modules\[[33](https://arxiv.org/html/2608.25523#bib.bib13),[22](https://arxiv.org/html/2608.25523#bib.bib34),[25](https://arxiv.org/html/2608.25523#bib.bib35),[15](https://arxiv.org/html/2608.25523#bib.bib36),[10](https://arxiv.org/html/2608.25523#bib.bib37)\]\. At the serving layer, Parrot exposes application dataflow; InferCept and Continuum preserve KV caches across external waits; and Autellix and Astraea schedule calls by program progress or lifecycle state\[[13](https://arxiv.org/html/2608.25523#bib.bib4),[1](https://arxiv.org/html/2608.25523#bib.bib10),[12](https://arxiv.org/html/2608.25523#bib.bib9),[16](https://arxiv.org/html/2608.25523#bib.bib6),[17](https://arxiv.org/html/2608.25523#bib.bib7)\]\. AugServe and Teola optimize augmented applications\[[30](https://arxiv.org/html/2608.25523#bib.bib8),[28](https://arxiv.org/html/2608.25523#bib.bib17)\]; KVCOMM and DroidSpeak transfer KV caches across contexts or models\[[34](https://arxiv.org/html/2608.25523#bib.bib18),[14](https://arxiv.org/html/2608.25523#bib.bib19)\]; and multi\-agent systems optimize routing, data access, allocation, batching, and runtime execution\[[4](https://arxiv.org/html/2608.25523#bib.bib15),[29](https://arxiv.org/html/2608.25523#bib.bib16),[3](https://arxiv.org/html/2608.25523#bib.bib20),[23](https://arxiv.org/html/2608.25523#bib.bib21),[24](https://arxiv.org/html/2608.25523#bib.bib22)\]\. KVFlow is a workflow\-aware KV\-cache management system that uses an Agent Step Graph to guide KV\-node eviction and prefetching, temporarily skipping requests whose caches are still loading\[[18](https://arxiv.org/html/2608.25523#bib.bib14)\]\.Topasinstead makes the memory contention between resident agent\-prefix caches and running requests part of workflow scheduling, using DAG progress and reuse to reduce task\-level JCT\.

## VIIConclusion

Multi\-agent LLM workflows couple prefix residency with ready\-request admission under a shared GPU\-memory budget\.Topasjointly schedules these decisions, balancing expected remaining\-path reduction and near\-term reuse against prefix movement and preemption\. Relative to the best\-performing baseline for each workload and metric, it reduces mean and p99 task JCT on synthetic DAG workloads by up to 39\.8% and 49\.4%, respectively\. It lowers mean JCT on both MetaGPT workloads, with reductions reaching 22\.0%; on MetaGPT\-TL, it also lowers p99 JCT by 26\.6%\.

## References

- \[1\]\(2024\)InferCept: efficient intercept support for augmented large language model inference\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[2\]A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, A\. Tumanov, and R\. Ramjee\(2024\)Taming throughput\-latency tradeoff in LLM inference with sarathi\-serve\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[3\]A\. Amayuelas, J\. Yang, S\. Agashe, A\. Nagarajan, A\. Antoniades, X\. E\. Wang, and W\. Wang\(2025\)Self\-resource allocation in multi\-agent llm systems\.arXiv preprint arXiv:2504\.02051\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[4\]J\. Chen, J\. Shi, Q\. Chen, and M\. Guo\(2025\)Kairos: low\-latency multi\-agent serving with shared llms and excessive loads in the public cloud\.arXiv preprint arXiv:2508\.06948\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[5\]T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[6\]B\. Gao, Z\. He, P\. Sharma, Q\. Kang, D\. Jevdjic, J\. Deng, X\. Yang, Z\. Yu, and P\. Zuo\(2024\)Cost\-efficient large language model serving for multi\-turn conversations with cachedattention\.InUSENIX Annual Technical Conference \(USENIX ATC\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[7\]I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. Zhong\(2024\)Prompt cache: modular attention reuse for low\-latency inference\.InProceedings of Machine Learning and Systems \(MLSys\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[8\]S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber\(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§V\-A](https://arxiv.org/html/2608.25523#S5.SS1.p3.1)\.
- \[9\]C\. Hu, H\. Huang, J\. Hu, J\. Xu, X\. Chen, T\. Xie, C\. Wang, S\. Wang, Y\. Bao, N\. Sun, and Y\. Shan\(2024\)MemServe: context caching for disaggregated LLM serving with elastic memory pool\.arXiv preprint arXiv:2406\.17565\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[10\]O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts\(2024\)DSPy: compiling declarative language model calls into self\-improving pipelines\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[11\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th ACM Symposium on Operating Systems Principles \(SOSP\),Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p2.1),[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[12\]H\. Li, Q\. Mang, R\. He, Q\. Zhang, H\. Mao, X\. Chen, H\. Zhou, A\. Cheung, J\. E\. Gonzalez, and I\. Stoica\(2025\)Continuum: efficient and robust multi\-turn llm agent scheduling with kv cache time\-to\-live\.arXiv preprint arXiv:2511\.02230\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[13\]C\. Lin, Z\. Han, C\. Zhang, Y\. Yang, F\. Yang, C\. Chen, and L\. Qiu\(2024\)Parrot: efficient serving of llm\-based applications with semantic variable\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 929–945\.Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p3.1),[§V\-A](https://arxiv.org/html/2608.25523#S5.SS1.p4.1),[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[14\]Y\. Liu, Y\. Huang, J\. Yao, S\. Feng, Z\. Gu, K\. Du, H\. Li, Y\. Cheng, J\. Jiang, S\. Lu, M\. Musuvathi, and E\. Choukse\(2024\)DroidSpeak: kv cache sharing for cross\-llm communication and multi\-llm serving\.arXiv preprint arXiv:2411\.02820\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[15\]P\. Lu, B\. Peng, H\. Cheng, M\. Galley, K\. Chang, Y\. N\. Wu, S\. Zhu, and J\. Gao\(2023\)Chameleon: plug\-and\-play compositional reasoning with large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[16\]M\. Luo, X\. Shi, C\. Cai, T\. Zhang, J\. Wong, Y\. Wang, C\. Wang, Y\. Huang, Z\. Chen, J\. E\. Gonzalez, and I\. Stoica\(2025\)Autellix: an efficient serving engine for llm agents as general programs\.arXiv preprint arXiv:2502\.13965\.Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p3.1),[§V\-A](https://arxiv.org/html/2608.25523#S5.SS1.p4.1),[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[17\]H\. Ni, J\. Zhang, G\. Li, Z\. Wang, R\. Wu, C\. Zhang, and H\. Tan\(2025\)Astraea: a state\-aware scheduling engine for llm\-powered agents\.arXiv preprint arXiv:2512\.14142\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[18\]Z\. Pan, A\. Patel, Z\. Hu, Y\. Shen, Y\. Guan, W\. Li, L\. Qin, Y\. Wang, and Y\. Ding\(2025\)KVFlow: efficient prefix caching for accelerating llm\-based multi\-agent workflows\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p3.1),[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[19\]P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. Bianchini\(2024\)Splitwise: efficient generative LLM inference using phase splitting\.InACM/IEEE Annual International Symposium on Computer Architecture \(ISCA\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[20\]C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong,et al\.\(2024\)Chatdev: communicative agents for software development\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 15174–15186\.Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p1.1)\.
- \[21\]R\. Qin, Z\. Li, W\. He, M\. Zhang, Y\. Wu, W\. Zheng, and X\. Xu\(2024\)Mooncake: a KVCache\-centric disaggregated architecture for LLM serving\.arXiv preprint arXiv:2407\.00079\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[22\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[23\]J\. Shen, N\. Wadlom, and Y\. Lu\(2025\)Batch query processing and optimization for agentic workflows\.arXiv preprint arXiv:2509\.02121\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[24\]J\. Shen, N\. Wadlom, L\. Zhou, D\. Wang, X\. Miao, L\. Fang, and Y\. Lu\(2025\)FlowMesh: a service fabric for composable llm workflows\.arXiv preprint arXiv:2510\.26913\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[25\]Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. Zhuang\(2023\)HuggingGPT: solving AI tasks with ChatGPT and its friends in hugging face\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[26\]Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Ré, I\. Stoica, and C\. Zhang\(2023\)FlexGen: high\-throughput generative inference of large language models with a single GPU\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[27\]V\. Srivatsa, Z\. He, R\. Abhyankar, D\. Li, and Y\. Zhang\(2024\)Preble: efficient distributed prompt scheduling for llm serving\.arXiv preprint arXiv:2407\.00023\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[28\]X\. Tan, Y\. Jiang, Y\. Yang, and H\. Xu\(2024\)Teola: towards end\-to\-end optimization of llm\-based applications\.arXiv preprint arXiv:2407\.00326\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[29\]N\. Wadlom, J\. Shen, and Y\. Lu\(2026\)Efficient llm serving for agentic workflows: a data systems perspective\.arXiv preprint arXiv:2603\.16104\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[30\]Y\. Wang, Z\. Jin, J\. Xu, W\. Lin, Y\. Chen, and W\. Chen\(2025\)AugServe: adaptive request scheduling for augmented large language model inference serving\.arXiv preprint arXiv:2512\.04013\.Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[31\]B\. Wu, Y\. Zhong, Z\. Zhang, S\. Liu, F\. Liu, Y\. Sun, G\. Huang, X\. Liu, and X\. Jin\(2023\)Fast distributed inference serving for large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[32\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2023\)Autogen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p1.1)\.
- \[33\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[34\]H\. Ye, Z\. Gao, M\. Ma, Q\. Wang, Y\. Fu, M\. Chung, Y\. Lin, Z\. Liu, J\. Zhang, D\. Zhuo, and Y\. Chen\(2025\)KVCOMM: online cross\-context kv\-cache communication for efficient llm\-based multi\-agent systems\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p2.1)\.
- \[35\]L\. Ye, Z\. Tao, Y\. Huang, and Y\. Li\(2024\)ChunkAttention: efficient self\-attention with prefix\-aware KV cache and two\-phase partition\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[36\]G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun\(2022\)Orca: a distributed serving system for transformer\-based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[37\]L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng\(2024\)SGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I](https://arxiv.org/html/2608.25523#S1.p2.1),[§V\-A](https://arxiv.org/html/2608.25523#S5.SS1.p1.1),[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.
- \[38\]Y\. Zhong, S\. Liu, J\. Chen, J\. Hu, Y\. Zhu, X\. Liu, X\. Jin, and H\. Zhang\(2024\)DistServe: disaggregating prefill and decoding for goodput\-optimized large language model serving\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI\),Cited by:[§VI](https://arxiv.org/html/2608.25523#S6.p1.1)\.