HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

# HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
Source: [https://arxiv.org/html/2607.22578](https://arxiv.org/html/2607.22578)
Zhiqing Tang✉\{\}^\{\\textrm\{\{\\char 0\\relax\}\}\}Hongrui LiangJianxiong GuoJiong LouTian WangWeijia Jia

###### Abstract

The proliferation of Large Language Models \(LLMs\) has shifted serving systems from processing isolated requests to orchestrating high\-concurrency, multi\-tenant agentic workflows\. However, existing solutions typically prioritize intra\-workflow optimization, largely neglecting the significant potential for inter\-workflow optimization\. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end\-to\-end performance of concurrent workflows\. Through fine\-grained orchestration, HeraSys eliminates cross\-workflow computational redundancy via structural node merging and reuse\. Furthermore, HeraSys introduces a load\-aware joint scheduling policy that dynamically manages execution order by evaluating both inter\- and intra\-query priorities\. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput\. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2\.17×\\timesand increases serving throughput by up to 1\.85×\\timesunder strict latency guarantees\.

LLM Serving, Agentic Workflows, Scheduling, Performance Optimization

## 1Introduction

The rapid development of Large Language Models \(LLMs\) has moved modern application development from simple single\-turn conversational interfaces to complex agentic workflows composed of multiple LLM calls and external tools\. To execute these tasks, applications are typically represented as Directed Acyclic Graphs \(DAGs\) containing interdependent steps\. Currently, serving systems primarily optimize performance in a single dimension: either focusing on the dependencies between LLM inference tasks\(Kwonet al\.,[2023](https://arxiv.org/html/2607.22578#bib.bib16); Linet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib10); Zhonget al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib23); Shenet al\.,[2026](https://arxiv.org/html/2607.22578#bib.bib32)\)or the orchestration logic within a single workflow\(Chase and LangChain Team,[2024](https://arxiv.org/html/2607.22578#bib.bib43),[2025](https://arxiv.org/html/2607.22578#bib.bib44); Wuet al\.,[2024b](https://arxiv.org/html/2607.22578#bib.bib13)\)\. These approaches effectively optimize individual requests but do not address the broader context of system\-wide execution\.

In multi\-tenant settings, however, serving systems typically treat concurrent workflows as independent instances\. This approach overlooks critical application\-layer information shared across requests\. While some recent works attempt to mitigate this issue, they generally lack a holistic view of the global context, failing to effectively address the inherent redundancies and resource contention in concurrent execution\(Meiet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib14); Liuet al\.,[2025a](https://arxiv.org/html/2607.22578#bib.bib31); Linet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib10); Luoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib27); Shenet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib26)\)\. We observe that by breaking this request\-level isolation to jointly optimize cross\-workflow computational redundancy and global coordinated scheduling, significant improvements can be realized in end\-to\-end performance\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x1.png)\(a\)Data isolation and redundancy in modular orchestration\.
![Refer to caption](https://arxiv.org/html/2607.22578v1/x2.png)\(b\)Fusion opportunities exposed by fine\-grained orchestration\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x3.png)\(c\)Extension of fusion logic to agentic workflows with function calls\.

Figure 1:Computational redundancy and fusion opportunities\.The first challenge is how to mitigate the substantial overhead caused by treating workflows in isolation, which obscures critical task redundancy\(Shenet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib26); Gimet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib24)\)\. For example, in production LLM applications, multiple concurrent queries often involve embedding and retrieving the same batch of common document chunks, as shown in Figure[1\(b\)](https://arxiv.org/html/2607.22578#S1.F1.sf2)\. Such fine\-grained data redundancy allows for fusion and deduplication\. Similarly, in agentic workflows, Figure[1\(c\)](https://arxiv.org/html/2607.22578#S1.F1.sf3)illustrates that different agents may need to invoke the same tools or process similar prefix contexts\(Abhyankaret al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib18)\)\. However, as shown in Figure[1\(a\)](https://arxiv.org/html/2607.22578#S1.F1.sf1), existing modular designs lack a global view and fail to detect these overlaps, forcing the system to execute redundant tasks\. This not only increases system load but also limits system throughput\. Consequently, exploiting these fine\-grained structural commonalities via cross\-workflow fusion presents a promising avenue to eliminate computational waste\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x4.png)\(a\)FCFS: Head\-of\-Line Blocking\.
![Refer to caption](https://arxiv.org/html/2607.22578v1/x5.png)\(b\)SJF/SRTF: Long Task Starvation\.
![Refer to caption](https://arxiv.org/html/2607.22578v1/x6.png)\(c\)Ours: Resource Reservation\.

Figure 2:Mitigating blocking and starvation via resource skewing\.The second challenge is how to balance latency and fairness in scheduling, an objective complicated by workload heterogeneity\. Real\-world workloads typically mix short tasks \(e\.g\., simple QA\) and long tasks \(e\.g\., long document analysis, multi\-round iterative reasoning\)\. In this setting, simple First\-Come\-First\-Served \(FCFS\) or Round\-Robin scheduling often leads to Head\-of\-Line \(HOL\) Blocking: as shown in Figure[2\(a\)](https://arxiv.org/html/2607.22578#S1.F2.sf1), a long\-running task may occupy resources for an extended period, delaying numerous subsequent short tasks\. Conversely, as seen in Figure[2\(b\)](https://arxiv.org/html/2607.22578#S1.F2.sf2), strictly prioritizing short tasks \(SJF/SRTF\) may cause resource starvation for long tasks, resulting in unfairness\(Luoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib27); Wuet al\.,[2024a](https://arxiv.org/html/2607.22578#bib.bib28)\)\. To address this, we find that combining resource reservation for long tasks with prioritization for short tasks prevents starvation and balances performance with Quality of Service \(QoS\), as demonstrated in Figure[2\(c\)](https://arxiv.org/html/2607.22578#S1.F2.sf3)\.

In this paper, we propose HeraSys, a fine\-grained end\-to\-end serving system for multi\-workflow scenarios \(§[2\.1](https://arxiv.org/html/2607.22578#S2.SS1)\)\. HeraSys adopts a two\-layer optimization architecture: \(I\) The Pre\-Execution Graph Fuser identifies and fuses redundant subgraphs across queries before execution, generating a more efficient graph\. \(II\) The Runtime Load\-Aware Joint Scheduler dynamically prioritizes incoming query tasks, biases resource allocation towards potential bottlenecks, and adjusts batch sizes and pipeline decomposition granularity based on system load\. This ensures the system maintains high efficiency and meets service requirements under dynamic load changes\. We implement HeraSys on a server node equipped with NVIDIA RTX 4090 GPUs, using vLLM as the inference backend\. We evaluate HeraSys against representative baselines, including LangGraph\(Chase and LangChain Team,[2025](https://arxiv.org/html/2607.22578#bib.bib44)\), LlamaIndex\(Liu and LlamaIndex Team,[2025](https://arxiv.org/html/2607.22578#bib.bib45)\), and the state\-of\-the\-art system Ayo\(Tanet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib15)\), using diverse workloads such as RAG, Web Search, and mixed scenarios\. Extensive experiments validate the effectiveness of our design, showing that HeraSys delivers substantial performance gains in both end\-to\-end latency and system throughput compared to state\-of\-the\-art baselines\.

The main contributions of this paper are as follows:

1. 1\.We design a dual\-layer optimization architecture to address the bottlenecks of existing serving systems in multi\-workflow scenarios\. This architecture overcomes the traditional request\-level isolation, achieving joint optimization of resource utilization and scheduling efficiency through the integration of pre\-execution graph fusion and runtime dynamic scheduling\.
2. 2\.We propose a cross\-query graph node reuse mechanism that identifies and eliminates redundant computational subgraphs across workflows, significantly reducing computational overhead \(§[3\.3](https://arxiv.org/html/2607.22578#S3.SS3)\)\.
3. 3\.We propose a load\-aware joint scheduling strategy\. By integrating dynamic long/short query classification, biased resource allocation, and adaptive batching, this strategy effectively reduces the end\-to\-end latency of query processing \(§[3\.4](https://arxiv.org/html/2607.22578#S3.SS4)\)\.
4. 4\.We implement HeraSys and demonstrate the effectiveness of our approach through extensive experimental comparisons with state\-of\-the\-art systems\.

##### Conflict of Interest Disclosure\.

The authors declare no financial conflicts of interest related to this work\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x7.png)\(a\)Traditional workflow orchestration\.
![Refer to caption](https://arxiv.org/html/2607.22578v1/x8.png)\(b\)Fine\-grained workflow orchestration\.

Figure 3:Structural comparison of orchestration paradigms\. Fine\-grained orchestration decomposes modular architectures into fine\-grained primitive nodes, such as splitting Indexing into Embedding and Ingestion, and splitting LLM Prefilling into partial and full phases, exposing greater parallelism and optimization space\.Additionally, HeraSys decomposes LLM Decoding into sequential sub\-nodes and dynamically regulates the pipeline decomposition granularity\.

## 2Related Work

### 2\.1Fine\-Grained Workflow Orchestration

Early LLM frameworks used modular abstractions \(Figure[3\(a\)](https://arxiv.org/html/2607.22578#S1.F3.sf1)\) to simplify orchestration, obscuring upper\-layer control flow and data dependencies from serving systems\. This limited end\-to\-end optimization\(Liu and LlamaIndex Team,[2025](https://arxiv.org/html/2607.22578#bib.bib45); Chase and LangChain Team,[2024](https://arxiv.org/html/2607.22578#bib.bib43),[2025](https://arxiv.org/html/2607.22578#bib.bib44); Wuet al\.,[2024b](https://arxiv.org/html/2607.22578#bib.bib13)\)\. To address the bottlenecks of modular encapsulation, recent research has moved towards fine\-grained orchestration\. As shown in Figure[3\(b\)](https://arxiv.org/html/2607.22578#S1.F3.sf2), Ayo\(Tanet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib15)\)proposes a task\-primitive\-based approach, decoupling modular tasks into general computational nodes to enable better parallelism\. Similarly, other works focus on optimizing data pipelines\(Linet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib10); Shenet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib26); Honget al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib34); Qianet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib9); Khattabet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib8)\)\. Although these systems improve intra\-workflow efficiency by adjusting orchestration granularity, they often miss global optimization opportunities across workflows\.

### 2\.2Multi\-level Redundancy Elimination

Eliminating redundancy is key for system throughput\. At the inference level, vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.22578#bib.bib16)\)and SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib12)\)use block\-based memory and radix tree\-based prefix caching to enable KV cache reuse\. Approaches like GPTCache\(Bang,[2023](https://arxiv.org/html/2607.22578#bib.bib25)\)and Prompt Cache\(Gimet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib24)\)explore reuse based on semantic similarity or schema modularity\. Other research targets cross\-instance distributed deduplication, enabling low\-latency sharing via distributed cache pools or encoding techniques\(Qinet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib17); Yaoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib33); Shenet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib26); Liuet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib7),[2025b](https://arxiv.org/html/2607.22578#bib.bib6)\)\. While these works reduce redundancy at various levels, they fail to detect the complex global graph topology within workflows and cannot identify structural redundancies before execution\.

### 2\.3Scheduling for Heterogeneous Workflows

LLM workflows are highly heterogeneous\. While early inference schedulers introduced techniques such as iteration\-level scheduling to reduce batching\-induced blocking\(Yuet al\.,[2022](https://arxiv.org/html/2607.22578#bib.bib11); Wuet al\.,[2024a](https://arxiv.org/html/2607.22578#bib.bib28); Leviathanet al\.,[2023](https://arxiv.org/html/2607.22578#bib.bib3)\), they struggle to handle complex workflows\. To address this, DistServe\(Zhonget al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib23)\), Sarathi\-Serve\(Agrawalet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib4)\), and Splitwise\(Patelet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib5)\)decouple inference phases to improve resource utilization\. Autellix\(Luoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib27)\)analyzes HOL blocking and proposes scheduling based on attained service time\. Hermes\(Liuet al\.,[2025a](https://arxiv.org/html/2607.22578#bib.bib31)\)handles volatility using Probabilistic Demand Graphs, and Helix\(Meiet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib14)\)optimizes cluster scheduling via max\-flow formulation\. However, under dynamic workloads, these methods are often limited by modeling overhead or bias and lack fairness for long\-tail tasks\. Thus, a lightweight, feedback\-driven mechanism is needed to reduce tail latency and ensure fairness\.

## 3Method

### 3\.1Problem Formulation

We formulate the multi\-tenant query scheduling and resource allocation problem as an online optimization task\. At any timett, the system receives and processes a set of arriving queries𝒬t\\mathcal\{Q\}\_\{t\}\.

Workload Model\.Each queryq∈𝒬tq\\in\\mathcal\{Q\}\_\{t\}is modeled as a DAGGq=\(𝒱q,ℰq\)G\_\{q\}=\(\\mathcal\{V\}\_\{q\},\\mathcal\{E\}\_\{q\}\), where𝒱q\\mathcal\{V\}\_\{q\}is the set of task nodes andℰq\\mathcal\{E\}\_\{q\}represents their sequential dependencies\. For each nodev∈𝒱qv\\in\\mathcal\{V\}\_\{q\}, we define the following attributes:wvw\_\{v\}represents the workload estimation \(specifically, input text size or FLOPs for non\-LLM nodes, and expected token count for LLM nodes\);pvp\_\{v\}specifies the set of executable device types, e\.g\., Embedding or LLM instances;dvd\_\{v\}denotes the reverse topological depth, defined as the distance from the node to the sink; andsvs\_\{v\}indicates the splittability upper bound, allowing the node to be parallelly split intokksub\-nodes\.

Resource Model and Decision Variables\.Letℛ\\mathcal\{R\}denote the set of available system resources\. Each resourcer∈ℛr\\in\\mathcal\{R\}is characterized by its typeτ​\(r\)\\tau\(r\), parallelism capacityc​a​prcap\_\{r\}, and batching limitBrB\_\{r\}\. At each decision step, the system determines four variables:xv,r∈\{0,1\}x\_\{v,r\}\\in\\\{0,1\\\}indicates whether to schedule nodevvon resourcerr;bv,r∈ℕ\+b\_\{v,r\}\\in\\mathbb\{N\}^\{\+\}represents the batch size;av,ra\_\{v,r\}denotes the resource share allocation; andkv∈\[1,sv\]k\_\{v\}\\in\[1,s\_\{v\}\]is the pipeline decomposition quantity\. As illustrated in Figure[3\(b\)](https://arxiv.org/html/2607.22578#S1.F3.sf2),kvk\_\{v\}allows the system to decompose linear task chains \(e\.g\., Embedding→\\toSearch→\\toRerank\) intokkparallel sub\-pipelines along the data dimension to exploit fine\-grained parallelism\.

Optimization Objective\.Our objective is to balance average latency with fairness for long queries\. We define the global objective function as:

min𝒥=λ⋅1\|𝒬t\|​∑q∈𝒬tCq\+\(1−λ\)⋅maxq∈𝒬t⁡Cq,\\displaystyle\\min\\quad\\mathcal\{J\}=\\ \\lambda\\cdot\{\\frac\{1\}\{\|\\mathcal\{Q\}\_\{t\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\}\}\+\(1\-\\lambda\)\\cdot\{\\max\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\}\},\(1\)whereCqC\_\{q\}denotes the end\-to\-end completion time of queryqq, andλ∈\[0,1\]\\lambda\\in\[0,1\]is a coefficient that balances the average and maximum latency\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x9.png)Figure 4:System overview of HeraSys\.
### 3\.2System Overview

As illustrated in Figure[4](https://arxiv.org/html/2607.22578#S3.F4), HeraSys comprises four primary components\.

Declarative Frontend and Profiler\.This component parses user\-defined DAG workflows into fine\-grained task nodes \(𝒱q\\mathcal\{V\}\_\{q\}\)\. It uses a profiling library to estimate the workloadwvw\_\{v\}and remaining timeSqS\_\{q\}to guide scheduling\.

Pre\-Execution Graph Fuser \(§[3\.3](https://arxiv.org/html/2607.22578#S3.SS3)\)\.This module identifies redundant subgraphs across queries and eliminates duplicate computation through node fusion\. It also raises the priority of shared nodes to accelerate query execution\.

Runtime Scheduler \(§[3\.4](https://arxiv.org/html/2607.22578#S3.SS4)\)\.The scheduler dynamically prioritizes tasks using a joint priority functionπv\\pi\_\{v\}and resource skewing to prevent starvation\. It adaptively adjusts the batch sizebv,rb\_\{v,r\}and pipeline decompositionkvk\_\{v\}based on system load\.

Backend Engine Pool\.This layer manages heterogeneous resources \(e\.g\., LLMs, Embedding models\) to execute dispatched tasks\. It returns results to the scheduler to trigger subsequent dependencies\.

### 3\.3Pre\-Execution Optimization: Semantic Graph Fusion

Node Semantic Signature\.To identify redundant computations, we use the semantic signature as a unique identifier to determine node equivalence\. For any nodevv, its signature is a hash summary derived from its operator semantics and input context:

S​i​g​\(v\)=ℋ​\(τv,Θp,𝒟i​n,⨁u∈Pre​\(v\)S​i​g​\(u\)\),\\begin\{matrix\}Sig\(v\)=\\mathcal\{H\}\\left\(\\tau\_\{v\},\\Theta\_\{p\},\\mathcal\{D\}\_\{in\},\\bigoplus\_\{u\\in\\text\{Pre\}\(v\)\}Sig\(u\)\\right\),\\end\{matrix\}\(2\)whereτv\\tau\_\{v\}represents the operator type;Θp\\Theta\_\{p\}contains parameters affecting the result \(e\.g\., Top\-K for retrieval\);𝒟i​n\\mathcal\{D\}\_\{in\}summarizes the input data that determine the output of the node; and⨁\\bigoplusrepresents the ordered combination of predecessor signatures\. For example, in retrieval or embedding nodes,𝒟i​n\\mathcal\{D\}\_\{in\}is derived from the normalized query text or the document chunk content involved in that node\. This addressing mechanism allows the system to map logically independent but computationally equivalent tasks to identical signatures across different queries\.

Dynamic Fusion and Reuse Mechanism\.The system implements two complementary deduplication strategies based on node arrival time\.

\(I\) Topological Fusion within Time Window\. For concurrent queries submitted within a short window, the system identifies redundancy via signature matching\. When a nodevnv\_\{n\}in a new query matches a nodevev\_\{e\}in the ready queue, the fuser removesvnv\_\{n\}from the execution graph and cancels its resource allocation\. It then performs DAG edge redirection, modifying the topology to connect the downstream dependencies ofvnv\_\{n\}to the existing nodevev\_\{e\}\. This physically fuses multiple logical queries to share a single execution instance\. Upon completion, the output ofvev\_\{e\}flows to the downstream nodes of all related queries\.

\(II\) Asynchronous Cache Reuse\. For requests outside the time window, we employ asynchronous reuse\. If a new nodevnv\_\{n\}matches a completed nodevhv\_\{h\}, the system directly retrieves the result, skipping scheduling and execution\. Ifvnv\_\{n\}matches a nodevrv\_\{r\}that is currently executing,vnv\_\{n\}is not submitted to the engine but waits forvrv\_\{r\}to finish\. Oncevrv\_\{r\}completes, the system shares the result withvnv\_\{n\}\. This mechanism operates on the fine\-grained workflow graph\. For LLM operators, shared prefixes can be represented as decomposed prefill nodes, so matching such nodes enables HeraSys to redirect downstream dependencies or reuse completed results\. Beyond LLM prefix reuse, the same graph\-level mechanism also covers retrieval, reranking, tool execution, and other intermediate subgraphs\.

Reuse\-Driven Priority Adjustment\.Graph fusion not only alters the graph structure but also influences scheduling\. A highly reused node is not treated as a standard single task, but as a critical node enabling multiple downstream paths\. To reflect this, we introduce a reuse gain to adjust the scheduling priority\. For a fused nodev∗v^\{\*\}, its priority contributionϕv∗\\phi\_\{v^\{\*\}\}is defined as the sum of the contributions of all merged logical nodes \(the definition ofϕv\\phi\_\{v\}is detailed in §[3\.4](https://arxiv.org/html/2607.22578#S3.SS4)\):

ϕv∗=∑v∈ℳ​\(v∗\)ϕv\.\\begin\{matrix\}\\phi\_\{v^\{\*\}\}=\\sum\_\{v\\in\\mathcal\{M\}\(v^\{\*\}\)\}\\phi\_\{v\}\.\\end\{matrix\}\(3\)This ensures that nodes shared by multiple queries receive higher priority\. By prioritizing these common sub\-tasks, the system accelerates the progress of multiple concurrent workflows simultaneously\.

### 3\.4Runtime Optimization: Load\-Aware Joint Scheduling

Query Feature Extraction\.The system uses two key runtime metrics to guide scheduling decisions\.

\(I\) Dynamic Remaining Time Estimation \(S^q\\widehat\{S\}\_\{q\}\)\. We combine offline profiling with execution progress to predict remaining time\. To enable fine\-grained preemption and accurate tracking, we decompose long decoding operators into sequential sub\-nodes \(Figure[3\(b\)](https://arxiv.org/html/2607.22578#S1.F3.sf2)\)\. For a queryqq,S^q\\widehat\{S\}\_\{q\}is the sum of expected execution times for all nodes in the current unfinished subgraph\. To account for LLM inference variability, we introduce a runtime correction factorℱ\\mathcal\{F\}\. If actual execution exceeds expectations,ℱ\\mathcal\{F\}increases the estimate\(Luoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib27)\)\. The estimate at timettis:

S^q​\(t\)=ℱ​\(Aq​\(t\)∑v∈𝒱d​o​n​eq​\(t\)T¯v\+ϵT\)⋅∑v∈𝒱r​e​m​a​i​nq​\(t\)T¯v\\widehat\{S\}\_\{q\}\(t\)=\\mathcal\{F\}\\left\(\\frac\{A\_\{q\}\(t\)\}\{\\sum\_\{v\\in\\mathcal\{V\}\_\{done\}^\{q\}\(t\)\}\\bar\{T\}\_\{v\}\+\\epsilon\_\{T\}\}\\right\)\\cdot\\sum\_\{v\\in\\mathcal\{V\}\_\{remain\}^\{q\}\(t\)\}\\bar\{T\}\_\{v\}\(4\)whereAq​\(t\)A\_\{q\}\(t\)is the attained service time \(accumulated execution time\) for queryqq, andT¯v\\bar\{T\}\_\{v\}denotes the predicted runtime of nodevv\. SinceS^q\\widehat\{S\}\_\{q\}is aggregated from node\-level predictions, stages such as prefill, decoding, embedding, and retrieval contribute according to their predicted runtimes, whileℱ\\mathcal\{F\}updates the estimate based on online execution progress\. This metric allows the scheduler to identify short tasks nearing completion, prioritizing them to reduce average latency\.

\(II\) Node Contribution \(ϕv\\phi\_\{v\}\)\. We introduceϕv\\phi\_\{v\}to quantify the contribution of nodevvto query progress\. This metric considers three topological features: downstream remaining workload, path length, and the number of parallel branches enabled\. The node contribution is a weighted sum of these features:

ϕv=∑u∈𝒩v\(ω1⋅wu\+ω2⋅du\)\+ω3⋅\|𝒩v\|,\\phi\_\{v\}=\\sum\_\{u\\in\\mathcal\{N\}\_\{v\}\}\\left\(\\omega\_\{1\}\\cdot\{w\_\{u\}\}\+\\omega\_\{2\}\\cdot\{d\_\{u\}\}\\right\)\+\\omega\_\{3\}\\cdot\|\\mathcal\{N\}\_\{v\}\|,\(5\)whereNv\{N\_\{v\}\}is the set of successor nodes\. Nodes with highϕv\\phi\_\{v\}are prioritized to accelerate the progress of the current task stream\.

Dynamic Partitioning and Resource Skewing\.To prevent long\-task starvation, the system adopts a partitioning method based on load distribution\.

\(I\) Long/Short Query Partitioning\. The scheduler monitors the distribution ofS^q\\widehat\{S\}\_\{q\}for active queries, calculating the meanμs\\mu\_\{s\}and standard deviationσs\\sigma\_\{s\}\. We define a dynamic thresholdτl​o​a​d=μs\+η⋅σs\\tau\_\{load\}=\\mu\_\{s\}\+\\eta\\cdot\\sigma\_\{s\}\. IfS^q\>τl​o​a​d\\widehat\{S\}\_\{q\}\>\\tau\_\{load\}, the query is marked as a long query\. Its computationally intensive nodes are marked as heavy nodes \(yv=1y\_\{v\}=1\), while others remain normal \(yv=0y\_\{v\}=0\)\.

\(II\) Resource Skewing\. To address starvation, the scheduler implements resource skewing\. The scheduler maintains separate candidate queues for short and long queries on each resource type\. It enforces a minimum resource reservation, ensuring that at least a specific proportion of resources is allocated to heavy nodes of long queries:

∑v:yv=1av,r≥ρ⋅c​a​pr\.\\begin\{matrix\}\\sum\_\{v:y\_\{v\}=1\}a\_\{v,r\}\\geq\\rho\\cdot cap\_\{r\}\.\\end\{matrix\}\(6\)This reservation isolates resource competition between tasks with different characteristics\. It maintains low latency for short queries while guaranteeing progress for long tasks, preventing unfairness and tail latency degradation\.

Joint Priority Scheduling\.Combining these metrics, the scheduler uses a joint priority algorithm to order nodes in the ready queue\. The algorithm balances latency, structural efficiency, and fairness via a composite priority functionπv\\pi\_\{v\}:

πv=αS^q​\(v\)\+ϵ\+γ⋅ϕv\+δ⋅yv,\\pi\_\{v\}=\{\\frac\{\\alpha\}\{\\widehat\{S\}\_\{q\(v\)\}\+\\epsilon\}\}\+\{\\gamma\\cdot\\phi\_\{v\}\}\+\{\\delta\\cdot y\_\{v\}\},\(7\)where the first term implements dynamic Shortest Remaining Time First \(SRTF\) to reduce average latency\(Schrage,[1968](https://arxiv.org/html/2607.22578#bib.bib1)\)\. The second term incorporates topological information, prioritizing critical nodes that enable downstream work to improve efficiency\. The third term prevents starvation by boosting the priority of heavy nodes in long tasks, working alongside resource skewing\.

Adaptive Execution Adjustment\.The system employs a load\-adaptive mechanism to optimize performance under varying pressure\. We address the global objective function:

min𝒥t=λt⋅1\|𝒬t\|​∑q∈𝒬tCq\+\(1−λt\)⋅maxq∈𝒬t⁡Cq,\\min\\quad\\mathcal\{J\}\_\{t\}=\\lambda\_\{t\}\\cdot\\frac\{1\}\{\|\\mathcal\{Q\}\_\{t\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\}\+\(1\-\\lambda\_\{t\}\)\\cdot\\max\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\},\(8\)whereλt∈\[0,1\]\\lambda\_\{t\}\\in\[0,1\]is dynamically determined by system load\. The scheduler switches between two states:

\(I\) In high\-load intervals \(λt→1\\lambda\_\{t\}\\to 1\), the scheduler increases the batching windowbv,rb\_\{v,r\}to aggregate larger batches for higher throughput\. Simultaneously, it reduces pipeline splitskvk\_\{v\}to minimize scheduling overhead\.

\(II\) In low\-load intervals \(λt→0\\lambda\_\{t\}\\to 0\), the scheduler reduces the batching window to execute tasks immediately\. It also increaseskvk\_\{v\}, decomposing nodes into micro\-batches to exploit data parallelism and reduce end\-to\-end latency\.

Table 1:End\-to\-end tail latency performance \(P95P\_\{95\}andP99P\_\{99\}\) across different workflows and concurrency levels\. The Speedup row lists the improvement of HeraSys in each scenario\.![Refer to caption](https://arxiv.org/html/2607.22578v1/x10.png)Figure 5:End\-to\-end average latency comparison across different workloads under varying concurrency levels\.

## 4Experiments

### 4\.1Experimental Setup

Testbed\.We implement the HeraSys prototype on Ray\(Moritzet al\.,[2018](https://arxiv.org/html/2607.22578#bib.bib40)\)\. For the Graph Fuser, we identify structural relationships using reverse traversal and compute node semantic hashes via SHA\-256\(NIST and Dang,[2015](https://arxiv.org/html/2607.22578#bib.bib37)\)\. For the Runtime Scheduler, we use Ray metrics to monitor system load, manage pending tasks using priority queues, and implement global scheduling via Ray’s Actor model\. We run experiments on a server node with 4 NVIDIA GeForce RTX 4090 GPUs\. To ensure consistency across backends, we use vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.22578#bib.bib16)\)for all systems\. We use Llama\-3\-8B\(Meta LLaMA Team,[2024](https://arxiv.org/html/2607.22578#bib.bib41)\)as the core LLM, bge\-large\-en\-v1\.5\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.22578#bib.bib39)\)for embedding \(stored in PostgreSQL\(Momjian and PostgreSQL Global Development Group,[2025](https://arxiv.org/html/2607.22578#bib.bib42)\)with pgvector\(Kane,[2025](https://arxiv.org/html/2607.22578#bib.bib46)\)\), and bge\-reranker\-large for reranking\. For external retrieval, we integrate the Google Search API\.

Baselines\.To evaluate HeraSys, we compare it against three representative systems:

- •LlamaIndex\(Liu and LlamaIndex Team,[2025](https://arxiv.org/html/2607.22578#bib.bib45)\):A data framework connecting LLMs with external data, LlamaIndex optimizes indexing and retrieval pipelines\. It provides a complete toolchain from data ingestion to query routing\. At runtime, it follows predefined logic, sequentially executing steps such as retrieval, reranking, and response synthesis\.
- •LangGraph\(Chase and LangChain Team,[2025](https://arxiv.org/html/2607.22578#bib.bib44)\):A modular agent orchestration framework, LangGraph models workflows as execution graphs of LLM calls and tool executions, managing context flow via a state machine\. At runtime, LangGraph creates an independent state graph for each request and schedules execution in topological order within its isolated scope\.
- •Ayo\(Tanet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib15)\):Ayo is a state\-of\-the\-art fine\-grained orchestration system\. Ayo proposes a decoupled architecture based on “Task Primitives”, decomposing LLM applications into standardized operators\. It implements data parallelism within statically defined pipelines, supporting heterogeneous resources\.

Workloads\.We construct four workloads with distinct characteristics based on public datasets to simulate multi\-tenant traffic\.

- •Standard RAG:The system embeds user queries and documents using bge\-large\-en\-v1\.5, followed by vector retrieval in PostgreSQL\. We feed the top\-12 relevant chunks into the core LLM \(Llama\-3\-8B\) to synthesize the answer\. The query set is sampled from the MS MARCO dataset\(Bajajet al\.,[2018](https://arxiv.org/html/2607.22578#bib.bib36)\)\.
- •RAG with Rerank:This extends the standard RAG workflow to enhance accuracy\. A reranker \(bge\-reranker\-large\) scores the initial top\-12 chunks\. The system selects the top\-8 chunks based on these scores for generation\.
- •Web Search Agent:This simulates a multi\-step agentic task\. The LLM rewrites the user intent into a search query, invokes the Google Search API, and synthesizes results to generate a response\. Test queries are sampled from TruthfulQA\(Linet al\.,[2022](https://arxiv.org/html/2607.22578#bib.bib38)\)\.
- •Mixed Workload:This consists of 75% RAG tasks \(mixed standard and rerank\) and 25% Web Search tasks to simulate a heterogeneous scenario where tasks with different patterns coexist\.

To mimic production traffic, we submit requests using a multi\-threaded client to simulate concurrent access\. We vary concurrency across 6, 30, 100, and 300 to cover light to saturated loads\. We inject 5%–15% duplicate queries targeting hot documents following a Zipfian distribution\(Breslauet al\.,[1999](https://arxiv.org/html/2607.22578#bib.bib35)\)to evaluate fusion\. For resource allocation, we deploy two vLLM instances on two GPUs, and run 3 Embedding and 3 Reranking instances on the remaining GPUs to reduce bottlenecks and maximize performance\. In the main experiments, HeraSys uses a fixed scheduler configuration, withα=1\.0\\alpha=1\.0,γ=0\.3\\gamma=0\.3, andδ=0\.5\\delta=0\.5for the joint priority function in Equation[7](https://arxiv.org/html/2607.22578#S3.E7), andρ=0\.2\\rho=0\.2for resource skewing in Equation[6](https://arxiv.org/html/2607.22578#S3.E6)\. Finally, we optimize configurations for all baselines, including batch size tuning, to ensure fair comparison under latency constraints\.

### 4\.2End\-to\-End Latency Performance and Fairness

We first evaluate the end\-to\-end latency performance of HeraSys compared to baseline systems under different workloads and concurrency levels\. As shown in Table[1](https://arxiv.org/html/2607.22578#S3.T1)and Figure[5](https://arxiv.org/html/2607.22578#S3.F5), HeraSys outperforms all baselines across all metrics\.

Average Latency\.Figure[5](https://arxiv.org/html/2607.22578#S3.F5)shows that HeraSys reduces latency by 35\.2% and 30\.7% in the two RAG workflows \(w/ and w/o Rerank\), respectively, compared to the baselines\. This is due to the SRTF strategy in HeraSys, which reduces average latency\. In contrast, other baselines ignore real\-time task duration, even though Ayo optimizes data pipelines through fine\-grained task primitives\. Notably, HeraSys achieves larger gains in the Web Search workflow, reducing average latency by 41\.8%\. Since this scenario calls external search APIs, other baselines suffer from blocking waits due to network I/O and API limits\. HeraSys uses the Graph Fusion mechanism to fuse and reuse redundant queries, thereby reducing external calls and waiting times\. This enables subsequent tasks to proceed quickly and optimizes end\-to\-end latency\.

Tail Latency and Fairness\.The advantages of HeraSys in reducing tail latency and enhancing fairness for long queries are shown in the P95/P99 columns of Table[1](https://arxiv.org/html/2607.22578#S3.T1)\. Due to the lack of application\-level information, LangGraph and LlamaIndex experience severe HOL Blocking once they encounter long document processing or generation tasks, causing P95 and P99 latency to increase sharply under high concurrency\. Even Ayo, which supports fine\-grained parallelism, maintains high tail latency under high load due to the lack of guarantees for long\-tail tasks, making it difficult to meet strict SLO requirements\. In contrast, HeraSys reduces P95 latency by 48\.5% and P99 latency by 53\.9%\. This confirms the effectiveness of our Resource Skewing mechanism, which successfully avoids the starvation issues common in pure SRTF strategies by enforcing a minimum resource reservation for long\-tail tasks\. Consequently, HeraSys provides a fairer and more predictable quality of service while ensuring the response speed of short tasks\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x11.png)Figure 6:Evaluation of service throughput under different concurrency settings and workloads\.
### 4\.3Throughput Scalability

We further measure the service throughput \(QPS\) of each system under latency constraints, as shown in Figure[6](https://arxiv.org/html/2607.22578#S4.F6)\. As concurrent pressure increases, HeraSys shows higher throughput scalability, achieving a throughput up to 1\.50 times that of Ayo, and nearly 2 times that of traditional modular frameworks \(LangGraph and LlamaIndex\)\. This significant performance gap highlights the limitations of baseline systems: both LangGraph and Ayo essentially treat each request as an independent computation graph\. Even when multiple users simultaneously query hot documents, these systems repeatedly execute the same Embedding and retrieval operators, resulting in wasted resources\. Conversely, HeraSys’s throughput gains result from the deduplication capabilities of the Graph Fuser\. As concurrency increases, the probability of sharing subgraphs and nodes between requests increases, allowing HeraSys to identify and merge these redundant computations\. Simultaneously, HeraSys’s load\-adaptive regulation maintains higher batch sizes and reduced pipeline decomposition granularity under high load, thereby reducing scheduling overhead\. This allows the system to convert increased load pressure into efficient deduplication and adaptive regulation, supporting higher service throughput than baselines on the same physical resources\.

![Refer to caption](https://arxiv.org/html/2607.22578v1/x12.png)Figure 7:Ablation study of HeraSys core components under the mixed workload with 100 concurrency\. Note that “w/ FCFS” and “w/ SRTF” represent the system variants using FCFS and SRTF scheduling strategies\.Table 2:Variations in the Average Number of Nodes and Latency with Adaptive Pipeline Decomposition on RAG w/ Rerank\. Note that Ayo and “HeraSys \(w/o\)” employ a fixed decomposition strategy, statically splitting the workflow into two parallel pipelines, whereas HeraSys adjusts the pipeline granularity based on load\.
### 4\.4Mechanism Contribution and Ablation

To quantify the contribution of each core mechanism of HeraSys, we conduct an ablation study, as shown in Figure[7](https://arxiv.org/html/2607.22578#S4.F7)and Table[2](https://arxiv.org/html/2607.22578#S4.T2)\.

Graph Fusion and Scheduling Strategy\.Under the mixed workload with high concurrency \(Concurrency 100\), we conduct tests on the Graph Fuser and Runtime Scheduler to assess their impacts on system performance\. Figure[7](https://arxiv.org/html/2607.22578#S4.F7)shows that disabling the Graph Fuser \(w/o Fusion\) limits system capacity; the total throughput of Full HeraSys is 1\.16×\\timesthat without fusion\. This confirms that scheduling optimization alone is insufficient, and redundancy removal is crucial for improving throughput\. In terms of latency, reverting the Load\-Aware Scheduler to a standard FCFS strategy increases P99 latency by 12\.5%\. While using only the SRTF strategy reduces P50 \(by approximately 13%\), it increases P99 latency by 22\.3% compared to Full HeraSys\. This highlights the limitations of single strategies in complex workflows and the value of HeraSys’s joint optimization, which balances average speed and long\-tail fairness\.

Adaptive Pipeline Decomposition\.We perform comparative experiments on RAG w/ Rerank workflows under varying load conditions to further verify the effectiveness of adaptive pipeline decomposition\. Table[2](https://arxiv.org/html/2607.22578#S4.T2)shows how adaptive pipeline decomposition dynamically adjusts graph granularity based on load\. Under low load \(the first section of Table[2](https://arxiv.org/html/2607.22578#S4.T2)\), HeraSys increases the average number of nodes per query from 19\.0 to 24\.7 \(\+30%\), performing finer\-grained splitting to utilize idle resources for parallel computation, thereby reducing latency\. Conversely, under high load \(the second section of Table[2](https://arxiv.org/html/2607.22578#S4.T2)\), HeraSys reduces the average node count to 15\.6 \(\-18%\) to reduce overhead\. HeraSys does not rely on static configuration but balances parallelism and overhead, ensuring high performance across varying loads\.

Table 3:Parameter sensitivity under the mixed workload with 100 concurrency\.
### 4\.5Parameter Sensitivity

We evaluate the sensitivity of HeraSys to scheduler parameters under the mixed workload with 100 concurrency\. We keep the system setting unchanged and vary only the parameters in the joint priority function and resource skewing\. As shown in Table[3](https://arxiv.org/html/2607.22578#S4.T3), different coarse configurations lead to different performance tendencies: “Short\-first” shifts the system toward lower median latency, “Throughput” shifts it toward higher average efficiency and QPS, and “Tail\-fairness” shifts it toward lower high\-percentile latency\. This reflects the role of these parameters in balancing short\-query responsiveness, structural efficiency, and long\-query protection in mixed serving scenarios\.

## 5Conclusion

In this paper, we presented HeraSys, a fine\-grained end\-to\-end serving system designed for multi\-workflow scenarios, enabling cross\-workflow optimization\. Building upon this architecture, we designed a dual\-layer optimization mechanism: prior to execution, the system identified and consolidated cross\-query computational redundancy, reducing system overhead; at runtime, HeraSys introduced a joint scheduling strategy that dynamically evaluated task characteristics and system load to adjust execution order and data granularity, effectively addressing resource contention in heterogeneous workloads\. We evaluated HeraSys through extensive experiments across diverse scenarios, which validated the effectiveness of the proposed architecture\. We believe that the cross\-workflow optimization approach in HeraSys provides insights for building efficient and scalable LLM application serving systems\.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of China \(NSFC\) under Grant 62302048, Grant U25A20436, and Grant 62272050; in part by Guangdong Higher Education Association under Grant 24GQN97; in part by the Guangdong Provincial Higher Education Institutions under Grant 2024KTSCX219; and in part by Beijing Normal University at Zhuhai Education Reform Project under Grant jx2025037\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning systems by optimizing the serving efficiency of Large Language Models\. This work contributes to reducing computational resource consumption and operational costs, thereby fostering the development of greener and more efficient LLM serving\. Additionally, it significantly enhances the user experience of LLM applications by ensuring faster and fairer responses\. We do not foresee any direct negative societal consequences, as this research focuses on the underlying serving infrastructure rather than generative content capabilities\.

## References

- R\. Abhyankar, Z\. He, V\. Srivatsa, H\. Zhang, and Y\. Zhang \(2024\)InferCept: efficient intercept support for augmented large language model inference\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p3.1)\.
- A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. Gulavani, A\. Tumanov, and R\. Ramjee \(2024\)Taming Throughput\-Latency tradeoff in LLM inference with Sarathi\-Serve\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 117–134\.Cited by:[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- P\. Bajaj, D\. Campos, N\. Craswell, L\. Deng, J\. Gao, X\. Liu, R\. Majumder, A\. McNamara, B\. Mitra, T\. Nguyen, M\. Rosenberg, X\. Song, A\. Stoica, S\. Tiwary, and T\. Wang \(2018\)MS marco: a human generated machine reading comprehension dataset\.arXiv preprint arXiv:1611\.09268\.Cited by:[1st item](https://arxiv.org/html/2607.22578#S4.I2.i1.p1.1)\.
- F\. Bang \(2023\)GPTCache: an open\-source semantic cache for LLM applications enabling faster answers and cost savings\.InProceedings of the 3rd Workshop for Natural Language Processing Open Source Software \(NLP\-OSS 2023\),pp\. 212–218\.Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- L\. Breslau, P\. Cao, L\. Fan, G\. Phillips, and S\. Shenker \(1999\)Web caching and zipf\-like distributions: evidence and implications\.InINFOCOM,pp\. 126–134\.Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p4.4)\.
- H\. Chase and LangChain Team \(2024\)LangChain\.Note:[https://github\.com/langchain\-ai/langchain](https://github.com/langchain-ai/langchain)Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- H\. Chase and LangChain Team \(2025\)LangGraph\.Note:[https://python\.langchain\.com/docs/langgraph/](https://python.langchain.com/docs/langgraph/)Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§1](https://arxiv.org/html/2607.22578#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1),[2nd item](https://arxiv.org/html/2607.22578#S4.I1.i2.p1.1.1)\.
- I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. Zhong \(2024\)Prompt cache: modular attention reuse for low\-latency inference\.InMLSys,Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber \(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- A\. Kane \(2025\)Pgvector\.Note:[https://github\.com/pgvector/pgvector](https://github.com/pgvector/pgvector)Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. V\. A, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2024\)DSPy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InSOSP,pp\. 611–626\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.arXiv preprint arXiv:2211\.17192\.Cited by:[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- C\. Lin, Z\. Han, C\. Zhang, Y\. Yang, F\. Yang, C\. Chen, and L\. Qiu \(2024\)Parrot: efficient serving of LLM\-based applications with semantic variable\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 929–945\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§1](https://arxiv.org/html/2607.22578#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InACL \(1\),pp\. 3214–3252\.Cited by:[3rd item](https://arxiv.org/html/2607.22578#S4.I2.i3.p1.1)\.
- J\. Liu and LlamaIndex Team \(2025\)LlamaIndex\.Note:[https://github\.com/run\-llama/llama\_index](https://github.com/run-llama/llama_index)Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2607.22578#S4.I1.i1.p1.1.1)\.
- Y\. Liu, Z\. Gan, Z\. Gan, W\. Wang, C\. Chen, Y\. Shan, X\. Chen, Z\. Han, Y\. Zhu, S\. Sun, and M\. Guo \(2025a\)Efficient serving of llm applications with probabilistic demand modeling\.arXiv preprint arXiv:2506\.14851\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- Y\. Liu, Y\. Cheng, J\. Yao, Y\. An, X\. Chen, S\. Feng, Y\. Huang, S\. Shen, R\. Zhang, K\. Du, and J\. Jiang \(2025b\)LMCache: an efficient kv cache layer for enterprise\-scale llm inference\.arXiv preprint arXiv:2510\.09665\.Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- Y\. Liu, H\. Li, Y\. Cheng, S\. Ray, Y\. Huang, Q\. Zhang, K\. Du, J\. Yao, S\. Lu, G\. Ananthanarayanan, M\. Maire, H\. Hoffmann, A\. Holtzman, and J\. Jiang \(2024\)CacheGen: kv cache compression and streaming for fast large language model serving\.InProceedings of the ACM SIGCOMM 2024 Conference,pp\. 38–56\.Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- M\. Luo, X\. Shi, C\. Cai, T\. Zhang, J\. Wong, Y\. Wang, C\. Wang, Y\. Huang, Z\. Chen, J\. E\. Gonzalez, and I\. Stoica \(2025\)Autellix: an efficient serving engine for llm agents as general programs\.arXiv preprint arXiv:2502\.13965\.Cited by:[§A\.1\.4](https://arxiv.org/html/2607.22578#A1.SS1.SSS4.p2.16),[§1](https://arxiv.org/html/2607.22578#S1.p2.1),[§1](https://arxiv.org/html/2607.22578#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2607.22578#S3.SS4.p2.6)\.
- Y\. Mei, Y\. Zhuang, X\. Miao, J\. Yang, Z\. Jia, and R\. Vinayak \(2025\)Helix: serving large language models over heterogeneous gpus and network via max\-flow\.InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1,pp\. 586–602\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- Meta LLaMA Team \(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- B\. Momjian and PostgreSQL Global Development Group \(2025\)PostgreSQL\.Note:[https://www\.postgresql\.org/](https://www.postgresql.org/)Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- P\. Moritz, R\. Nishihara, S\. Wang, A\. Tumanov, R\. Liaw, E\. Liang, M\. Elibol, Z\. Yang, W\. Paul, M\. I\. Jordan, and I\. Stoica \(2018\)Ray: a distributed framework for emerging AI applications\.In13th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 18\),pp\. 561–577\.Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- NIST and Q\. Dang \(2015\)Secure hash standard\.Federal Inf\. Process\. Stds\. \(NIST FIPS\), National Institute of Standards and Technology, Gaithersburg, MD\.Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. Bianchini \(2024\)Splitwise: efficient generative llm inference using phase splitting\.InISCA,pp\. 118–132\.Cited by:[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong, J\. Xu, D\. Li, Z\. Liu, and M\. Sun \(2024\)ChatDev: communicative agents for software development\.InACL \(1\),pp\. 15174–15186\.Cited by:[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- R\. Qin, Z\. Li, W\. He, J\. Cui, F\. Ren, M\. Zhang, Y\. Wu, W\. Zheng, and X\. Xu \(2025\)Mooncake: trading more storage for less computation — a KVCache\-centric architecture for serving LLM chatbot\.In23rd USENIX Conference on File and Storage Technologies \(FAST 25\),pp\. 155–170\.Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- L\. Schrage \(1968\)Letter to the editor—a proof of the optimality of the shortest remaining processing time discipline\.Oper\. Res\.16\(3\),pp\. 687–690\.Cited by:[1st item](https://arxiv.org/html/2607.22578#A2.I2.i1.p1.4),[§3\.4](https://arxiv.org/html/2607.22578#S3.SS4.p7.2)\.
- J\. Shen, N\. Wadlom, and Y\. Lu \(2026\)Batch query processing and optimization for agentic workflows\.arXiv preprint arXiv:2509\.02121\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1)\.
- J\. Shen, N\. Wadlom, L\. Zhou, D\. Wang, X\. Miao, L\. Fang, and Y\. Lu \(2025\)FlowMesh: a service fabric for composable llm workflows\.arXiv preprint arXiv:2510\.26913\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p2.1),[§1](https://arxiv.org/html/2607.22578#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- X\. Tan, Y\. Jiang, Y\. Yang, and H\. Xu \(2025\)Towards end\-to\-end optimization of llm\-based applications with ayo\.Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1),[3rd item](https://arxiv.org/html/2607.22578#S4.I1.i3.p1.1.1)\.
- B\. Wu, Y\. Zhong, Z\. Zhang, S\. Liu, F\. Liu, Y\. Sun, G\. Huang, X\. Liu, and X\. Jin \(2024a\)Fast distributed inference serving for large language models\.arXiv preprint arXiv:2305\.05920\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2024b\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversations\.InFirst Conference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22578#S2.SS1.p1.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie \(2024\)C\-pack: packed resources for general chinese embeddings\.InSIGIR,pp\. 641–649\.Cited by:[§4\.1](https://arxiv.org/html/2607.22578#S4.SS1.p1.1)\.
- J\. Yao, H\. Li, Y\. Liu, S\. Ray, Y\. Cheng, Q\. Zhang, K\. Du, S\. Lu, and J\. Jiang \(2025\)CacheBlend: fast large language model serving for rag with cached knowledge fusion\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 94–109\.Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun \(2022\)Orca: a distributed serving system for Transformer\-Based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),pp\. 521–538\.Cited by:[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. W\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2607.22578#S2.SS2.p1.1)\.
- Y\. Zhong, S\. Liu, J\. Chen, J\. Hu, Y\. Zhu, X\. Liu, X\. Jin, and H\. Zhang \(2024\)DistServe: disaggregating prefill and decoding for goodput\-optimized large language model serving\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 193–210\.Cited by:[§1](https://arxiv.org/html/2607.22578#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.22578#S2.SS3.p1.1)\.

## Appendix AProblem Modeling and Solving Strategy

This appendix provides the mathematical modeling, constraint definitions, and design of the online solving strategy for HeraSys\.

### A\.1Problem Formulation

#### A\.1\.1Workload and Graph Model

Let𝒬t\\mathcal\{Q\}\_\{t\}be the set of active queries at timett\(i\.e\., queries that have arrived byttand are not yet completed\)\. Each queryq∈𝒬tq\\in\\mathcal\{Q\}\_\{t\}is represented by a Directed Acyclic Graph \(DAG\)Gq=\(𝒱q,ℰq\)G\_\{q\}=\(\\mathcal\{V\}\_\{q\},\\mathcal\{E\}\_\{q\}\), where nodes represent operators or sub\-tasks, and edges represent execution dependencies\.

Each nodev∈𝒱qv\\in\\mathcal\{V\}\_\{q\}has the following attributes:

1. 1\.Workload \(wvw\_\{v\}\):Estimated computation volume\. For non\-LLM nodes, this represents input text size or FLOPs; for LLM nodes, it represents the expected number of tokens\.
2. 2\.Compatible Resources \(pvp\_\{v\}\):A set of executable engine/device types \(e\.g\., Embedding instances, LLM instances\)\.
3. 3\.Reverse Topological Depth \(dvd\_\{v\}\):The number of layers from the node to the sink node\. Smaller values indicate proximity to the final output\.
4. 4\.Splittability \(svs\_\{v\}\):A flag and upper bound indicating if the node allows parallel splitting into up tosvs\_\{v\}execution units\. If nodevvis split with factorkv\>1k\_\{v\}\>1, it is executed as\{vi\}i=1kv\\\{v\_\{i\}\\\}\_\{i=1\}^\{k\_\{v\}\}in parallel, and the logical node completes when all units complete:Cv=maxi⁡Cv,iC\_\{v\}=\\max\_\{i\}C\_\{v,i\}\.
5. 5\.Performance Profile \(θv\\theta\_\{v\}\):Labels for operator type, overhead, and parallel efficiency, used for throughput function calibration\.
6. 6\.Overhead \(o​v​e​r​h​e​a​dvoverhead\_\{v\}\):Fixed scheduling and synchronization overhead \(e\.g\., kernel launch, split/merge, communication\), used in the runtime model\.
7. 7\.Node Contribution \(ϕv\\phi\_\{v\}\):A metric quantifying the contribution of nodevvto query progress\. It integrates three topological features: downstream remaining workload, downstream path length, and the number of enabled parallel branches\. Defined as: ϕv=∑u∈𝒩v\(ω1⋅wu\+ω2⋅du\)\+ω3⋅\|𝒩v\|,\\phi\_\{v\}=\\sum\_\{u\\in\\mathcal\{N\}\_\{v\}\}\\left\(\\omega\_\{1\}\\cdot\{w\_\{u\}\}\+\\omega\_\{2\}\\cdot\{d\_\{u\}\}\\right\)\+\\omega\_\{3\}\\cdot\|\\mathcal\{N\}\_\{v\}\|,\(9\)where𝒩v\\mathcal\{N\}\_\{v\}is the set of successor nodes ofvv\. We determine weightsω\\omegabased on dimensional normalization and prior empirical experience\. Nodes with highϕv\\phi\_\{v\}are prioritized to accelerate the current workflow\.

#### A\.1\.2Resource Model

Letℛ\\mathcal\{R\}bethe set of resource types\. Each resource typer∈ℛr\\in\\mathcal\{R\}is characterized by its parallel capacityc​a​prcap\_\{r\}\(the number of parallel engines/instances\) and batching limitBrB\_\{r\}\.

#### A\.1\.3Decision Variables

At a scheduling eventtt\(subscript omitted for brevity\), let𝒱r​e​a​d​y​\(t\)\\mathcal\{V\}\_\{ready\}\(t\)denote the set of ready nodes whose predecessors are completed\. The system determines \(forv∈𝒱r​e​a​d​y​\(t\)v\\in\\mathcal\{V\}\_\{ready\}\(t\)\):

1. 1\.xv,r∈\{0,1\}x\_\{v,r\}\\in\\\{0,1\\\}: Whether nodevvis scheduled on resource typerrat this event \(unscheduled nodes havexv,r=0x\_\{v,r\}=0for allrr\)\.
2. 2\.bv,r∈ℕ\+b\_\{v,r\}\\in\\mathbb\{N\}^\{\+\}: Thebatch sizeformed for nodevvon resourcerr\(constrained byBrB\_\{r\}\)\.
3. 3\.av,r∈\[0,1\]a\_\{v,r\}\\in\[0,1\]:Resource share allocation\(e\.g\., GPU memory/SM ratio, CPU cores\) on typerrfor nodevv\.
4. 4\.kv∈ℕ\+k\_\{v\}\\in\\mathbb\{N\}^\{\+\}:Split factor\(number of parallel execution units\) for nodevv, wherekv=1k\_\{v\}=1means no splitting\.
5. 5\.FvF\_\{v\}: Start time of nodevv\.
6. 6\.TvT\_\{v\}: Execution duration of nodevv\.
7. 7\.CvC\_\{v\}: Completion time of nodevv, whereCv=Fv\+TvC\_\{v\}=F\_\{v\}\+T\_\{v\}\.

We useq​\(v\)q\(v\)to denote the query that contains nodevv\. The completion time of queryqqisCq=maxv∈𝒱q⁡CvC\_\{q\}=\\max\_\{v\\in\\mathcal\{V\}\_\{q\}\}C\_\{v\}\. Ifkv\>1k\_\{v\}\>1, we denote theii\-th execution unit byviv\_\{i\}with auxiliary runtimeTv,iT\_\{v,i\}and completion timeCv,i=Fv\+Tv,iC\_\{v,i\}=F\_\{v\}\+T\_\{v,i\}, and the logical node completion time isCv=maxi⁡Cv,iC\_\{v\}=\\max\_\{i\}C\_\{v,i\}\.

#### A\.1\.4Auxiliary Functions

Throughput Function:μr​\(bv,r,av,r;θv\)\\mu\_\{r\}\(b\_\{v,r\},a\_\{v,r\};\\theta\_\{v\}\)denotes the throughput rate on resourcerrwith batch sizebband shareaa, calibrated by node profileθv\\theta\_\{v\}\.

Remaining Service Time Estimation:The estimated remaining time for queryqq,S^q\\widehat\{S\}\_\{q\}, is defined as:

S^q​\(t\)=ℱ​\(Aq​\(t\)∑v∈𝒱d​o​n​eq​\(t\)T¯v\+ϵT\)⋅∑v∈𝒱r​e​m​a​i​nq​\(t\)T¯v,\\widehat\{S\}\_\{q\}\(t\)=\\mathcal\{F\}\\left\(\\frac\{A\_\{q\}\(t\)\}\{\\sum\_\{v\\in\\mathcal\{V\}\_\{done\}^\{q\}\(t\)\}\\bar\{T\}\_\{v\}\+\\epsilon\_\{T\}\}\\right\)\\cdot\\sum\_\{v\\in\\mathcal\{V\}\_\{remain\}^\{q\}\(t\)\}\\bar\{T\}\_\{v\},\(10\)where𝒱d​o​n​eq​\(t\)\\mathcal\{V\}\_\{done\}^\{q\}\(t\)and𝒱r​e​m​a​i​nq​\(t\)\\mathcal\{V\}\_\{remain\}^\{q\}\(t\)denote the completed and remaining nodes of queryqqat timett,T¯v\\bar\{T\}\_\{v\}denotes the predicted runtime of nodevv, andAq​\(t\)A\_\{q\}\(t\)is the attained \(accumulated\) service time ofqqso far \(sum of measured runtimes of completed \(sub\-\)nodes\)\.ϵT\>0\\epsilon\_\{T\}\>0avoids division by zero at the beginning\.ℱ\\mathcal\{F\}is a runtime correction factor \(e\.g\., if LLM inference exceeds expectations,ℱ\\mathcal\{F\}increasesS^q\\widehat\{S\}\_\{q\}\)\.PLAS Heuristic:\(Luoet al\.,[2025](https://arxiv.org/html/2607.22578#bib.bib27)\)Since LLM decoding is unpredictable and often non\-preemptive, we decompose a decoding node into sequential sub\-nodes \(decoding\-1, decoding\-2, …\), creating preemption points\. After each sub\-node, we updateAq​\(t\)A\_\{q\}\(t\)with the observed runtime and refreshS^q​\(t\)\\widehat\{S\}\_\{q\}\(t\)\.

#### A\.1\.5Objective Function

Our objective is to balance average latency with fairness for long queries\. We define the objective function as:

min𝒥=λ⋅1\|𝒬t\|​∑q∈𝒬tCq\+\(1−λ\)⋅maxq∈𝒬t⁡Cq,\\min\\quad\\mathcal\{J\}=\\ \\lambda\\cdot\{\\frac\{1\}\{\|\\mathcal\{Q\}\_\{t\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\}\}\+\(1\-\\lambda\)\\cdot\{\\max\_\{q\\in\\mathcal\{Q\}\_\{t\}\}C\_\{q\}\},\(11\)where the first term represents average completion time \(optimizing throughput/short queries\), and the second term represents maximum completion time \(optimizing tail latency\)\. In the rolling\-horizon setting,CqC\_\{q\}represents the completion time under the current commitments plus the candidate decision at this event\.λ∈\[0,1\]\\lambda\\in\[0,1\]is an adaptive balancing coefficient\.λ\\lambdavaries based on system load, depending on the number of active queries and pending nodes\.

### A\.2Constraints

#### A\.2\.1Hard Constraints

1. 1\.Dependency Order:For each queryq∈𝒬tq\\in\\mathcal\{Q\}\_\{t\}and each nodev∈𝒱qv\\in\\mathcal\{V\}\_\{q\}, Fv≥max\{u:u→v∈ℰq\}⁡Cu\.F\_\{v\}\\geq\\max\_\{\\\{u:u\\to v\\in\\mathcal\{E\}\_\{q\}\\\}\}C\_\{u\}\.\(12\)
2. 2\.Compatibility, Capacity & Exclusivity: - •Compatibility:xv,r=0,∀r∉pvx\_\{v,r\}=0,\\forall r\\notin p\_\{v\}\. - •Capacity:∑vav,r≤c​a​pr,∀r∈ℛ\\sum\_\{v\}a\_\{v,r\}\\leq cap\_\{r\},\\forall r\\in\\mathcal\{R\}\. - •Exclusivity \(allow waiting\):∑rxv,r≤1\\sum\_\{r\}x\_\{v,r\}\\leq 1and0≤av,r≤xv,r,∀v0\\leq a\_\{v,r\}\\leq x\_\{v,r\},\\forall v\.No hybrid execution:a node \(if scheduled\) runs on a single resource type\.
3. 3\.Batching & Throughput: - •Ifxv,r=1x\_\{v,r\}=1:1≤bv,r≤Br1\\leq b\_\{v,r\}\\leq B\_\{r\}\. Batching occurs only among ready nodes with consistentθv\\theta\_\{v\}and parallelizability\. - •Ifxv,r=1x\_\{v,r\}=1, we define the per\-unit runtime Tv,i=o​v​e​r​h​e​a​dv\+wv,iμr​\(bv,r,av,rkv;θv\),i=1,…,kv,T\_\{v,i\}=overhead\_\{v\}\+\\frac\{w\_\{v,i\}\}\{\\mu\_\{r\}\\\!\\left\(b\_\{v,r\},\\frac\{a\_\{v,r\}\}\{k\_\{v\}\};\\theta\_\{v\}\\right\)\},\\quad i=1,\\dots,k\_\{v\},\(13\)and the node runtime/finish time as Tv=maxi∈\{1,…,kv\}⁡Tv,i,Cv=Fv\+Tv\.T\_\{v\}=\\max\_\{i\\in\\\{1,\\dots,k\_\{v\}\\\}\}T\_\{v,i\},\\quad C\_\{v\}=F\_\{v\}\+T\_\{v\}\.\(14\)
4. 4\.Parallel Splitting & Load Balancing: - •1≤kv≤sv,∀v∈𝒱q1\\leq k\_\{v\}\\leq s\_\{v\},\\forall v\\in\\mathcal\{V\}\_\{q\}\. - •Ifkv\>1k\_\{v\}\>1, we executevvas\{vi\}i=1kv\\\{v\_\{i\}\\\}\_\{i=1\}^\{k\_\{v\}\}with an equal workload split: wv,i=wvkv,i=1,…,kv,w\_\{v,i\}=\\frac\{w\_\{v\}\}\{k\_\{v\}\},\\quad i=1,\\dots,k\_\{v\},\(15\)and assume equal resource\-share split across units, i\.e\., each unit receivesav,r/kva\_\{v,r\}/k\_\{v\}on the chosen resource type\. The logical node completes when all units complete, i\.e\.,Cv=maxi⁡Cv,iC\_\{v\}=\\max\_\{i\}C\_\{v,i\}\. We include synchronization overheads ino​v​e​r​h​e​a​dvoverhead\_\{v\}\. - •Pipeline split consistency:For a decomposed linear chain, we enforce the same split factor along pipeline edgesℰqp​i​p​e⊆ℰq\\mathcal\{E\}\_\{q\}^\{pipe\}\\subseteq\\mathcal\{E\}\_\{q\}: ku=kv,∀\(u→v\)∈ℰqp​i​p​e⊆ℰq\.k\_\{u\}=k\_\{v\},\\quad\\forall\(u\\to v\)\\in\\mathcal\{E\}\_\{q\}^\{pipe\}\\subseteq\\mathcal\{E\}\_\{q\}\.\(16\)

#### A\.2\.2Scheduling Strategies \(Optional Soft Constraint Formulation\)

We model these strategies as soft constraints with penalties, though implementations may use scoring logic\. Hereεϕ\>0\\varepsilon\_\{\\phi\}\>0andεS\>0\\varepsilon\_\{S\}\>0are penalty weights\.

1. 1\.Downstream Contribution Priority:For competing ready nodesu,vu,von the same resource type, ifϕu\>ϕv\\phi\_\{u\}\>\\phi\_\{v\}, thenuuenables more downstream work\. We encourageuuto start no later thanvvvia a hinge penalty: εϕ⋅∑\(u,v\):ϕu\>ϕvmax⁡\{0,Fu−Fv\}\.\\varepsilon\_\{\\phi\}\\cdot\\sum\_\{\(u,v\):\\phi\_\{u\}\>\\phi\_\{v\}\}\\max\\\{0,F\_\{u\}\-F\_\{v\}\\\}\.\(17\)
2. 2\.Short Query Priority \(SRTF\):For competing ready nodesv∈𝒱q,v′∈𝒱q′v\\in\\mathcal\{V\}\_\{q\},v^\{\\prime\}\\in\\mathcal\{V\}\_\{q^\{\\prime\}\}on the same resource type, ifS^q<S^q′\\widehat\{S\}\_\{q\}<\\widehat\{S\}\_\{q^\{\\prime\}\}, we apply a penalty for scheduling the short query behind the long one: εS⋅max⁡\{0,Fv−Fv′\}\.\\varepsilon\_\{S\}\\cdot\\max\\\{0,F\_\{v\}\-F\_\{v^\{\\prime\}\}\\\}\.\(18\)
3. 3\.Long Query Resource Skewing:We monitor the distribution\(μs,σs\)\(\\mu\_\{s\},\\sigma\_\{s\}\)\(mean and standard deviation\) of activeS^q\\widehat\{S\}\_\{q\}\. We define a dynamic thresholdτl​o​a​d=μs\+η⋅σs\\tau\_\{load\}=\\mu\_\{s\}\+\\eta\\cdot\\sigma\_\{s\}, whereη\>0\\eta\>0is a hyperparameter\. - •IfS^q\>τl​o​a​d\\widehat\{S\}\_\{q\}\>\\tau\_\{load\}, the system marks queryqqas a long query\. Its heavy nodes are markedyv=1y\_\{v\}=1\(otherwiseyv=0y\_\{v\}=0\)\. - •Flexible resource reservation:For each resource typer∈ℛr\\in\\mathcal\{R\}, define the set of ready heavy nodesℋr​\(t\)=\{v∈𝒱r​e​a​d​y​\(t\):yv=1,r∈pv\}\\mathcal\{H\}\_\{r\}\(t\)=\\\{v\\in\\mathcal\{V\}\_\{ready\}\(t\):y\_\{v\}=1,\\ r\\in p\_\{v\}\\\}\. Ifℋr​\(t\)≠∅\\mathcal\{H\}\_\{r\}\(t\)\\neq\\emptyset, we reserve a minimum share for heavy nodes: ∑v∈ℋr​\(t\)av,r≥ρ⋅c​a​pr,∀r∈ℛ,\\sum\_\{v\\in\\mathcal\{H\}\_\{r\}\(t\)\}a\_\{v,r\}\\geq\\rho\\cdot cap\_\{r\},\\quad\\forall r\\in\\mathcal\{R\},\(19\)whereρ∈\(0,1\]\\rho\\in\(0,1\]is the reserved fraction\. This reservation is flexible: ifℋr​\(t\)=∅\\mathcal\{H\}\_\{r\}\(t\)=\\emptyset, the system allocates it to any ready nodes; when heavy nodes exist, we serve them with higher priority within the reserved budget\. - •Heavy nodes receive minimum resource guarantees and higher splitting: ifxv,r=1x\_\{v,r\}=1, we enforceav,r≥ηl​o​n​g⋅yva\_\{v,r\}\\geq\\eta\_\{long\}\\cdot y\_\{v\}andkv≥κl​o​n​g⋅yvk\_\{v\}\\geq\\kappa\_\{long\}\\cdot y\_\{v\}, whereηl​o​n​g∈\(0,1\]\\eta\_\{long\}\\in\(0,1\]andκl​o​n​g∈ℕ\+\\kappa\_\{long\}\\in\\mathbb\{N\}^\{\+\}are hyperparameters\.
4. 4\.Redundancy Elimination & Result Sharing:We merge nodesℳ​\(v∗\)\\mathcal\{M\}\(v^\{\*\}\)identified as semantically identical into a single executionv∗v^\{\*\}\. - •Nodesv∈ℳ​\(v∗\)v\\in\\mathcal\{M\}\(v^\{\*\}\)depend on the output ofv∗v^\{\*\}\. - •The contribution of the representative node is the sum:ϕv∗=∑v∈ℳ​\(v∗\)ϕv\\phi\_\{v^\{\*\}\}=\\sum\_\{v\\in\\mathcal\{M\}\(v^\{\*\}\)\}\\phi\_\{v\}\. This naturally increases the priority of shared nodes\.

### A\.3Online Solving Strategy

#### A\.3\.1Online Framework

1. 1\.Rolling Horizon \+ Greedy/Approximation:At each eventtt, we make decisions only for ready nodes of active queries in𝒬t\\mathcal\{Q\}\_\{t\}\. We solve the optimization problem \(with soft constraints\) for the ready set, update states \(x,b,a,k,F,T,Cx,b,a,k,F,T,C\), and advance to the next event\.
2. 2\.λ\\lambda\-Adaptation:The balancing coefficientλ\\lambdaadapts to load\. Under high load, we increaseλ\\lambdato lower average completion time; when tail latency increases, we decreaseλ\\lambdato suppressCm​a​xC\_\{max\}\.
3. 3\.Non\-Static Critical Path:We drive scheduling using SRTF/PLAS heuristics and Downstream Contribution \(ϕv\\phi\_\{v\}\), using resource skewing to handle long tails, rather than static critical path analysis\.

#### A\.3\.2Algorithm Design

The execution logic of the scheduling is formalized in Algorithm[1](https://arxiv.org/html/2607.22578#alg1), illustrating how the online scheduler performs decisions upon being triggered by events, including New Query Arrivals, Node Completions, and Resource Releases\.

Algorithm 1Pseudocode of the online scheduler\.0:Active queries

𝒬t\\mathcal\{Q\}\_\{t\}; resource types

ℛ\\mathcal\{R\}with

c​a​prcap\_\{r\}and

BrB\_\{r\}; parameters

\(α,γ,δ,ρ\)\(\\alpha,\\gamma,\\delta,\\rho\)\.

1:On each event

e∈\{arrival,completion,release\}e\\in\\\{\\textit\{arrival\},\\textit\{completion\},\\textit\{release\}\\\}\.

2:

⊳\\trianglerightarrival: new query arrives;completion: a node finishes;release: a resource becomes available\.

3:Phase 1: State Update & Ready Set Construction

4:for all

q∈𝒬tq\\in\\mathcal\{Q\}\_\{t\}do

5:Update attained service time

Aq​\(t\)A\_\{q\}\(t\)and remaining\-time estimate

S^q​\(t\)\\widehat\{S\}\_\{q\}\(t\)\.

6:endfor

7:Construct ready set

𝒱r​e​a​d​y​\(t\)=\{v:all predecessors of​v​are completed\}\\mathcal\{V\}\_\{ready\}\(t\)=\\\{v:\\text\{ all predecessors of \}v\\text\{ are completed\}\\\}\.

8:Merge semantic duplicates

ℳ​\(v∗\)\\mathcal\{M\}\(v^\{\*\}\)into representatives

v∗v^\{\*\}for reuse and result sharing\.

9:Insert each ready node into per\-resource queues \(

short\_queuer\\textit\{short\\\_queue\}\_\{r\}/

long\_queuer\\textit\{long\\\_queue\}\_\{r\}\) for all compatible

r∈pvr\\in p\_\{v\}\.

10:Phase 2: Prioritization

11:for all

v∈𝒱r​e​a​d​y​\(t\)v\\in\\mathcal\{V\}\_\{ready\}\(t\)do

12:Compute contribution

ϕv\\phi\_\{v\}and priority

πv=α/\(S^q​\(v\)\+ϵ\)\+γ​ϕv\+δ​yv\\pi\_\{v\}=\\alpha/\(\\widehat\{S\}\_\{q\(v\)\}\+\\epsilon\)\+\\gamma\\phi\_\{v\}\+\\delta y\_\{v\}\.

13:Update the priority key of

vvin the corresponding queue\(s\)\.

14:endfor

15:Phase 3: Joint Allocation

16:for all

r∈ℛr\\in\\mathcal\{R\}do

17:while

c​a​prcap\_\{r\}remains and queues are non\-emptydo

18:Select from

short\_queuer\\textit\{short\\\_queue\}\_\{r\}and

long\_queuer\\textit\{long\\\_queue\}\_\{r\}\(skewing/reservation with

ρ\\rhowhen applicable\)\.

19:Form candidate batch

B​a​t​c​hBatchfrom queue head \(consistent type, limit

BrB\_\{r\}\)\.

20:Search feasible

\(b,a\)\(b,a\); compute

TvT\_\{v\}for

v∈B​a​t​c​hv\\in Batch; pick the configuration with max net gain\.

21:For heavy nodes \(

yv=1,sv\>1y\_\{v\}=1,s\_\{v\}\>1\), choose split

kv∈\[1,sv\]k\_\{v\}\\in\[1,s\_\{v\}\]based on

λ\\lambda\.

22:Commit

\(x,b,a,k,F,T,C\)\(x,b,a,k,F,T,C\), update resource availability and

S^q\\widehat\{S\}\_\{q\}, and proceed to next event\.

23:endwhile

24:endfor

25:return

Cq=max⁡CvC\_\{q\}=\\max C\_\{v\}when all queries are complete\.

Phase 1: State Update & Ready Set Construction

1. 1\.Online update ofAq​\(t\)A\_\{q\}\(t\)andS^q​\(t\)\\widehat\{S\}\_\{q\}\(t\)for SRTF/PLAS metrics\.
2. 2\.FormReady Set: Nodes with all predecessors completed\. Specifically, for each resource typerr, we maintain two queues \(short\_queuerandlong\_queuer\)\. We insert a ready nodevvinto queues of its compatible typesr∈pvr\\in p\_\{v\}, and route it toshort\_queuerorlong\_queuerbased on its query label\.
3. 3\.Redundancy Detection:We merge semantically matching nodesℳ​\(v∗\)\\mathcal\{M\}\(v^\{\*\}\)into representative nodev∗v^\{\*\}to reduce computation and latency\.

Phase 2: PrioritizationCalculate priority scoreπv\\pi\_\{v\}for each ready node:

πv=αS^q​\(v\)\+ϵ\+γ⋅ϕv\+δ⋅yv,\\pi\_\{v\}=\{\\frac\{\\alpha\}\{\\widehat\{S\}\_\{q\(v\)\}\+\\epsilon\}\}\+\{\\gamma\\cdot\\phi\_\{v\}\}\+\{\\delta\\cdot y\_\{v\}\},\(20\)whereϵ\>0\\epsilon\>0prevents division by zero\.

- •Term 1 \(SRTF/PLAS\):Prioritizes queries with small remaining timeS^q\\widehat\{S\}\_\{q\};S^q\\widehat\{S\}\_\{q\}is updated online after decoding sub\-nodes using PLAS\-inspired correction\.
- •Term 2 \(Topology\):Prioritizes critical nodes enabling downstream work \(ϕv\\phi\_\{v\}\)\.
- •Term 3 \(Anti\-Starvation\):Increases priority for heavy nodes \(yvy\_\{v\}\) in long queries\.

We maintain eachshort\_queuerandlong\_queueras a priority queue keyed byπv\\pi\_\{v\}\(descending\)\.

Phase 3: Joint AllocationFor each resource typerr, while capacity/shares remain:

1. 1\.Batch Construction:We select the next node fromshort\_queuerandlong\_queuer\(e\.g\., pick the higherπv\\pi\_\{v\}\), then extract a candidate batch \(B​a​t​c​hBatch\) from the queue head \(consistent type, limitBrB\_\{r\}\)\.
2. 2\.Determine Batch Size \(bb\) & Share \(aa\): - •We search the setb∈\{1,…,min⁡\(Br,\|B​a​t​c​h\|\)\}b\\in\\\{1,\\dots,\\min\(B\_\{r\},\|Batch\|\)\\\}and candidate strategies foraa\(Equal/Skewed/Weighted\)\. - •We computeTvT\_\{v\}for∀v∈B​a​t​c​h\\forall v\\in Batch\. - •We calculate the score based on marginal improvement to𝒥\\mathcal\{J\}\(approx\.\) minus soft constraint penalties\. - •We select the configuration with maximum net gain\.
3. 3\.Parallel Splitting \(kk\):For heavy nodes \(yv=1,sv\>1y\_\{v\}=1,s\_\{v\}\>1\), we split withkv∈\[1,sv\]k\_\{v\}\\in\[1,s\_\{v\}\]based onλ\\lambda\. We use equal workload splitwv,i=wv/kvw\_\{v,i\}=w\_\{v\}/k\_\{v\}and treat\{vi\}i=1kv\\\{v\_\{i\}\\\}\_\{i=1\}^\{k\_\{v\}\}as parallel execution units sharing the batch configuration\.
4. 4\.Enforcing Skewing:Ifyv=1y\_\{v\}=1andxv,r=1x\_\{v,r\}=1, we ensureav,ra\_\{v,r\}meetsηl​o​n​g\\eta\_\{long\}\.
5. 5\.Commit:We fix \(x,b,a,k,F,T,Cx,b,a,k,F,T,C\), update resource availability andS^q\\widehat\{S\}\_\{q\}, and proceed to next event\.

Termination:When all queries are complete, outputCq=max⁡CvC\_\{q\}=\\max C\_\{v\}\.

## Appendix BTheoretical Analysis

### B\.1NP\-Hardness Proof

Theorem:The \(offline\) HeraSys scheduling problem is NP\-hard\.

Proof:We reduce from the precedence constrained scheduling on identical parallel machines,P​\|p​r​e​c\|​Cm​a​xP\|prec\|C\_\{max\}, which is known to be NP\-hard\. Consider an instance ofP​\|p​r​e​c\|​Cm​a​xP\|prec\|C\_\{max\}withnnjobs, processing times\{pj\}\\\{p\_\{j\}\\\}, precedence constraints, andmmidentical machines\. We construct a HeraSys instance as follows:

- •Create a single queryqqwhose DAG nodes\{vj\}j=1n\\\{v\_\{j\}\\\}\_\{j=1\}^\{n\}correspond to jobs, and add edges to match the precedence constraints\.
- •Use a single resource typerrwith capacitymm, disable batching \(Br=1B\_\{r\}=1\), and disable splitting \(svj=1s\_\{v\_\{j\}\}=1for alljj\)\.
- •Setpvj=\{r\}p\_\{v\_\{j\}\}=\\\{r\\\},o​v​e​r​h​e​a​dvj=0overhead\_\{v\_\{j\}\}=0, and chooseθvj\\theta\_\{v\_\{j\}\}such thatμr​\(1,1;θvj\)=1\\mu\_\{r\}\(1,1;\\theta\_\{v\_\{j\}\}\)=1, yieldingTvj=wvj=pjT\_\{v\_\{j\}\}=w\_\{v\_\{j\}\}=p\_\{j\}when scheduled\.

Since the active query set size\|𝒬t\|=1\|\\mathcal\{Q\}\_\{t\}\|=1, the objective reduces to minimizingCq=maxv∈𝒱q⁡CvC\_\{q\}=\\max\_\{v\\in\\mathcal\{V\}\_\{q\}\}C\_\{v\}, which equals the makespanCm​a​xC\_\{max\}\. Any feasible schedule for the HeraSys instance corresponds to a feasible schedule for the originalP​\|p​r​e​c\|​Cm​a​xP\|prec\|C\_\{max\}instance with the same makespan, and vice versa\. Therefore, an optimal solution to HeraSys yields an optimal solution toP​\|p​r​e​c\|​Cm​a​xP\|prec\|C\_\{max\}, implying HeraSys is NP\-hard\.

### B\.2Algorithm Properties and Advantages

#### B\.2\.1Competitiveness and Optimality

Since future arrival times are unknown, this is an online algorithm\.

- •SRTF Time Priority \(with PLAS correction\):The first term ofπv\\pi\_\{v\}\(1/\(S^q\+ϵ\)1/\(\\widehat\{S\}\_\{q\}\+\\epsilon\)\) approximates the SRTF policy\. Theoretically, SRTF minimizes the average waiting time in single\-processor preemptive scheduling environments\(Schrage,[1968](https://arxiv.org/html/2607.22578#bib.bib1)\)\. In our setting, we updateS^q\\widehat\{S\}\_\{q\}online at decoding sub\-node boundaries using attained service timeAq​\(t\)A\_\{q\}\(t\), based on PLAS\.
- •Preemption Design & Decoding Decomposition:In traditional LLM serving, the unpredictability of decoding tasks often leads to non\-preemptive behavior\. We decompose decoding operators into sequentially dependent sub\-nodes\. Upon completion of a sub\-node, we updateAq​\(t\)A\_\{q\}\(t\)with the observed runtime and refreshS^q​\(t\)\\widehat\{S\}\_\{q\}\(t\), enabling PLAS\-style priority decay for long\-attained queries while preserving preemption points\.
- •Sub\-optimality:Due to the greedy strategy \(considering only current Ready nodes\), global optimality is not guaranteed\. For example, executing a short task now might block a soon\-to\-arrive critical long task\. Theλ\\lambda\-adaptation and resource skewing mechanisms attempt to balance average and worst\-case performance\.

#### B\.2\.2Liveness and Safety

- •Deadlock\-Free:The DAG structure guarantees acyclic dependencies\. The Ready Set contains only nodes with completed predecessors\. Resource allocation is atomic per event, preventing hold\-and\-wait cycles\.
- •Starvation Mitigation \(Heuristic\):The algorithm prevents starving long queries via resource reservation, heavy\-node boosting \(yvy\_\{v\}\), minimum shareηl​o​n​g\\eta\_\{long\}, and adaptiveλ\\lambda\. A formal starvation\-free guarantee can be obtained by adding anagingterm toπv\\pi\_\{v\}\(or enforcing a hard priority floor once a query waits longer than a threshold\), which we leave as an optional extension\.

#### B\.2\.3Rationale Against Static Critical Path

In online scheduling scenarios constrained by limited and non\-fully parallel resources, determining the “Critical Path” is computationally infeasible at the moment of decision\-making\. The actual critical path typically emerges only post\-facto after scheduling is complete, rather than serving as available a priori information \(accurately predicting the critical path beforehand incurs significant computational overhead and costs\)\.

- •FvF\_\{v\}depends on contention, batching, splitting, and future arrivals, which are unknown a priori\.
- •Forcing priority for the estimated longest path can degrade∑Cq\\sum C\_\{q\}\.Counter\-example:A long queryqLq\_\{L\}with a long path but high parallelism might block critical resources needed by multiple short, resource\-sensitive queriesqSq\_\{S\}\.
- •Instead, we use Node Topology Contribution \(ϕv\\phi\_\{v\}\) as a proxy for Longest Remaining Path adapted for dynamic environments, identifying potential bottlenecks in real\-time\.

### B\.3Complexity Analysis

We analyze the computational complexity of the HeraSys online scheduling algorithm for a single decision event\. LetNNdenote the total number of active nodes in the system, andNr​e​a​d​yN\_\{ready\}denote the number of ready nodes \(subset ofNNwhose dependencies are satisfied\) at the current timestamp\. Let\|ℛ\|\|\\mathcal\{R\}\|be the number of distinct resource types\. We introduceBm​a​x=maxr⁡BrB\_\{max\}=\\max\_\{r\}B\_\{r\}as the maximum batch size limit andSm​a​x=maxv⁡svS\_\{max\}=\\max\_\{v\}s\_\{v\}as the maximum split factor\.

The scheduling procedure is divided into three sequential phases\. We analyze the asymptotic complexity for each:

1\. Phase 1: State Update and Redundancy Elimination

- •State Update:Updating the remaining service timeS^q\\widehat\{S\}\_\{q\}and identifying long queries involves iterating through active queries\. Since node states update incrementally upon completion events, the overhead is linear with respect to the number of nodes processed in the current event window, bounded byO​\(Nr​e​a​d​y\)O\(N\_\{ready\}\)\.
- •Redundancy Elimination:The system employs a hash\-based mechanism to detect duplicates\. For each ready node, computing the semantic hash and querying the global duplication map takesO​\(1\)O\(1\)time on average \(assuming a good hash function\)\. Thus, processing the entire ready set takesO​\(Nr​e​a​d​y\)O\(N\_\{ready\}\)\.

2\. Phase 2: Prioritization and Sorting

- •Scoring:Calculating the priority scoreπv\\pi\_\{v\}for all ready nodes involves constant\-time arithmetic operations per node, totalingO​\(Nr​e​a​d​y\)O\(N\_\{ready\}\)\.
- •Sorting:The ready nodes are sorted to form candidate queues for each resource type\. Using an efficient comparison\-based sorting algorithm \(e\.g\., Quicksort or Timsort\), this operation requiresO​\(Nr​e​a​d​y​log⁡Nr​e​a​d​y\)O\(N\_\{ready\}\\log N\_\{ready\}\)\. This step typically dominates the complexity of this phase\.

3\. Phase 3: Joint AllocationThe allocator iterates through resource types and greedily assigns tasks\.

- •Search Space:For each allocation decision, the algorithm constructs a candidate batch and searches for the optimal configuration\(b,a,k\)\(b,a,k\)\. The search iterates through feasible batch sizesb∈\[1,min⁡\(\|B​a​t​c​h\|,Bm​a​x\)\]b\\in\[1,\\min\(\|Batch\|,B\_\{max\}\)\]and split countsk∈\[1,Sm​a​x\]k\\in\[1,S\_\{max\}\]\.
- •Cost per Decision:The configuration search costsO​\(Bm​a​x⋅Sm​a​x\)O\(B\_\{max\}\\cdot S\_\{max\}\)\. Scoring each configuration uses a local surrogate ofΔ​𝒥\\Delta\\mathcal\{J\}computed from cached per\-query estimates \(thusO​\(1\)O\(1\)per configuration\)\. If one instead recomputes the exact global objective with full dependency propagation, the worst\-case cost per configuration can beO​\(\|ℰ\|\)O\(\|\\mathcal\{E\}\|\)\.
- •Total Allocation Cost:In the worst case, the allocator iterates through all nodes in the candidate queues once to assign them or determine they cannot be scheduled, resulting inO​\(\|ℛ\|⋅Nr​e​a​d​y⋅Bm​a​x⋅Sm​a​x\)O\(\|\\mathcal\{R\}\|\\cdot N\_\{ready\}\\cdot B\_\{max\}\\cdot S\_\{max\}\), which is linear inNr​e​a​d​yN\_\{ready\}whenBm​a​x,Sm​a​xB\_\{max\},S\_\{max\}are treated as constants\.

Overall ComplexityAggregating the three phases, the total time complexity per scheduling event is dominated by the sorting operation in Phase 2 \(under the local\-surrogate scoring implementation\):

𝒯e​v​e​n​t=O​\(Nr​e​a​d​y​log⁡Nr​e​a​d​y\)\\mathcal\{T\}\_\{event\}=O\(N\_\{ready\}\\log N\_\{ready\}\)\(21\)
Note thatNr​e​a​d​yN\_\{ready\}is constrained by DAG dependencies and represents only theexecution frontier\(i\.e\., the instantaneous available parallelism\) of the graph, satisfyingNr​e​a​d​y≪NN\_\{ready\}\\ll N\. Given that LLM inference tasks \(especially the prefill and decoding phases\) are computationally intensive, the scheduling overhead incurred by this log\-linear complexity is negligible relative to task execution time\. This low asymptotic complexity ensures that HeraSys scales efficiently to high\-concurrency scenarios without becoming a system bottleneck\.

### B\.4Limitations

1. 1\.Dependency onS^q\\widehat\{S\}\_\{q\}Estimation:The algorithm relies onS^q\\widehat\{S\}\_\{q\}for SRTF\. While decomposition assists correction, large prediction errors \(common in LLM generation\) can degrade performance\.
2. 2\.Local Minima:Greedy batching might force high\-priority nodes to wait to fill a batch, or optimize throughput at the expense ofmax⁡Cq\\max C\_\{q\}\.λ\\lambda\-adaptation mitigates this but does not eliminate it\.
3. 3\.Splitting Overhead:Splitting reduces latency but introduces synchronization overhead \(o​v​e​r​h​e​a​dvoverhead\_\{v\}\)\. If split too finely, Amdahl’s Law limits speedup, and costs may outweigh benefits\.