Threshold-Based Exclusive Batching for LLM Inference

# Threshold-Based Exclusive Batching for LLM Inference
Source: [https://arxiv.org/html/2606.00516](https://arxiv.org/html/2606.00516)
###### Abstract

Mixed batching \(MB\)—interleaving prefill and decode in a single batch—has become the standard scheduling strategy for large language model \(LLM\) inference due to its efficiency in maximizing compute and memory utilization\. However, through controlled experiments, we find that prefill–decode interference inflates MB’s per\-step marginal cost above that of pure decode\. On the high\-bandwidth H200 \(4\.8 TB/s\), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth\-constrained RTX PRO 6000 \(1\.792 TB/s\), this threshold plummets to just 20%\. Consequently, the optimal choice between MB and exclusive batching \(EB\) fundamentally depends on GPU memory bandwidth, model size, and workload composition\. We derive a closed\-form condition for this EB–MB performance crossover, along with asymptotically optimal phase\-switching thresholds and memory\-safe batch sizing for EB\. Optimized EB achieves up to 41\.9% higher throughput on bandwidth\-constrained GPUs, while MB retains its advantage on high\-bandwidth hardware with larger models\. Our hybrid scheduler EB\+applies this condition online to dynamically switch between EB and MB without manual intervention\. Under non\-stationary traffic with distribution or concurrency shifts, EB\+attains the highest or near\-highest throughput in every setting, outperforming MB by up to 36\.4%\.111Code:[https://github\.com/weifang231/eb\-vllm](https://github.com/weifang231/eb-vllm)\.

Machine Learning, ICML

\\hyper@makecurrent

Hfootnote

## 1Introduction

Large language model \(LLM\) inference consists of two distinct phases with fundamentally different computational characteristics\. The*prefill*phase processes input tokens in parallel to populate the key\-value \(KV\) cache, making it compute\-bound\. The*decode*phase generates tokens autoregressively, requiring repeated memory accesses to the KV cache, making it memory\-bandwidth\-bound\(Kwonet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib13); Popeet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib15); Wanget al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib4)\)\. This dichotomy creates an inherent inefficiency: during decoding, GPU compute units remain underutilized, while during prefill, memory bandwidth is not fully exploited\.

Two dominant scheduling paradigms have emerged\.*Mixed batching*\(MB\)\(Agrawalet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib6)\)interleaves prefill and decode operations within the same batch, simultaneously utilizing GPU compute for prefill and memory bandwidth for decode\.*Exclusive batching*\(EB\) processes prefill and decode in separate batches, alternating phases by a scheduling rule\. In this work we study a*capacity\-triggered*policy that switches to the prefill phase wheneverkkdecode slots become idle, denoting this family EB\(kk\)\. MB has gained widespread adoption, with major inference engines including vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib13)\)and SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib16)\)transitioning to MB as their default scheduling mode\. Throughout the paper we use v1 to denote vLLM v1 \(as the MB baseline\) and v0 to denote the vLLM v0 exclusive\-batching scheduler, equivalent to EB\(k=1k\{=\}1\) when saturated\.

In the context of single\-GPU deployments, an intriguing dichotomy nonetheless persists in practice: while Western inference engines have largely standardized on MB, many large\-scale production systems in China continue to favor EB\. A plausible contributing factor is hardware—GPUs accessible in the Chinese market operate under tighter memory\-bandwidth budgets, owing in part to export restrictions\. This raises a concrete question: does memory bandwidth fundamentally alter the EB\-vs\-MB trade\-off, and if so, when should each strategy be preferred?

To probe this question, we examine the marginal cost of processing tokens under different batching disciplines\. We model the iteration time for processingntokn\_\{\\mathrm\{tok\}\}tokens in a single batch asTiter=α\+β⋅ntokT\_\{\\text\{iter\}\}=\\alpha\+\\beta\\cdot n\_\{\\mathrm\{tok\}\}, whereα\\alphais the fixed overhead andβ\\betathe marginal cost per token\. Crucially, bothα\\alphaandβ\\betadepend on \(1\) the hardware profile \(e\.g\., available memory bandwidth\) and \(2\) the batch composition, which we characterize by the*decode ratio*r:=ndecode/ntokr:=n\_\{\\mathrm\{decode\}\}/n\_\{\\mathrm\{tok\}\}, the fraction of decode tokens in a batch \(r=1r\{=\}1for pure decode;0<r<10\{<\}r\{<\}1for a mixed batch\)\.

Controlled experiments on two GPUs with contrasting bandwidth—NVIDIA RTX PRO 6000 \(1\.792 TB/s\) and NVIDIA H200 \(4\.8 TB/s\)—reveal a sharp hardware dependence\. The crossover point at which the mixed\-batch marginal cost exceeds the pure\-decode cost falls atr≈20%r\\\!\\approx\\\!20\\%on the RTX PRO 6000 but only atr≈80%r\\\!\\approx\\\!80\\%on the H200 \(Figure[1](https://arxiv.org/html/2606.00516#S1.F1); averaged over 20 runs, variance negligible\)\. These observations lead to our central hypothesis: EB with an optimized threshold should outperform MB in bandwidth\-constrained environments, while MB should retain its advantage on high\-bandwidth hardware, motivating both an analysis of the EB–MB crossover and the design of adaptive EB scheduling\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x1.png)Figure 1:Execution time for mixed batches \(Qwen3\-4B, ctx=16\) on RTX PRO 6000 \(left\) and H200 \(right\)\.Kernel profiling \(Figure[2](https://arxiv.org/html/2606.00516#S1.F2)\) localizes the source of this gap: for a fixed total token count, GEMM time is largely invariant torr, whereas Attention time grows withrr\. Mixed\-batch Attention becomes slower than pure\-decode Attention atr≈20%r\\\!\\approx\\\!20\\%on the RTX PRO 6000, but only near80%80\\%on the H200\. Consistent patterns hold across model scales \(Gemma\-3\-1B\-IT, Qwen3\-8B, Qwen3\-30B\-A3B\) and decode context lengths \(Appendix[A\.2](https://arxiv.org/html/2606.00516#A1.SS2)\): both curves shift uniformly without altering their shape or crossover points\.

This cross\-GPU gap is consistent with bandwidth limitations\(Williamset al\.,[2009](https://arxiv.org/html/2606.00516#bib.bib46)\): decode attention streams the full KV\-cache context per token and is therefore memory\-bandwidth\-bound\. On bandwidth\-constrained GPUs, co\-locating prefills and decodes intensifies bandwidth contention and disproportionately inflates Attention latency; on high\-bandwidth GPUs, this interference is weaker\. This also suggests that current FlashAttention kernels\(Daoet al\.,[2022](https://arxiv.org/html/2606.00516#bib.bib36); Dao,[2024](https://arxiv.org/html/2606.00516#bib.bib37); Shahet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib38)\)are not fully optimized for mixed\-batch inference on bandwidth\-limited GPUs, motivating a re\-examination of the EB\-vs\-MB trade\-off\.

Based on these findings, we derive a closed\-form condition characterizing when EB outperforms MB, governed by the marginal\-cost gap between mixed and exclusive batches \(Section[3](https://arxiv.org/html/2606.00516#S3)\)\. To realize optimal EB in practice, we derive an analytical expression for the normalized thresholdθ∗=k∗/N\\theta^\{\*\}=k^\{\*\}/Nand develop an adaptive scheduler, denoted EB\(k^∗\\hat\{k\}^\{\*\}\), that computesk^∗\\hat\{k\}^\{\*\}online from workload characteristics at runtime\. We further use the same crossover condition to drive online mode selection between EB and MB\.

This paper makes the following contributions:

- •We provide empirical evidence that MB does not universally dominate EB, and derive a closed\-form condition for the EB–MB crossover governed by the marginal\-cost gap and amortized fixed\-cost advantage\.
- •We develop an adaptive EB scheduler with closed\-form, asymptotically optimal phase\-switching thresholds under stochastic output\-length distributions, paired with memory\-aware batch sizing that has probabilistic OOM guarantees\.
- •We propose EB\+, a hybrid scheduler that applies the crossover condition online to switch between EB and MB at runtime, remaining robust under non\-stationary traffic and matching or exceeding PD\-disaggregation throughput without extra hardware\.
- •We validate these results on four NVIDIA GPUs spanning 0\.9–8\.0 TB/s bandwidth: EB\(k^∗\\hat\{k\}^\{\*\}\) achieves up to 41\.9% throughput gains on bandwidth\-constrained GPUs; on high\-bandwidth hardware with larger models, v1 retains its advantage; and EB\+delivers the highest joint TTFT/TPOT goodput across traffic regimes\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x2.png)Figure 2:Kernel time breakdown for Qwen3\-4B on RTX PRO 6000 \(top\) and H200 \(bottom\) vs\. decode ratiorrat three total token counts\. Black line: total mixed\-batch cost\. Right panels add the EB cost \(red dashed\) and EB\-vs\-MB threshold \(blue dashed, Eq\. \([7](https://arxiv.org/html/2606.00516#S3.E7)\)\); EB is favored when the black line exceeds the blue\.
## 2Related Work

The decode phase is memory\-bandwidth\-bound even at large batch sizes, leaving over 50% of GPU compute idle\(Recasenset al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib25)\)\. Its Byte\-per\-FLOP demand exceeds prefill’s by∼100×\{\\sim\}100\\times, and this bottleneck deepens across GPU generations as compute outpaces bandwidth\(Liet al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib24)\)\. Three scheduling strategies emerged in response\.

Exclusive Batchingprocesses one phase per iteration\(Yuet al\.,[2022](https://arxiv.org/html/2606.00516#bib.bib7); Kwonet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib13); NVIDIA,[2023a](https://arxiv.org/html/2606.00516#bib.bib14); Aminabadiet al\.,[2022](https://arxiv.org/html/2606.00516#bib.bib49)\), avoiding intra\-iteration prefill–decode interference;Panget al\.\([2025](https://arxiv.org/html/2606.00516#bib.bib18)\)extend this regime with a Lagrangian analysis of prefill\-vs\-decode insertion\.

Mixed Batchingco\-locates prefill and decode tokens via*chunked prefill*, splitting long prefills under a fixed token budget\. Sarathi\(Agrawalet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib6),[2024](https://arxiv.org/html/2606.00516#bib.bib5)\)introduced this with decode\-maximal batching, with a similar formulation in DeepSpeed\-FastGen’s Dynamic SplitFuse\(Holmeset al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib17)\); it is now the default in vLLM \(v1\), SGLang\(Zhenget al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib16)\), TGI\(HuggingFace,[2023](https://arxiv.org/html/2606.00516#bib.bib39)\), TensorRT\-LLM\(NVIDIA,[2023b](https://arxiv.org/html/2606.00516#bib.bib40)\), and LightLLM\(ModelTC,[2023](https://arxiv.org/html/2606.00516#bib.bib41)\)\. Chunking itself is orthogonal to EB/MB: EB can also chunk prefills to bound iteration time\. MB pays two costs: co\-location induces prefill–decode interference, and the aggressive chunking it requires \(forced by sharing the token budget with decode\) inflates MoE memory traffic by up to 39% via redundant expert reloads\(Leeet al\.,[2026](https://arxiv.org/html/2606.00516#bib.bib19)\)and scales long\-context KV\-cache loads asO​\(N2\)O\(N^\{2\}\)forNN\-chunk prompts\(Zhonget al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib20)\)\.

Disaggregated Servingeliminates inter\-phase interference via dedicated P/D GPU pools \(DistServe\(Zhonget al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib20)\), Splitwise\(Patelet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib21)\), Mooncake\(Qinet al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib12)\)\)\. The cost is twofold: architecturally, KV\-cache transfer overhead \(severe without NVLink/IB\) and a doubled minimum GPU count; operationally, the static P:D split underutilizes decode under imbalance\(Shiet al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib22)\)and cannot adapt to demand shifts\(Honget al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib23)\)\.

Orthogonal techniques target other layers of the serving stack: MoE serving\(Rajbhandariet al\.,[2022](https://arxiv.org/html/2606.00516#bib.bib10); Galeet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib11)\)and quantization\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib44); Linet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib45)\)optimize the model, KV cache management\(Zhanget al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib8); Xiaoet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib9)\)compresses memory, and speculative decoding\(Leviathanet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib42); Miaoet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib43)\)reduces decode iterations\.

Within the scheduling\-policy axis, we optimize EB via a closed\-form online controller and combine it with MB into an adaptive hybrid \(EB\+\) that navigates the EB/MB trade\-off on a single GPU pool\.

## 3Scheduling Model for Exclusive Batching

We develop a scheduling framework for LLM serving underexclusive batching, where prefill and decode operations cannot be executed concurrently\. We formulate the phase\-switching decision as an optimization problem, derive optimal thresholds, and present an online adaptive algorithm\.

### 3\.1Problem Formulation

System Model\.Consider a system withNN*slots*\(the*maximum batch size*\); each slot can hold the state of one in\-progress*request*and is either*busy*\(currently occupied by a request\) or*idle*\(vacated upon completion\)\. The system alternates between prefill and decode phases and operates in a saturated regime: the queue is consistently backlogged, so each batch operates at capacityNN\.

During a decode phase, which begins withNNbusy slots, a slot becomes idle whenever its corresponding request completes \(i\.e\., generates an end\-of\-sequence token\)\. A*phase switch*occurs when the number of idle slots reaches a predetermined thresholdkk: the system then enters a prefill phase, loadingkknew requests into the vacated slots\.

During this prefill phase, as thekknew requests are processed, the\(N−k\)\(N\-k\)incomplete requests remain inactive—their KV cache is preserved but no tokens are generated\. Once the prefill completes, the system resumes decoding with a full batch ofNNrequests, consisting of the\(N−k\)\(N\-k\)ongoing requests and thekknewly admitted ones\. This dynamic creates a throughput\-latency trade\-off:largekk\(late switching\) leaves completed slots idle longer and inflates queued requests’ wait time, whilesmallkk\(early switching\) amortizes the fixed prefill overheadαp\\alpha\_\{p\}over fewer requests and reduces efficiency\.

Timing Model\.Each request has a random input lengthLL\(i\.i\.d\. with meanμL\\mu\_\{L\}\) and output lengthOOdrawn from a general distributionFFwith hazard rateh​\(t\)h\(t\), meanμO\\mu\_\{O\}, and standard deviationσO\\sigma\_\{O\}\. Assuming linear iteration time and modeling each prefill phase as a single batch, the prefill time for a batch ofkkrequests \(i\.e\., the length of a prefill phase\) isTp​\(k\)=αp\+βp​∑i=1kLiT\_\{p\}\(k\)=\\alpha\_\{p\}\+\\beta\_\{p\}\\textstyle\\sum\_\{i=1\}^\{k\}L\_\{i\}, whereαp\\alpha\_\{p\}is the fixed prefill overhead andβp\\beta\_\{p\}the per\-token prefill cost\. The decode iteration time at a batch sizennistd​\(n\)=αd\+βd​nt\_\{d\}\(n\)=\\alpha\_\{d\}\+\\beta\_\{d\}n, and the total decode timeTd​\(k;N\)T\_\{d\}\(k;N\)is the sum of iteration times in a decode phase, depending on both the starting batch sizeNNand the numberkkof requests that complete before the phase ends\.

Throughput Optimization Problem\.We define throughput as the average number of request completions per unit time and seek the thresholdkkand batch sizeNNthat maximize it subject to a KV\-cache memory constraint\. LetCCdenote the KV\-cache capacity andXmax​\(k,N\)X\_\{\\max\}\(k,N\)the peak instantaneous memory footprint of the unconstrained execution\. The problem is formulated as follows:

maxk,N\\displaystyle\\max\_\{k,N\}\\quadTP​\(k,N\):=k𝔼​\[Td​\(k;N\)\]\+αp\+βp​k​μL,\\displaystyle\\mathrm\{TP\}\(k,N\):=\\frac\{k\}\{\\mathbb\{E\}\[T\_\{d\}\(k;N\)\]\+\\alpha\_\{p\}\+\\beta\_\{p\}k\\mu\_\{L\}\},\(1\)s\.t\.k∈\{1,…,N\},\\displaystyle k\\in\\\{1,\\ldots,N\\\},Pr⁡\(Xmax​\(k,N\)\>C\)≤ϵ,\\displaystyle\\Pr\(X\_\{\\max\}\(k,N\)\>C\)\\leq\\epsilon,\(2\)
Approximate Solution Approach\.We adopt a*decoupled approximation*that decomposes the joint optimization into two sequential steps\. First, we derive the optimal normalized thresholdθ∗=k∗/N\\theta^\{\*\}=k^\{\*\}/Nin an asymptotic regime asN→∞N\\to\\infty, wherek∗k^\{\*\}is the optimal solution to \([1](https://arxiv.org/html/2606.00516#S3.E1)\) without the memory constraint \([2](https://arxiv.org/html/2606.00516#S3.E2)\) for a fixedNN\. We show thatθ∗\\theta^\{\*\}converges to a limitθ0\\theta\_\{0\}that depends exclusively on the output distribution and is independent ofNN\(Section[3\.2](https://arxiv.org/html/2606.00516#S3.SS2)\)\. Second, we fixk=⌊θ0​N⌋k=\\lfloor\\theta\_\{0\}N\\rfloorand determine the maximum batch sizeN∗N^\{\*\}that satisfies \([2](https://arxiv.org/html/2606.00516#S3.E2)\) \(Section[3\.3](https://arxiv.org/html/2606.00516#S3.SS3)\)\.

While this decoupled method solves an approximation to the original problem \([1](https://arxiv.org/html/2606.00516#S3.E1)\)—meaning the thresholdθ0\\theta\_\{0\}is optimal asymptotically and may not perfectly coincide with the exact finite\-NNsolution—it provides significant analytical advantages\. Specifically, the approximation yields tractable closed\-form expressions, revealing important structural insights into LLM inference scheduling and enabling efficient online adaptation\. We empirically validate its near\-optimal performance in Section[4\.2](https://arxiv.org/html/2606.00516#S4.SS2)\.

Furthermore, to derive a closed\-form expression for the expected duration of a decode phase,𝔼​\[Td​\(k;N\)\]\\mathbb\{E\}\[T\_\{d\}\(k;N\)\], we apply*fluid approximation*\. Here, the stochastic decode completion process is approximated by its deterministic limit as the system scale approaches infinity\. This approximation is highly accurate in the large\-batch regime typical of modern LLM serving\.

### 3\.2Optimal Switching Threshold

Intuitively, balancing the amortization of the fixed prefill cost against the waste of idle decode slots depends fundamentally on how likely additional completions are in the near future\. Thus, the*hazard rate*h​\(t\)=f​\(t\)/F¯​\(t\)h\(t\)=f\(t\)/\\bar\{F\}\(t\)of the output\-length distribution plays a key role: a higher hazard rate means completions arrive faster \(favoring delayed switching\), while a lower rate makes waiting costly \(favoring earlier switching\)\.

We first analyze the constant failure rate \(CFR\) case, which admits a closed\-form solution, and then extend the analysis to the increasing failure rate \(IFR\) case, which better captures real LLM workloads where longer\-running requests become progressively more likely to complete\.

#### 3\.2\.1Base Threshold Under CFR

Assume that the decode length distribution exhibits a constant hazard rateh​\(t\)=p0h\(t\)=p\_\{0\}\(i\.e\., geometric output lengths\), implyingμO=1/p0\\mu\_\{O\}=1/p\_\{0\}\. Under the fluid approximation, the scaled number of decode requests decreases according to the differential equationn˙​\(t\)=−p0​n​\(t\)\\dot\{n\}\(t\)=\-p\_\{0\}\\,n\(t\)during a decode phase, yielding the expected duration:

𝔼​\[Td​\(k;N\)\]=\[βd​N​θ−αd​ln⁡\(1−θ\)\]​p0−1,\\mathbb\{E\}\[T\_\{d\}\(k;N\)\]=\\left\[\\beta\_\{d\}N\\theta\-\\alpha\_\{d\}\\ln\(1\-\\theta\)\\right\]p\_\{0\}^\{\-1\},whereθ=kN\\theta=\\frac\{k\}\{N\}\. Substituting𝔼​\[Td​\(k;N\)\]\\mathbb\{E\}\[T\_\{d\}\(k;N\)\]into \([1](https://arxiv.org/html/2606.00516#S3.E1)\), we obtain the throughput under EB:

TPEB​\(k,N\)=\[αp−αd​μO​ln⁡\(1−θ\)k\+βEBw​\(μL\+μO\)\]−1,\\mathrm\{TP\}\_\{\\mathrm\{EB\}\}\(k,N\)=\\left\[\\tfrac\{\\alpha\_\{p\}\-\\alpha\_\{d\}\\mu\_\{O\}\\ln\(1\-\\theta\)\}\{k\}\+\\beta\_\{\\mathrm\{EB\}\}^\{w\}\(\\mu\_\{L\}\+\\mu\_\{O\}\)\\right\]^\{\-1\},whereβEBw=βp​μL\+βd​μOμL\+μO\\beta\_\{\\mathrm\{EB\}\}^\{w\}=\\frac\{\\beta\_\{p\}\\mu\_\{L\}\+\\beta\_\{d\}\\mu\_\{O\}\}\{\\mu\_\{L\}\+\\mu\_\{O\}\}represents the workload\-weighted average of the prefill and decode marginal costs\. Solvingk∗=arg​maxk⁡TPEB​\(k,N\)k^\{\*\}=\\operatorname\*\{arg\\,max\}\_\{k\}\\mathrm\{TP\}\_\{\\mathrm\{EB\}\}\(k,N\)yields the optimal switching threshold for the unconstrained problem \([1](https://arxiv.org/html/2606.00516#S3.E1)\)\. The following proposition establishes the limiting behavior ofk∗k^\{\*\}as the batch sizeN→∞N\\to\\infty\.

###### Proposition 3\.1\(Base Threshold under CFR\)\.

Under a constant hazard rateh​\(t\)=p0h\(t\)=p\_\{0\}and the fluid approximation of the decode phase duration, the limiting optimal normalized thresholdθ0:=limN→∞θ∗=limN→∞k∗/N\\theta\_\{0\}:=\\lim\\limits\_\{N\\to\\infty\}\\theta^\{\*\}=\\lim\\limits\_\{N\\to\\infty\}k^\{\*\}/Nis the unique solution to

θ0​\(1−θ0\)−1\+ln⁡\(1−θ0\)=p0​αp​αd−1,\\theta\_\{0\}\(1\-\\theta\_\{0\}\)^\{\-1\}\+\\ln\(1\-\\theta\_\{0\}\)=p\_\{0\}\\alpha\_\{p\}\\alpha\_\{d\}^\{\-1\},\(3\)which depends only on the single ratiop0​αp/αdp\_\{0\}\\alpha\_\{p\}/\\alpha\_\{d\}\. In particular, it is independent ofNN,μL\\mu\_\{L\}, and the per\-token costsβp,βd\\beta\_\{p\},\\beta\_\{d\}\. We defineζ≜−ln⁡\(1−θ0\)\>0\\zeta\\triangleq\-\\ln\(1\-\\theta\_\{0\}\)\>0for brevity in what follows\. \(Proof in Appendix[B](https://arxiv.org/html/2606.00516#A2)\.\)

This proposition implies thatθ∗\\theta^\{\*\}can be accurately approximated byθ0\\theta\_\{0\}in the large\-batch regime\. To facilitate solving the capacity\-constrained problem in the second step, we use the approximated valuek0∗:=⌊θ0​N⌋k^\{\*\}\_\{0\}:=\\lfloor\\theta\_\{0\}N\\rflooras our practical solution to the unconstrained first\-step problem\. The simplicity of this parameter dependence makes the threshold highly practical to deploy: it is computed once from the easily measured tuple\(αp,αd,p0\)\(\\alpha\_\{p\},\\alpha\_\{d\},p\_\{0\}\)\.

The optimal throughput for the first\-step problem is then given by:

TPEB​\(k0∗,N\)=\[αp\+αd​ζ​μOk0∗\+βEBw​\(μL\+μO\)\]−1,\\mathrm\{TP\}\_\{\\mathrm\{EB\}\}\(k^\{\*\}\_\{0\},N\)=\\left\[\\frac\{\\alpha\_\{p\}\+\\alpha\_\{d\}\\zeta\\mu\_\{O\}\}\{k^\{\*\}\_\{0\}\}\+\\beta\_\{\\mathrm\{EB\}\}^\{w\}\(\\mu\_\{L\}\+\\mu\_\{O\}\)\\right\]^\{\-1\},\(4\)

#### 3\.2\.2Extension to IFR

Real LLM workloads exhibit IFR: as a request generates more tokens, it becomes progressively more likely to complete in the near future\. Within a decode phase, this accelerating completion rate reduces the additional wall\-clock cost of waiting for one more idle slot\. Consequently, the opportunity cost of delaying the phase switch is lower than under CFR, allowing the system to profitably accumulate more completions before switching\. In short, IFR workloads support*higher*optimal switching thresholds than the CFR baseline\.

Because the strength of IFR varies across workloads, the exact optimal limitθ∗\\theta^\{\*\}can differ substantially from the base CFR limitθ0\\theta\_\{0\}, motivating an analytical correction\. We model IFR using a linear hazard rateh​\(t\)=p0\+η​th\(t\)=p\_\{0\}\+\\eta twithη\>0\\eta\>0\.

###### Theorem 3\.2\(IFR Threshold Correction\)\.

Under a linear hazard rateh​\(t\)=p0\+η​th\(t\)=p\_\{0\}\+\\eta twithη\>0\\eta\>0, the optimal threshold admits the expansionθ∗=θ0\+Δ​θ\+O​\(η2\)\\theta^\{\*\}=\\theta\_\{0\}\+\\Delta\\theta\+O\(\\eta^\{2\}\), whereθ0\\theta\_\{0\}is the CFR base threshold from Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1)and

Δ​θ=η​\(1−θ0\)2p02​θ0​\[ζ​\(θ01−θ0−ζ2\)⏟duration effect\+βd​Nαd​\(ζ−θ0\)⏟per\-token cost effect\],\\Delta\\theta=\\frac\{\\eta\(1\-\\theta\_\{0\}\)^\{2\}\}\{p\_\{0\}^\{2\}\\,\\theta\_\{0\}\}\\bigg\[\\underbrace\{\\zeta\\left\(\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}\\right\)\}\_\{\\text\{duration effect\}\}\+\\underbrace\{\\frac\{\\beta\_\{d\}N\}\{\\alpha\_\{d\}\}\(\\zeta\-\\theta\_\{0\}\)\}\_\{\\text\{per\-token cost effect\}\}\\bigg\],\(5\)withζ=−ln⁡\(1−θ0\)\\zeta=\-\\ln\(1\-\\theta\_\{0\}\)\. The correction satisfiesΔ​θ\>0\\Delta\\theta\>0for allη\>0\\eta\>0\. \(Proof in Appendix[C](https://arxiv.org/html/2606.00516#A3); empirical validation in Appendix[E\.3](https://arxiv.org/html/2606.00516#A5.SS3)\.\)

The IFR correction reveals two structural features absent in the base case\. First, unlike theNN\-independentθ0\\theta\_\{0\},Δ​θ\\Delta\\thetadepends explicitly onρ=βd​N/αd\\rho=\\beta\_\{d\}N/\\alpha\_\{d\}, so IFR effects amplify for large batches with high per\-token overhead \(ρ≫1\\rho\\gg 1\)\. Second, the prefactorη/p02\\eta/p\_\{0\}^\{2\}automatically rescales the correction across workloads with varying degrees of IFR\.

### 3\.3Memory\-Constrained Batch Sizing

Givenθ0\\theta\_\{0\}and settingk=k0∗=⌊θ0​N⌋k=k^\{\*\}\_\{0\}=\\lfloor\\theta\_\{0\}N\\rfloor, we now determine the maximum batch sizeN∗N^\{\*\}such that the memory constraint \([2](https://arxiv.org/html/2606.00516#S3.E2)\) is satisfied\. Memory evolves dynamically during decode—increasing as tokens are generated and dropping abruptly upon request completion—creating a sawtooth pattern whose peak determines feasibility\.

###### Proposition 3\.3\(Memory\-Safe Batch Size\)\.

Under the CFR model with thresholdθ0\\theta\_\{0\}\(i\.e\.,k=⌊θ0​N⌋k=\\lfloor\\theta\_\{0\}N\\rfloor\), the maximum batch size satisfyingPr⁡\(Xmax​\(k,N\)\>C\)≤ϵ\\Pr\(X\_\{\\max\}\(k,N\)\>C\)\\leq\\epsilonis

N∗=⌊C−ln⁡\(1/ϵ\)/\(p02​μL\)μL\+1−θ0θ0​p0​ln⁡11−θ0⌋\.N^\{\*\}=\\left\\lfloor\\frac\{C\-\\ln\(1/\\epsilon\)/\(p\_\{0\}^\{2\}\\mu\_\{L\}\)\}\{\\mu\_\{L\}\+\\frac\{1\-\\theta\_\{0\}\}\{\\theta\_\{0\}\\,p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\_\{0\}\}\}\\right\\rfloor\.\(6\)\(Proof in Appendix[D](https://arxiv.org/html/2606.00516#A4)\.\)

Here, we utilize the geometric model for analytical tractability\. The solution \([6](https://arxiv.org/html/2606.00516#S3.E6)\) also serves as a conservative bound for IFR scenarios, as IFR workloads yield lower peak memory due to more predictable completion patterns\.

### 3\.4Online Adaptive Algorithm

We design an online controller that jointly adapts\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)by estimating workload parameters\(p^0,η^,μ^L\)\(\\hat\{p\}\_\{0\},\\hat\{\\eta\},\\hat\{\\mu\}\_\{L\}\)from recent requests and evaluating the closed\-form expressions of Sections[3\.2](https://arxiv.org/html/2606.00516#S3.SS2)–[3\.3](https://arxiv.org/html/2606.00516#S3.SS3)at the current estimates\. Here, the hat notation \(⋅^\\hat\{\\cdot\}\) denotes the empirically estimated values of unknown parameters\.

Online estimation\.The system maintains sliding windows of the most recent output lengths \(𝒲O\\mathcal\{W\}\_\{O\}\) and input lengths \(𝒲L\\mathcal\{W\}\_\{L\}\)\. From𝒲O\\mathcal\{W\}\_\{O\}, we estimate the empirical hazard rate at stepttas the fraction of completions at exactlyttamong those still active attt:h^​\(t\)=\#​\{O∈𝒲O:O=t\}/\#​\{O∈𝒲O:O≥t\}\\hat\{h\}\(t\)=\\\#\\\{O\\in\\mathcal\{W\}\_\{O\}:O\{=\}t\\\}/\\\#\\\{O\\in\\mathcal\{W\}\_\{O\}:O\{\\geq\}t\\\}, and fith^​\(t\)=p^0\+η^​t\\hat\{h\}\(t\)=\\hat\{p\}\_\{0\}\+\\hat\{\\eta\}tvia weighted least squares overt∈\[1,t95\]t\\in\[1,t\_\{95\}\], wheret95t\_\{95\}is the 95th percentile of recent output lengths\. From𝒲L\\mathcal\{W\}\_\{L\}, we estimateμ^L\\hat\{\\mu\}\_\{L\}via the sample mean\.

Threshold and batch\-size update\.Given\(p^0,η^\)\(\\hat\{p\}\_\{0\},\\hat\{\\eta\}\), we computeθ^0\\hat\{\\theta\}\_\{0\}by solving \([3](https://arxiv.org/html/2606.00516#S3.E3)\) and apply the IFR correction \([5](https://arxiv.org/html/2606.00516#S3.E5)\) to obtainθ^∗=θ^0\+Δ​θ^\\hat\{\\theta\}^\{\*\}=\\hat\{\\theta\}\_\{0\}\+\\widehat\{\\Delta\\theta\}\(clipped to a practical range\[θmin,θmax\]\[\\theta\_\{\\min\},\\theta\_\{\\max\}\]\)\. BecauseΔ​θ^\\widehat\{\\Delta\\theta\}depends onNNthroughρ=βd​N/αd\\rho=\\beta\_\{d\}N/\\alpha\_\{d\}, we evaluate \([5](https://arxiv.org/html/2606.00516#S3.E5)\) using the previous cycle’sN^∗\\hat\{N\}^\{\*\}, yielding a one\-step fixed\-point update that we empirically observe to converge within a few cycles\. We then compute a memory\-safe batch size by applying Proposition[3\.3](https://arxiv.org/html/2606.00516#S3.Thmtheorem3)withθ=θ^∗\\theta=\\hat\{\\theta\}^\{\*\}and\(p0,μL\)=\(p^0,μ^L\)\(p\_\{0\},\\mu\_\{L\}\)=\(\\hat\{p\}\_\{0\},\\hat\{\\mu\}\_\{L\}\), yieldingN^∗=N∗​\(θ^∗,p^0,μ^L;C,ϵ\)\\hat\{N\}^\{\*\}=N^\{\*\}\(\\hat\{\\theta\}^\{\*\},\\hat\{p\}\_\{0\},\\hat\{\\mu\}\_\{L\};C,\\epsilon\)\. Finally, we setk^∗=⌊θ^∗​N^∗⌋\\hat\{k\}^\{\*\}=\\lfloor\\hat\{\\theta\}^\{\*\}\\hat\{N\}^\{\*\}\\rfloorand apply the updated tuple\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)\. Independent of this periodic update cycle, a runtime KV\-aware gate \(Appendix[D\.6](https://arxiv.org/html/2606.00516#A4.SS6)\) monitors memory and defers the decode→\\toprefill transition whenever instantaneous KV occupancy leaves insufficient headroom\. This absorbs any random memory excursions thatN^∗\\hat\{N\}^\{\*\}may underestimate under high\-σO\\sigma\_\{O\}workloads\. Pseudo\-code is given in Algorithm[1](https://arxiv.org/html/2606.00516#alg1)\(Appendix[D\.7](https://arxiv.org/html/2606.00516#A4.SS7)\)\.

### 3\.5Adaptive Mode Selection: EB\+

Having derived \([4](https://arxiv.org/html/2606.00516#S3.E4)\) to approximate the throughput of EB\(k^∗\\hat\{k\}^\{\*\}\), we now formulate the throughput of MB\. By comparing the two, we deduce the condition \(referred to as the*crossover condition*\) under which one strategy outperforms the other\. Building on this analytical foundation, we then designEB\+, a hybrid scheduler that adaptively switches between EB and MB at runtime to globally maximize efficiency\.

MB Versus EB\.We first approximate the throughput of MB in steady\-state\. LetNpN\_\{p\}be the average number of requests finishing prefill per iteration\. By Little’s Law, the average number of tokens decoded in each iteration isNp​μON\_\{p\}\\mu\_\{O\}\. Because the MB discipline generates exactly one decoded token per request per iteration, the average number of decoding requests in a batch is directlyNd=Np​μON\_\{d\}=N\_\{p\}\\mu\_\{O\}\. Consequently, the average effective batch size per iteration isN=Np\+Nd=Np​\(1\+μO\)N=N\_\{p\}\+N\_\{d\}=N\_\{p\}\(1\+\\mu\_\{O\}\)\.

The system throughput under MB is thus formulated as:

TPMB​\(N\)\\displaystyle\\mathrm\{TP\}\_\{\\mathrm\{MB\}\}\(N\)=Np​\[αMB\+βMBe​\(Np​μL\+Nd\)\]−1\\displaystyle=N\_\{p\}\\left\[\\alpha\_\{\\mathrm\{MB\}\}\+\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(N\_\{p\}\\mu\_\{L\}\+N\_\{d\}\)\\right\]^\{\-1\}=\[αMB​\(1\+μO\)​N−1\+βMBe​\(μL\+μO\)\]−1,\\displaystyle=\\left\[\\alpha\_\{\\mathrm\{MB\}\}\(1\+\\mu\_\{O\}\)N^\{\-1\}\+\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(\\mu\_\{L\}\+\\mu\_\{O\}\)\\right\]^\{\-1\},whereαMB\\alpha\_\{\\mathrm\{MB\}\}is the fixed overhead for launching a mixed batch, andβMBe\\beta\_\{\\mathrm\{MB\}\}^\{e\}is the empirically measured effective per\-token marginal cost\.

When standardizingNNto be both the maximum batch size for EB and the effective average batch size for MB, we can directly compare their throughputs\.

###### Proposition 3\.4\.

Under the steady\-state approximations above,TPMB​\(N\)\>TPEB​\(k0∗,N\)\\mathrm\{TP\}\_\{\\mathrm\{MB\}\}\(N\)\>\\mathrm\{TP\}\_\{\\mathrm\{EB\}\}\(k^\{\*\}\_\{0\},N\)if and only if

βMBe−βEBw<1μL\+μO​\[αp\+αd​ζ​μOk0∗−αMB​\(1\+μO\)N\]\.\\beta\_\{\\mathrm\{MB\}\}^\{e\}\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}<\\frac\{1\}\{\\mu\_\{L\}\+\\mu\_\{O\}\}\\\!\\left\[\\frac\{\\alpha\_\{p\}\+\\alpha\_\{d\}\\zeta\\mu\_\{O\}\}\{k^\{\*\}\_\{0\}\}\-\\frac\{\\alpha\_\{\\mathrm\{MB\}\}\(1\{\+\}\\mu\_\{O\}\)\}\{N\}\\right\]\\\!\.\(7\)

In Proposition[3\.4](https://arxiv.org/html/2606.00516#S3.Thmtheorem4), the LHS represents the*marginal\-cost gap*incurred by co\-locating prefill and decode tokens in the same forward pass\. The RHS captures the*amortized fixed\-cost advantage*of MB, which requires fewer distinct kernel launches\. Substitutingk0∗=θ0​Nk^\{\*\}\_\{0\}=\\theta\_\{0\}N, both terms inside the bracket of the RHS scale as1/N1/N\. Therefore, the RHS isO​\(1/N\)O\(1/N\)and naturally vanishes at high utilization\. In saturated regimes, the comparison is entirely governed by the marginal\-cost gapβMBe−βEBw\\beta\_\{\\mathrm\{MB\}\}^\{e\}\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}\. On bandwidth\-constrained GPUs, prefill–decode interference drastically inflates this gap\. Figure[2](https://arxiv.org/html/2606.00516#S1.F2)confirms this empirically: on the H200, the mixed\-batch marginal cost remains low across most decode ratios, whereas on the RTX PRO 6000, bandwidth constraints inflate the cost above the threshold over intermediate ratios, heavily favoring EB\.

Online switching criterion\.At runtime, EB\+instantiates \([7](https://arxiv.org/html/2606.00516#S3.E7)\) using online system estimates and the EMA\-smoothed active batch occupancyNobsN\_\{\\text\{obs\}\}\. To avoid oversensitivity to the noisyη^\\hat\{\\eta\}estimate, we substitutek^0∗=θ^0​Nobs\\hat\{k\}^\{\*\}\_\{0\}=\\hat\{\\theta\}\_\{0\}N\_\{\\text\{obs\}\}\. \(Because the RHS scales as1/Nobs1/N\_\{\\text\{obs\}\}and the gapθ^∗−θ^0=O​\(η\)\\hat\{\\theta\}^\{\*\}\-\\hat\{\\theta\}\_\{0\}=O\(\\eta\)is bounded, the resulting estimation bias isO​\(1/Nobs\)O\(1/N\_\{\\text\{obs\}\}\)and vanishes under saturation\.\) By reversing the inequality from Proposition[3\.4](https://arxiv.org/html/2606.00516#S3.Thmtheorem4), we formally define the EB\+control rule—the system executes an exclusive batch \(EB\) when:

β^MBe​\(r^\)−βEBw\>\(αp\+αd​ζ^​μ^O\)/θ^0−αMB​\(1\+μ^O\)Nobs​\(μ^L\+μ^O\)\+δ,\\hat\{\\beta\}\_\{\\mathrm\{MB\}\}^\{e\}\(\\hat\{r\}\)\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}\>\\frac\{\(\\alpha\_\{p\}\+\\alpha\_\{d\}\\hat\{\\zeta\}\\hat\{\\mu\}\_\{O\}\)/\\hat\{\\theta\}\_\{0\}\-\\alpha\_\{\\mathrm\{MB\}\}\(1\{\+\}\\hat\{\\mu\}\_\{O\}\)\}\{N\_\{\\text\{obs\}\}\(\\hat\{\\mu\}\_\{L\}\+\\hat\{\\mu\}\_\{O\}\)\}\+\\delta,\(8\)and defaults to MB otherwise\. Here, the parameterβ^MBe​\(r^\)\\hat\{\\beta\}\_\{\\mathrm\{MB\}\}^\{e\}\(\\hat\{r\}\)at the estimated decode ratior^=μ^O/\(μ^L\+μ^O\)\\hat\{r\}=\\hat\{\\mu\}\_\{O\}/\(\\hat\{\\mu\}\_\{L\}\+\\hat\{\\mu\}\_\{O\}\)is retrieved from a one\-shot, per\-hardware kernel\-time profile \(see the calibration procedure in Appendix[A\.4](https://arxiv.org/html/2606.00516#A1.SS4)\)\. The scalarδ\\deltaserves as a tunable priority margin: settingδ\>0\\delta\>0lowers the threshold for MB \(favoring lower TTFT\), whileδ<0\\delta<0biases the system toward EB \(favoring higher token throughput\)\.

Traffic\- and workload\-awareness\.Equation \([8](https://arxiv.org/html/2606.00516#S3.E8)\) ensures adaptation along two distinct axes\. First, the RHS scales inversely with the observed occupancyNobsN\_\{\\text\{obs\}\}: under light traffic loads, the RHS is large, causing EB\+to safely default to MB\. As traffic load increases andNobsN\_\{\\text\{obs\}\}grows, the RHS shrinks; once resource contention dominates, the system dynamically shifts to EB\. Independently,β^MBe​\(r^\)\\hat\{\\beta\}\_\{\\mathrm\{MB\}\}^\{e\}\(\\hat\{r\}\)continuously tracks workload drift via the shifting decode ratio\. Through these mechanisms, EB\+cleanly subsumes both EB\(k^∗\\hat\{k\}^\{\*\}\) and MB as theoretical endpoints, seamlessly recovering the superior scheduling discipline regime\-by\-regime without the need for manual intervention\.

## 4Evaluation

### 4\.1Experimental Setup

Hardware\.We evaluate on four GPU platforms \(Table[1](https://arxiv.org/html/2606.00516#S4.T1)\); H200 and RTX PRO 6000 serve as primary high\-bandwidth and bandwidth\-constrained baselines, with B300 and L40S used for scalability\.

Table 1:Hardware specifications of evaluated GPUs\.Implementation\.We implement EB\(k^∗\\hat\{k\}^\{\*\}\) by extending vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2606.00516#bib.bib13)\)with exclusive batching and online\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)updates, leaving PagedAttention unchanged\. We compare three strategies—v0, v1, and EB\(k^∗\\hat\{k\}^\{\*\}\)—on the same vLLM build in BF16\.

Models\.Our primary models are Qwen3\-8B and Qwen3\-30B\-A3B \(MoE\)\(Yanget al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib30)\), with Gemma\-3\-1B\-IT\(Gemma Team,[2025](https://arxiv.org/html/2606.00516#bib.bib31)\)used for supplementary results\.

Scheduling strategies\.We compare three strategies\.v0: the vLLM v0 scheduler, an exclusive\-batching variant that switches to prefill whenever a slot opens; under our saturated regime \(batches always full atNN\) it reduces to EB\(k=1k\{=\}1\)\.v1: the vLLM v1 scheduler, implementing mixed batching with decode\-maximal allocation\.EB\(k^∗\\hat\{k\}^\{\*\}\): exclusive batching with our online controller from Section[3\.4](https://arxiv.org/html/2606.00516#S3.SS4), which adapts the phase\-switching thresholdk^∗\\hat\{k\}^\{\*\}and the memory\-safe batch sizeN^∗\\hat\{N\}^\{\*\}from estimated workload statistics\.

Workloads\.We evaluate both synthetic and real\-world workloads to span the prefill–decode mix\.*Synthetic workloads*fix the mean input and output token counts, with per\-request lengths drawn uniformly between0\.5×0\.5\\timesand1\.5×1\.5\\timesthe corresponding mean: Decode\-heavy \(128 input / 1024 output\), Balanced \(512/512\), and Prefill\-heavy \(1024/128\)\.*Real\-world workloads*: ShareGPT\(ShareGPT,[2023](https://arxiv.org/html/2606.00516#bib.bib27)\)\(conversational; 105 input /≤\\leq500 output\), LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib29)\)\(long\-context; 1,000–4,000 / 20\), NuminaMath\(Liet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib28)\)\(chain\-of\-thought reasoning; 122 / 800–4,000\), and WildChat\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib26)\)\(multi\-turn dialogue;≥\\geq6 turns\)\. Output lengths are truncated during serving; detailed statistics in Appendix[E\.1](https://arxiv.org/html/2606.00516#A5.SS1)\.

Metrics and traffic\.We report throughput in tokens/s \(prefill and decode\) and requests/s \(RPS\), and latency as TTFT \(s\), TPOT \(ms\), and ITL \(s\)\. Unless otherwise noted, experiments operate in a saturated regime \(concurrencyc=2,048c\{=\}2\{,\}048\) with4,0004\{,\}000requests per configuration \(WildChat: 3,000 multi\-turn,∼27,900\{\\sim\}27\{,\}900requests\)\.

### 4\.2Model Validation

We validate the analytical predictions from Section[3](https://arxiv.org/html/2606.00516#S3)on H200 with Qwen3\-8B and synthetic geometric workloads\.

Baseline threshold properties\.We test the structural properties of the CFR baseline thresholdθ0\\theta\_\{0\}from Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1): \(P1\) scale invariance inNN, \(P2\) insensitivity to mean input lengthμL\\mu\_\{L\}, and \(P3\) monotone decrease in mean output lengthμO\\mu\_\{O\}\. All three are empirically confirmed \(Appendix[E\.2](https://arxiv.org/html/2606.00516#A5.SS2)\)\.

Validation of decoupled optimization\.We validate the accuracy of our closed\-form, online adaptive controller on H200 using synthetic geometric output workloads\. We compare \(i\) exhaustive sweeps over fixed switching thresholdskkat a fixed batch sizeN=1024N=1024\(vLLM default\), and \(ii\) our online controller from Section[3\.4](https://arxiv.org/html/2606.00516#S3.SS4), which estimates\(p0,η\)\(p\_\{0\},\\eta\)online and updates\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)accordingly\.

Figure[3](https://arxiv.org/html/2606.00516#S4.F3)plots throughput and time per output token \(TPOT\) as functions ofkkfor three workloads\. The closed\-form choice achieves throughput comparable to, and in our experiments higher than, the best fixedkkin the sweep, improving throughput by up to8\.0%\(decode\-heavy\),3\.6%\(balanced\), and0\.6%\(prefill\-heavy\) over the best fixed\-kkpoint atN=1024N=1024\. The gain is consistent with EB\(k^∗\\hat\{k\}^\{\*\}\) co\-adapting bothθ^∗\\hat\{\\theta\}^\{\*\}andN^∗\\hat\{N\}^\{\*\}, while the sweep is constrained to a single batch size\. TPOT remains competitive: it is close to the best\-throughput setting in the decode\-heavy case and is substantially lower in the balanced and prefill\-heavy cases\. Overall, these results support that the decoupled, closed\-form selection eliminates the need for threshold search while retaining near\-optimal performance\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x3.png)Figure 3:Validation on H200 \(N=1024N\{=\}1024, geometric outputs\): EB\(k^∗\\hat\{k\}^\{\*\}\) matches the best fixed\-kksweep without manual tuning\.
### 4\.3End\-to\-End Performance

We evaluate on RTX PRO 6000 and H200, sweeping the token budgetB∈\{4096,…,18432\}B\\in\\\{4096,\\dots,18432\\\}and max batch sizeN∈\{256,…,2048\}N\\in\\\{256,\\dots,2048\\\}via grid search and reporting the best\-throughput configuration per scheduler \(per\-workload optima in Appendix[F\.1\.2](https://arxiv.org/html/2606.00516#A6.SS1.SSS2)\)\. For EB\(k^∗\\hat\{k\}^\{\*\}\), the adaptive controller computes a memory\-safe batch sizeN^∗\\hat\{N\}^\{\*\}online \(Proposition[3\.3](https://arxiv.org/html/2606.00516#S3.Thmtheorem3)\); the effective batch size ismin⁡\(N^∗,N\)\\min\(\\hat\{N\}^\{\*\},N\)\.

#### 4\.3\.1Synthetic Workloads

We first evaluate on synthetic workloads with controlled input/output length ratios to isolate how workload composition affects scheduling performance\.

Figure[4](https://arxiv.org/html/2606.00516#S4.F4)compares v0, v1, and EB\(k^∗\\hat\{k\}^\{\*\}\) across three input/output regimes on RTX PRO 6000 and H200\. v1 consistently reduces ITL relative to v0 via phase overlapping, but often sacrifices throughput under extreme imbalance\(Sunet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib50)\)\. EB\(k^∗\\hat\{k\}^\{\*\}\) improves throughput across all regimes on RTX PRO 6000 with competitive latency; relative to the best of v0/v1, its gains peak in balanced and decode\-heavy workloads \(v0 already excels on prefill\-heavy, narrowing the margin there\)\. On H200 the throughput gap to v0/v1 narrows substantially\. This confirms EB is particularly effective on bandwidth\-constrained GPUs, where avoiding prefill–decode contention yields clearer throughput benefits\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x4.png)Figure 4:Synthetic workloads \(Qwen3\-8B\): end\-to\-end performance on RTX PRO 6000 and H200\. Stars mark the best strategy per workload/metric\.
#### 4\.3\.2Real\-World Workloads

We further evaluate on four real\-world workloads that represent diverse serving scenarios\. Table[2](https://arxiv.org/html/2606.00516#S4.T2)summarizes results for both Qwen models on RTX PRO 6000 and H200; Gemma\-3\-1B\-IT results are deferred to Appendix[F\.2](https://arxiv.org/html/2606.00516#A6.SS2)\.

On the bandwidth\-constrained RTX PRO 6000, EB\(k^∗\\hat\{k\}^\{\*\}\) outperforms v1 on every workload for Qwen3\-8B \(average\+\+7\.9%\); on Qwen3\-30B\-A3B the gain shrinks to\+\+1\.4%on average, with two workloads slightly negative \(see analysis below\)\. On the high\-bandwidth H200, the performance gap narrows considerably and v1 matches or exceeds EB\(k^∗\\hat\{k\}^\{\*\}\) on several configurations: consistent with our analysis, abundant memory bandwidth mitigates prefill–decode contention and erodes EB’s advantage\. These results corroborate our central hypothesis that the EB–MB crossover is governed by memory bandwidth; we analyze the per\-workload spread under theDecode\-ratio sweet spotbelow\.

Table 2:Throughput \(RPS\) on real\-world workloads\. % Diff\. denotes the relative improvement of EB\(k^∗\\hat\{k\}^\{\*\}\) over v1, computed as\(EB​\(k^∗\)−v1\)/v1×100%\(\\mathrm\{EB\}\(\\hat\{k\}^\{\*\}\)\-\\mathrm\{v1\}\)/\\mathrm\{v1\}\\times 100\\%\.Effect of model size\.Eq\. \([7](https://arxiv.org/html/2606.00516#S3.E7)\) predicts that larger models, which incur higher per\-iteration fixed costsαp,αd,αMB\\alpha\_\{p\},\\alpha\_\{d\},\\alpha\_\{\\mathrm\{MB\}\}, inflating the RHS, make v1 \(MB\) more competitive\. Table[2](https://arxiv.org/html/2606.00516#S4.T2)aligns with this prediction: as we scale from Qwen3\-8B to Qwen3\-30B\-A3B, EB’s average gain over v1 drops from7\.9%to1\.4%on RTX PRO 6000, and from1\.5%to−\-2\.9%on H200\. We attribute this diminishing gap to theα\\alphascaling implied by Eq\. \([7](https://arxiv.org/html/2606.00516#S3.E7)\); the underlying interference\-convexity analysis is in Appendix[A\.3](https://arxiv.org/html/2606.00516#A1.SS3)\.

Decode\-ratio sweet spot\.A convexity analysis \(Appendix[A\.3](https://arxiv.org/html/2606.00516#A1.SS3)\) predicts EB’s largest gains at intermediate decode ratiosrr, diminishing at the extremes \(r→0r\\to 0orr→1r\\to 1\)\. Our workloads span this range: LongBench \(r≈0\.004r\\\!\\approx\\\!0\.004\), ShareGPT/WildChat \(r≈0\.5r\\\!\\approx\\\!0\.5–0\.70\.7\), and NuminaMath \(r\>0\.85r\\\!\>\\\!0\.85\)\. On RTX PRO 6000 with Qwen3\-8B, EB’s gains over v1 indeed peak at moderaterr\(ShareGPT \+15\.3%, WildChat \+11\.3%\) and shrink at the extremes \(LongBench \+3\.7%, NuminaMath \+1\.4%\); on Qwen3\-30B\-A3B, the high\-rrextreme \(NuminaMath,−7\.7%\-7\.7\\%\) is consistent with the two effects compounding, while mid\-rrworkloads split between convexity’s prediction \(WildChat\+11\.6%\+11\.6\\%\) andα\\alpha\-scaling \(ShareGPT−3\.1%\-3\.1\\%\); LongBench remains positive \(\+4\.8%\+4\.8\\%\)\.

#### 4\.3\.3Latency Analysis

TTFT\(Fig\.[5](https://arxiv.org/html/2606.00516#S4.F5)a\)\. v1 generally achieves lower TTFT than EB\(k^∗\\hat\{k\}^\{\*\}\) on most workloads by interleaving prefill with decode to reduce queuing delays\. However, on prefill\-heavy workloads \(LongBench\), EB\(k^∗\\hat\{k\}^\{\*\}\) achieves lower TTFT across both GPUs \(e\.g\., 169\.64 s vs\. 175\.56 s on RTX PRO 6000 with Qwen3\-8B\), as EB avoids fine\-grained chunk scheduling overhead\.

TPOT\(Fig\.[5](https://arxiv.org/html/2606.00516#S4.F5)b\)\. On RTX PRO 6000, EB\(k^∗\\hat\{k\}^\{\*\}\) achieves substantially lower TPOT than v1 on most workloads: 65% reduction on ShareGPT \(93\.68 ms vs\. 268\.97 ms, Qwen3\-8B\), 35% on WildChat \(113\.22 ms vs\. 173\.33 ms\), and 20% on NuminaMath \(65\.22 ms vs\. 81\.75 ms\)\. The exception is LongBench, where v1 achieves lower TPOT due to its prefill\-heavy nature\. On H200, the differences are smaller, though EB\(k^∗\\hat\{k\}^\{\*\}\) remains competitive on decode\-heavy workloads such as WildChat \(79\.04 ms vs\. 122\.97 ms\) and NuminaMath \(26\.87 ms vs\. 31\.03 ms\)\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x5.png)\(a\)TTFT \(s\)
![Refer to caption](https://arxiv.org/html/2606.00516v1/x6.png)\(b\)TPOT \(ms\)

Figure 5:TTFT \(top\) and TPOT \(bottom\) on real\-world workloads\.These results reflect a fundamental trade\-off: v1 favors TTFT through phase overlapping, while EB favors TPOT by avoiding bandwidth contention\.

### 4\.4Adaptive Mode Selection: EB\+

The results above commit to a single mode \(v1 or EB\(k^∗\\hat\{k\}^\{\*\}\)\)\. We now evaluate the hybrid scheduler EB\+\(Section[3\.5](https://arxiv.org/html/2606.00516#S3.SS5)\), which switches online via the crossover criterion \([8](https://arxiv.org/html/2606.00516#S3.E8)\)\.

Traffic\-level sensitivity\.Table[3](https://arxiv.org/html/2606.00516#S4.T3)\(top\) sweeps concurrencyc∈\{32,512,2048\}c\\in\\\{32,512,2048\\\}\. The two hardware columns realize different segments of the crossover\. On the bandwidth\-constrained RTX PRO 6000, EB\(k^∗\\hat\{k\}^\{\*\}\) is TTFT\-handicapped at low load \(∼1\.5×\{\\sim\}1\.5\\timesv1’s TTFT\), but EB\+selects MB atc=32c\{=\}32and recovers v1’s TTFT to within 1 ms; at moderate load it improves throughput by\+62%\+62\\%over v1 with1\.8×1\.8\\timeslower TPOT; and at high load it commits to EB, attaining the best throughput \(\+50%\+50\\%over v1\) with TTFT competitive to EB\(k^∗\\hat\{k\}^\{\*\}\)\. On H200, where v1 is already strong, EB\+reproduces v1’s throughput and TTFT atc=32,512c\{=\}32,512via its MB choice and matches EB\(k^∗\\hat\{k\}^\{\*\}\)’s throughput atc=2048c\{=\}2048\.

Non\-stationary workloads\.We test EB\+under \(i\) a*distribution shift*\(μL:1024→512→128\\mu\_\{L\}\\colon 1024\{\\to\}512\{\\to\}128,μO:128→512→1024\\mu\_\{O\}\\colon 128\{\\to\}512\{\\to\}1024; 2k requests/phase,c=2048c=2048\) and \(ii\) a*concurrency shift*\(c:32→512→1024→256→2048c\\colon 32\{\\to\}512\{\\to\}1024\{\\to\}256\{\\to\}2048\)\. EB\+attains throughput within∼\\sim1% of the better of v1 and EB\(k^∗\\hat\{k\}^\{\*\}\) in all four \(hardware×\\timesscenario\) cells \(Table[3](https://arxiv.org/html/2606.00516#S4.T3), bottom\): distribution shift gives\+36\.4%\+36\.4\\%over v1 on RTX PRO 6000 and\+3\.9%\+3\.9\\%on H200, and concurrency shift gives\+22\.6%\+22\.6\\%and\+0\.6%\+0\.6\\%respectively\. The EMA\-smoothedNobsN\_\{\\text\{obs\}\}enables rapid adaptation without manual re\-tuning\.

Table 3:EB\+under stationary traffic \(top three blocks; concurrencycc,μL=512,μO=256\\mu\_\{L\}\{=\}512,\\mu\_\{O\}\{=\}256\) and non\-stationary workloads \(bottom two blocks\)\. Qwen3\-8B\. TP in tok/s, TTFT in s, TPOT in ms\.Goodput and PD\-disaggregation\.Atc=512c\{=\}512with SLO \(TTFT<10<\\\!10s, TPOT<100<\\\!100ms\), EB\+attains 80\.3% joint goodput versus 77\.3% \(EB\(k^∗\\hat\{k\}^\{\*\}\)\) and 5\.8% \(v1\)\. On4×4\\timesRTX PRO 6000 it is best or near\-best in 7 of 9 workload×c\\times ccells against vLLM’s PD\-disaggregation scheduler, and matches or exceeds v1 on the 2\-GPU setup—without P:D tuning or OOMs\. See Appendices[F\.3](https://arxiv.org/html/2606.00516#A6.SS3),[F\.4](https://arxiv.org/html/2606.00516#A6.SS4)\.

### 4\.5Scalability

Due to the high cost of exhaustive evaluation, we assess scalability using theWildChatworkload with the same grid search procedure as in Section[4\.3](https://arxiv.org/html/2606.00516#S4.SS3)\.

#### 4\.5\.1Across GPU Platforms

Table[4](https://arxiv.org/html/2606.00516#S4.T4)extends the GPU comparison to L40S \(bandwidth\-constrained\) and B300 \(highest bandwidth\)\. On L40S, EB\(k^∗\\hat\{k\}^\{\*\}\) achieves41\.9%higher RPS than v1 \(14\.70 vs\. 10\.36\) with lower TTFT and TPOT\. On B300, EB\(k^∗\\hat\{k\}^\{\*\}\) is on par with v1 \(52\.34 vs\. 50\.47 RPS\) with comparable TTFT and TPOT—consistent with the bandwidth\-driven crossover established in Section[4\.3](https://arxiv.org/html/2606.00516#S4.SS3)\.

Table 4:Scalability across GPU platforms \(Qwen3\-8B, WildChat\)\. RPS in requests/s; TTFT/TPOT in s and ms respectively\.\*v0 does not support PyTorch 2\.9 and CUDA 13\.0, which are required for B300\.

#### 4\.5\.2Across Model Architectures

On RTX PRO 6000 with WildChat, Figure[6](https://arxiv.org/html/2606.00516#S4.F6)evaluates generalization across four model families: Llama\-3\.1\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib32)\), Mathstral\-7B\-v0\.1\(Mistral AI,[2024](https://arxiv.org/html/2606.00516#bib.bib35)\), Qwen2\.5\-Coder\-7B\(Huiet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib33)\), and DeepSeek\-R1\-Distill\-Qwen\-7B\(Guoet al\.,[2025](https://arxiv.org/html/2606.00516#bib.bib34)\)\. EB\(k^∗\\hat\{k\}^\{\*\}\) consistently achieves the highest throughput and lowest TPOT across all models\. Compared to v1, EB\(k^∗\\hat\{k\}^\{\*\}\) improves RPS by 8–17% and reduces TPOT by 17–47%, with the largest gains on Llama\-3\.1\-8B \(47% TPOT reduction\)\. On this bandwidth\-constrained GPU, EB dominates v1 across all tested architectures\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x7.png)Figure 6:Scalability across models on RTX PRO 6000\.
#### 4\.5\.3Robustness to Configuration Parameters

A configuration sweep over\(B,N\)\(B,N\)further shows that all three schedulers exhibit comparable robustness, so EB\(k^∗\\hat\{k\}^\{\*\}\) imposes no additional tuning burden \(Appendix[F\.1](https://arxiv.org/html/2606.00516#A6.SS1)\)\.

## 5Conclusion

This paper shows that neither mixed batching \(MB\) nor exclusive batching \(EB\) universally dominates LLM serving—the optimal choice is governed by a closed\-form crossover condition driven by GPU bandwidth, model size, and workload composition\. Our adaptive EB scheduler, with asymptotically optimal phase\-switching thresholds and memory\-safe batch sizing, achieves up to 41\.9% throughput improvement on bandwidth\-constrained GPUs\. The hybrid EB\+scheduler instantiates this crossover condition online, matching or exceeding both baselines across stationary and non\-stationary regimes without manual tuning, and rivaling disaggregated serving without extra hardware\. Future directions include tighter SLO\-aware switching and joint scheduling with sequence/pipeline parallelism\.

## Acknowledgments

The authors sincerely thank the anonymous reviewers for their insightful comments and constructive suggestions, which significantly improved the quality of this work\. This research was supported in part by the Hong Kong Research Grants Council under Grants 15508021 and 15511424\.

## Impact Statement

This paper advances the field of machine learning by improving the efficiency of LLM inference scheduling\. Our work is a systems\-level optimization: it makes existing models cheaper to serve rather than enabling new capabilities, and the scheduling strategies are agnostic to generated content\. A positive consequence is that EB’s gains on bandwidth\-constrained accelerators broaden the range of hardware on which LLMs can be deployed efficiently, lowering the cost of access\. We do not foresee negative societal consequences beyond those well\-established for LLM deployment\.

## References

- A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, A\. Tumanov, and R\. Ramjee \(2024\)Taming throughput\-latency tradeoff in LLM inference with Sarathi\-Serve\.InUSENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 117–134\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- A\. Agrawal, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, and R\. Ramjee \(2023\)Sarathi: efficient LLM inference by piggybacking decodes with chunked prefills\.arXiv preprint arXiv:2308\.16369\.Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p2.3),[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- R\. Y\. Aminabadi, S\. Rajbhandari, A\. A\. Awan, C\. Li, D\. Li, E\. Zheng, O\. Ruwase, S\. Smith, M\. Zhang, J\. Rasley, and Y\. He \(2022\)DeepSpeed\-Inference: enabling efficient inference of transformer models at unprecedented scale\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis \(SC\),External Links:[Document](https://dx.doi.org/10.1109/SC41404.2022.00051)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p2.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172)Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p5.4)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p7.1)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p7.1)\.
- S\. N\. Ethier and T\. G\. Kurtz \(1986\)Markov processes: characterization and convergence\.John Wiley & Sons\.Cited by:[§B\.1](https://arxiv.org/html/2606.00516#A2.SS1.p2.2)\.
- W\. Feller \(1971\)An introduction to probability theory and its applications\.2nd edition, Vol\.2,John Wiley & Sons,New York\.Cited by:[§D\.4](https://arxiv.org/html/2606.00516#A4.SS4.p1.3)\.
- T\. Gale, D\. Narayanan, C\. Young, and M\. Zaharia \(2023\)MegaBlocks: efficient sparse training with mixture\-of\-experts\.InConference on Machine Learning and Systems \(MLSys\),pp\. 288–304\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- Gemma Team \(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p3.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.5\.2](https://arxiv.org/html/2606.00516#S4.SS5.SSS2.p1.2)\.
- D\. Guo, D\. Yang, H\. Zhang,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§4\.5\.2](https://arxiv.org/html/2606.00516#S4.SS5.SSS2.p1.2)\.
- C\. Holmes, M\. Tanaka, M\. Wyatt, A\. A\. Awan, J\. Rasley, S\. Rajbhandari, R\. Y\. Aminabadi, H\. Qin, A\. Bakhtiari, L\. Kurilenko,et al\.\(2024\)DeepSpeed\-FastGen: high\-throughput text generation for LLMs via MII and DeepSpeed\-Inference\.arXiv preprint arXiv:2401\.08671\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- K\. Hong, L\. Chen, Z\. Wang, X\. Li, Q\. Mao, J\. Ma, C\. Xiong, G\. Wu, B\. Han, G\. Dai, Y\. Liang, and Y\. Wang \(2025\)Semi\-PD: towards efficient LLM serving via phase\-wise disaggregated computation and unified storage\.arXiv preprint arXiv:2504\.19867\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p4.1)\.
- HuggingFace \(2023\)Text generation inference\.Note:[https://github\.com/huggingface/text\-generation\-inference](https://github.com/huggingface/text-generation-inference)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- B\. Hui, J\. Yang, Z\. Cui,et al\.\(2024\)Qwen2\.5\-Coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§4\.5\.2](https://arxiv.org/html/2606.00516#S4.SS5.SSS2.p1.2)\.
- T\. G\. Kurtz \(1970\)Solutions of ordinary differential equations as limits of pure jump Markov processes\.Journal of Applied Probability7\(1\),pp\. 49–58\.Cited by:[§B\.1](https://arxiv.org/html/2606.00516#A2.SS1.p2.2),[§B\.3](https://arxiv.org/html/2606.00516#A2.SS3.p2.6)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InACM Symposium on Operating Systems Principles \(SOSP\),pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p1.1),[§1](https://arxiv.org/html/2606.00516#S1.p2.3),[§2](https://arxiv.org/html/2606.00516#S2.p2.1),[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p2.3)\.
- G\. Lee, J\. Kim, J\. Park, Y\. Lee, and J\. H\. Ahn \(2026\)From tokens to layers: redefining stall\-free scheduling for MoE serving with layered prefill\.InConference on Machine Learning and Systems \(MLSys\),Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. C\. Huang, K\. Rasul, L\. Yu, A\. Jiang, Z\. Shen, Z\. Qin, B\. Dong, L\. Zhou, Y\. Fleureau, G\. Lample, and S\. Polu \(2024\)NuminaMath\.Note:[https://huggingface\.co/AI\-MO/NuminaMath\-CoT](https://huggingface.co/AI-MO/NuminaMath-CoT)Dataset report available at[https://github\.com/project\-numina/aimo\-progress\-prize/blob/main/report/numina\_dataset\.pdf](https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf)Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p5.4)\.
- J\. Li, L\. Qu, T\. Zhang, G\. Chirkov, S\. Xu, P\. Cheng, and L\. Zhou \(2025\)FengHuang: next\-generation memory orchestration for AI inferencing\.arXiv preprint arXiv:2511\.10753\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)AWQ: activation\-aware weight quantization for on\-device LLM compression and acceleration\.InConference on Machine Learning and Systems \(MLSys\),Note:Best Paper AwardCited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi, C\. Shi, Z\. Chen, D\. Arfeen, R\. Abhyankar, and Z\. Jia \(2024\)SpecInfer: accelerating large language model serving with tree\-based speculative inference and verification\.InACM International Conference on Architectural Support for Programming Languages and Operating Systems \(ASPLOS\),Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- Mistral AI \(2024\)Mathstral\-7B\-v0\.1\.Note:[https://huggingface\.co/mistralai/Mathstral\-7B\-v0\.1](https://huggingface.co/mistralai/Mathstral-7B-v0.1)Released by Mistral AI, July 2024; announcement at[https://mistral\.ai/news/mathstral/](https://mistral.ai/news/mathstral/)Cited by:[§4\.5\.2](https://arxiv.org/html/2606.00516#S4.SS5.SSS2.p1.2)\.
- ModelTC \(2023\)LightLLM: a light and fast inference service for LLM\.Note:[https://github\.com/ModelTC/lightllm](https://github.com/ModelTC/lightllm)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- NVIDIA \(2023a\)FasterTransformer\.Note:[https://github\.com/NVIDIA/FasterTransformer](https://github.com/NVIDIA/FasterTransformer)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p2.1)\.
- NVIDIA \(2023b\)TensorRT\-LLM: a TensorRT toolbox for optimized large language model inference\.Note:[https://github\.com/NVIDIA/TensorRT\-LLM](https://github.com/NVIDIA/TensorRT-LLM)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- B\. Pang, K\. Li, R\. She, and F\. Wang \(2025\)Hybrid offline\-online scheduling method for large language model inference optimization\.arXiv preprint arXiv:2502\.15763\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p2.1)\.
- P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. Bianchini \(2024\)Splitwise: efficient generative LLM inference using phase splitting\.InInternational Symposium on Computer Architecture \(ISCA\),pp\. 118–132\.External Links:[Document](https://dx.doi.org/10.1109/ISCA59077.2024.00019)Cited by:[§F\.4](https://arxiv.org/html/2606.00516#A6.SS4.p1.1),[§2](https://arxiv.org/html/2606.00516#S2.p4.1)\.
- R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean \(2023\)Efficiently scaling Transformer inference\.InConference on Machine Learning and Systems \(MLSys\),Vol\.5,pp\. 606–624\.Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p1.1)\.
- R\. Qin, Z\. Li, W\. He, J\. Cui, F\. Ren, M\. Zhang, Y\. Wu, W\. Zheng, and X\. Xu \(2025\)Mooncake: trading more storage for less computation—a KVCache\-centric architecture for serving LLM chatbot\.InUSENIX Conference on File and Storage Technologies \(FAST\),pp\. 155–170\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p4.1)\.
- S\. Rajbhandari, C\. Li, Z\. Yao, M\. Zhang, R\. Y\. Aminabadi, A\. A\. Awan, J\. Rasley, and Y\. He \(2022\)DeepSpeed\-MoE: advancing mixture\-of\-experts inference and training to power next\-generation AI scale\.InInternational Conference on Machine Learning \(ICML\),pp\. 18332–18346\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- P\. G\. Recasens, F\. Agullo, Y\. Zhu, C\. Wang, E\. K\. Lee, O\. Tardieu, J\. Torres, and J\. L\. Berral \(2025\)Mind the memory gap: unveiling GPU bottlenecks in large\-batch LLM inference\.InIEEE International Conference on Cloud Computing \(CLOUD\),pp\. 277–287\.External Links:[Document](https://dx.doi.org/10.1109/CLOUD67622.2025.00036)Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p1.1)\.
- J\. Shah, G\. Bikshandi, Y\. Zhang, V\. Thakkar, P\. Ramani, and T\. Dao \(2024\)FlashAttention\-3: fast and accurate attention with asynchrony and low\-precision\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p7.1)\.
- ShareGPT \(2023\)ShareGPT: user\-shared ChatGPT conversations dataset\.Note:[https://sharegpt\.com/](https://sharegpt.com/)Accessed: 2024Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p5.4)\.
- X\. Shi, C\. Cai, J\. Du, and Z\. Jia \(2025\)Nexus: proactive intra\-GPU disaggregation of prefill and decode in LLM serving\.arXiv preprint arXiv:2507\.06608\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p4.1)\.
- B\. Sun, Z\. Huang, H\. Zhao, W\. Xiao, X\. Zhang, Y\. Li, and W\. Lin \(2024\)Llumnix: dynamic scheduling for large language model serving\.InUSENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 173–191\.Cited by:[§4\.3\.1](https://arxiv.org/html/2606.00516#S4.SS3.SSS1.p2.2)\.
- H\. Wang, X\. Xiao, M\. Yan, Z\. Zhu, D\. Han, D\. Wang, W\. Li, X\. Ye, C\. Hu, H\. Chen,et al\.\(2025\)A systematic characterization of LLM inference on GPUs\.arXiv preprint arXiv:2512\.01644\.Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p1.1)\.
- S\. Williams, A\. Waterman, and D\. Patterson \(2009\)Roofline: an insightful visual performance model for multicore architectures\.Communications of the ACM52\(4\),pp\. 65–76\.External Links:[Document](https://dx.doi.org/10.1145/1498765.1498785)Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p7.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- A\. Yang, A\. Yang, B\. Yang,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p3.1)\.
- G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun \(2022\)Orca: a distributed serving system for Transformer\-Based generative models\.InUSENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 521–538\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p2.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 34661–34710\.Cited by:[§2](https://arxiv.org/html/2606.00516#S2.p5.1)\.
- W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. Deng \(2024\)WildChat: 1M ChatGPT interaction logs in the wild\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2606.00516#S4.SS1.p5.4)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. L\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.\(2024\)SGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37,pp\. 62557–62583\.Cited by:[§1](https://arxiv.org/html/2606.00516#S1.p2.3),[§2](https://arxiv.org/html/2606.00516#S2.p3.2)\.
- Y\. Zhong, S\. Liu, J\. Chen, J\. Hu, Y\. Zhu, X\. Liu, X\. Jin, and H\. Zhang \(2024\)DistServe: disaggregating prefill and decoding for goodput\-optimized large language model serving\.InUSENIX Symposium on Operating Systems Design and Implementation \(OSDI\),pp\. 193–210\.Cited by:[§F\.4](https://arxiv.org/html/2606.00516#A6.SS4.p1.1),[§2](https://arxiv.org/html/2606.00516#S2.p3.2),[§2](https://arxiv.org/html/2606.00516#S2.p4.1)\.

## Appendix ACost Model: Empirical Validation and Extensions

### A\.1Empirical Verification of the Linear Iteration\-Time Model

Our theoretical derivations \(Propositions[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1),[3\.3](https://arxiv.org/html/2606.00516#S3.Thmtheorem3)and Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)\) build on the linear iteration\-time modelTp​\(k\)=αp\+βp​∑iLiT\_\{p\}\(k\)=\\alpha\_\{p\}\+\\beta\_\{p\}\\sum\_\{i\}L\_\{i\}in Section[3\.1](https://arxiv.org/html/2606.00516#S3.SS1)\. Although self\-attention has theoretically quadratic complexity in context length, we show empirically that the quadratic term is negligible in the regime where our scheduler actually operates\.

Scope: the budget that matters is the per\-iteration token budget\.Both MB \(vLLM v1\) and our EB scheduler operate under*chunked prefilling*: a long input is split across multiple iterations, with each iteration processing at mostB=max\_num\_batched\_tokensB=\\texttt\{max\\\_num\\\_batched\\\_tokens\}tokens \(typically 2K–8K, up to 16K on high\-end GPUs\)\. The linear model is therefore applied*per iteration*on at mostBBtokens; it does not need to hold over an entire long input\. All experiments in Section[4\.3](https://arxiv.org/html/2606.00516#S4.SS3)respect this budget\.

Per\-iteration validation within the practical regime\.We profile prefill iteration time on RTX PRO 6000 Blackwell \(96 GB\) across seven models spanning four architectures and hidden dimensionsd∈\[2048,5120\]d\\in\[2048,5120\]: Qwen3\-\{4B, 8B, 14B, 30B\-A3B \(MoE\)\}, Llama\-\{3\.2\-1B, 3\.1\-8B\}, and Mistral\-Nemo\-12B\. For each model we fitT=α\+β​ntokT=\\alpha\+\\beta\\,n\_\{\\text\{tok\}\}to measured prefill times and reportR2R^\{2\}within three budget ranges \(Table[5](https://arxiv.org/html/2606.00516#A1.T5)\)\. Within the practical rangeB≤8​KB\\leq 8\\text\{K\}, every model achievesR2\>0\.986R^\{2\}\>0\.986; extending toB≤16​KB\\leq 16\\text\{K\},R2R^\{2\}stays\>0\.979\>0\.979\. As a cross\-GPU check, repeating the protocol with Qwen3\-4B on H200 also yieldsR2=0\.998R^\{2\}=0\.998, with the quadratic correction improvingR2R^\{2\}by only0\.0010\.001\.

Table 5:Linear\-modelR2R^\{2\}within practical token\-budget ranges on RTX PRO 6000 Blackwell\.The robustness of the linear fit is consistent with the FLOP decomposition of prefill: GEMM dominates, contributing24​d224d^\{2\}FLOPs per token versus4​L​d4Ldfor attention\. Even atL=4096L=4096withd=2560d=2560, the attention fraction isL/\(6​d\)≈27%L/\(6d\)\\approx 27\\%, and GQA further reduces this share, so the quadratic\-in\-LLcomponent is a small fraction of overall prefill cost in the operating regime of interest\.

End\-to\-end serving at 128K context\.To verify that per\-iteration linearity translates to serving\-level gains at long context, we run end\-to\-end benchmarks at 128K input / 64 output tokens on the same GPU under vLLM’s defaultB=8192B=8192\. Each request’s prefill is chunked into∼16\{\\sim\}16iterations, all within the validated linear regime \(R2\>0\.99R^\{2\}\>0\.99atB≤8​KB\\leq 8\\text\{K\}\)\. Concurrency is capped at the maximum value that avoids OOM on a single 96 GB GPU, since the KV cache for a 128K\-token request is large\. Even at this low\-concurrency regime—where MB’s scheduling overhead is amortized and prefill–decode contention is minimal—EB\(k^∗\\hat\{k\}^\{\*\}\) achieves\+1\.4%\+1\.4\\%to\+4\.0%\+4\.0\\%higher throughput than v1 \(MB\) across all seven models \(Table[6](https://arxiv.org/html/2606.00516#A1.T6)\), confirming that exclusive batching provides consistent gains at long context\.

Table 6:End\-to\-end throughput \(tok/s\) at 128K input / 64 output tokens on RTX PRO 6000 \(96 GB\)\. Concurrency is the maximum that fits in GPU memory\.Stress test beyond practical settings\.For completeness, we additionally probe how the linear model behaves outside its intended operating regime by forcingB=131,072B=131\{,\}072so that a single iteration processes the entire 128K input without chunking\. This far exceeds typical deployment configurations, yet even here the linear fit retainsR2≥0\.943R^\{2\}\\geq 0\.943across all seven models \(Table[7](https://arxiv.org/html/2606.00516#A1.T7)\), indicating that the approximation degrades gracefully rather than breaking down\. Figure[7](https://arxiv.org/html/2606.00516#A1.F7)provides a visual summary spanning both regimes: the left panel shows the measured prefill time curves, and the right panel traces howR2R^\{2\}degrades smoothly as the fit range is extended from the practical budget to the full 128K range\.

Table 7:Linear\-modelR2R^\{2\}under the stress settingB=131,072B=131\{,\}072\(one iteration on the full 128K input\)\.![Refer to caption](https://arxiv.org/html/2606.00516v1/figs/prefill_linearity_all_models.png)Figure 7:Linear\-model validation across seven models on RTX PRO 6000 Blackwell\.\(Left\)Measured prefill iteration time vs\. context lengthLL\(up to 128K tokens\); the shaded band marks the typical per\-iteration token budget \(≤8​K\\leq 8\\text\{K\}\)\.\(Right\)Linear\-fitR2R^\{2\}as a function of the fit\-range upper bound\. Within the practical budget \(≤16​K\\leq 16\\text\{K\}\) all models retainR2\>0\.97R^\{2\}\>0\.97; even when stretched to 128K the approximation degrades gracefully, stayingR2≥0\.94R^\{2\}\\geq 0\.94\.
### A\.2Additional Marginal Cost and Kernel Breakdown Results

Section[1](https://arxiv.org/html/2606.00516#S1)presents marginal cost and kernel breakdown analyses using Qwen3\-4B\. Here we extend these to Gemma\-3\-1B\-IT, Qwen3\-8B, and Qwen3\-30B\-A3B \(a Mixture\-of\-Experts variant\) on the same two\-GPU setup \(RTX PRO 6000 and H200\), examining how model scale and architecture affect the prefill–decode interference phenomenon\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x8.png)Figure 8:Execution time vs\. total token count at various decode ratios for Gemma\-3\-1B\-IT \(left\), Qwen3\-8B \(middle\), and Qwen3\-30B\-A3B \(right\) on RTX PRO 6000 \(top\) and H200 \(bottom\)\.Marginal cost\.Figure[8](https://arxiv.org/html/2606.00516#A1.F8)reports execution time vs\. total token count at various decode ratios for each model on both GPUs\. For the two Qwen models, RTX PRO 6000 results are consistent with the Qwen3\-4B findings in the main text: mixed\-batch execution exceeds pure decode at moderate decode ratios\. On H200, the mixed\-batch curves remain between the pure\-prefill and pure\-decode baselines across most operating points, with crossover only at high decode ratios\. The interference effect intensifies with model size—Qwen3\-30B\-A3B exhibits a wider gap between mixed\-batch and pure\-decode costs than Qwen3\-8B on RTX PRO 6000, reflecting increased memory\-bandwidth pressure from larger \(or more numerous\) weight matrices\.

For Gemma\-3\-1B\-IT, however, the mixed\-batch curves remain strictly between the pure\-prefill and pure\-decode baselines on both GPUs, and no crossover is observed\. We attribute this to its lightweight architecture: a 1B\-parameter dense model has fewer weight matrices and attention heads, so even the RTX PRO 6000’s 1\.792 TB/s bandwidth is sufficient to serve mixed batches without severe contention\. This indicates that the interference effect is*model\-architecture\-dependent*—it emerges when the model’s memory footprint is large enough to saturate the available bandwidth\.

Kernel breakdown\.Figure[9](https://arxiv.org/html/2606.00516#A1.F9)\(\(a\)–\(c\)\) decomposes the iteration time into kernel\-level components\. We highlight three observations:

*\(i\) GEMM time is insensitive to batch composition\.*Across all three models and both GPUs, GEMM time remains largely invariant to decode ratio, confirming that the linear projection cost depends primarily on total token count rather than the prefill–decode composition\.

*\(ii\) Attention is the primary source of composition sensitivity for Qwen models\.*For both Qwen3\-8B and Qwen3\-30B\-A3B, as well as Qwen3\-4B in the main text, the Attention component accounts for the majority of the latency increase as decode ratio grows, and exhibits the same GPU\-dependent crossover behavior: mixed\-batch Attention becomes slower than pure\-decode Attention at a much lower decode ratio on the RTX PRO 6000 than on the H200\.

*\(iii\) Model\-specific kernel characteristics\.*For Qwen3\-30B\-A3B, MoE routing and expert computation dominate kernel time but remain insensitive to batch composition, behaving similarly to GEMM\. For Gemma\-3\-1B\-IT, the “Other” category \(elementwise operations, normalization, memory management\) grows noticeably with decode ratio on RTX PRO 6000: GEMM and Attention complete quickly for a 1B\-parameter model, so per\-token overheads \(KV\-cache management, kernel launch latency, synchronization\)—which scale with decode tokens—become proportionally significant\.

Taken together, these results show that the prefill–decode interference identified in Section[1](https://arxiv.org/html/2606.00516#S1)is robust across the Qwen model family at different scales, including MoE architectures\. The absence of interference for Gemma\-3\-1B\-IT further supports our bandwidth\-based explanation: the effect manifests when the model’s memory footprint is sufficient to create bandwidth contention, making it most relevant for the medium\-to\-large models that dominate production deployments\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x9.png)\(a\)Gemma\-3\-1B\-IT
![Refer to caption](https://arxiv.org/html/2606.00516v1/x10.png)\(b\)Qwen3\-8B
![Refer to caption](https://arxiv.org/html/2606.00516v1/x11.png)\(c\)Qwen3\-30B\-A3B

Figure 9:Kernel time breakdown by decode ratio on RTX PRO 6000 \(top\) and H200 \(bottom\) for \(a\) Gemma\-3\-1B\-IT, \(b\) Qwen3\-8B, and \(c\) Qwen3\-30B\-A3B\.![Refer to caption](https://arxiv.org/html/2606.00516v1/x12.png)Figure 10:Effect of decode context length on \(left\) iteration time, \(middle\) GEMM time, and \(right\) attention time for Qwen3\-4B on NVIDIA H200 \(total batch = 1024 tokens\)\.Effect of decode context length\.Figures[1](https://arxiv.org/html/2606.00516#S1.F1)and[2](https://arxiv.org/html/2606.00516#S1.F2)in the main text use ctx=16=16\. Figure[10](https://arxiv.org/html/2606.00516#A1.F10)extends those results by varying ctx to examine how KV cache length affects iteration time for Qwen3\-4B on H200 with a fixed total batch size of 1024 tokens\. Here,*decode context length*\(ctx\) denotes the number of tokens already accumulated in each decode request’s KV cache: at ctx=c=c, every decode token must streamcckey–value pairs from GPU memory during attention\.

Three patterns emerge\. First, Execution time \(left\) increases with ctx at all decode ratios, with the five curves ordered by ctx and diverging as decode percentage grows; at 0% decode \(pure prefill\) all curves converge because no KV cache is accessed\. Second, GEMM time \(middle\) is nearly invariant to ctx—the curves remain tightly clustered across the full decode range—confirming that linear\-projection cost depends only on the number of tokens, not on KV cache size\. Third, attention time \(right\) is the sole driver of context\-length sensitivity: it grows monotonically with ctx and accounts for almost all of the vertical spread visible in the execution\-time panel, with ctx=512=512reaching∼\\sim26 ms at 80% decode versus∼\\sim10 ms for ctx=16=16\.

Taken together, increasing ctx uniformly lifts the execution\-time and attention\-time curves without changing their shape or crossover structure\. This validates the claim in Section[1](https://arxiv.org/html/2606.00516#S1)that the qualitative conclusions of Figures[1](https://arxiv.org/html/2606.00516#S1.F1)–[2](https://arxiv.org/html/2606.00516#S1.F2)generalize across context lengths\.

### A\.3Convexity Analysis of Mixed\-Batch Cost

This appendix derives the*interference convexity index*κ\\kappareferenced in Section[3\.2](https://arxiv.org/html/2606.00516#S3.SS2)and shows that two qualitative predictions used in the evaluation—\(i\) more negativeκ\\kappafavors EB, and \(ii\) EB’s advantage peaks at intermediate decode ratios—follow as direct algebraic consequences of the crossover condition \([7](https://arxiv.org/html/2606.00516#S3.E7)\)\.

Setup\.Let𝒦​\(r\)\\mathcal\{K\}\(r\)denote the per\-iteration kernel time of a mixed batch of fixed total token count, expressed as a function of the decode ratior∈\[0,1\]r\\in\[0,1\]\. Let

𝒦¯​\(r\)=\(1−r\)​𝒦​\(0\)\+r​𝒦​\(1\)\\bar\{\\mathcal\{K\}\}\(r\)\\;=\\;\(1\-r\)\\,\\mathcal\{K\}\(0\)\+r\\,\\mathcal\{K\}\(1\)be the linear interpolation between pure\-prefill \(r=0r=0\) and pure\-decode \(r=1r=1\) costs\. The marginal\-cost gap on the LHS of \([7](https://arxiv.org/html/2606.00516#S3.E7)\) satisfies

βMBe​\(r\)−βEBw​\(r\)∝𝒦​\(r\)−𝒦¯​\(r\),\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(r\)\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}\(r\)\\;\\propto\\;\\mathcal\{K\}\(r\)\-\\bar\{\\mathcal\{K\}\}\(r\),\(9\)becauseβMBe​\(r\)\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(r\)is the per\-token cost of the mixed batch at ratiorr\(proportional to𝒦​\(r\)\\mathcal\{K\}\(r\)\), whileβEBw​\(r\)\\beta\_\{\\mathrm\{EB\}\}^\{w\}\(r\)is the workload\-weighted average of the exclusive prefill and decode per\-token costs \(proportional to𝒦¯​\(r\)\\bar\{\\mathcal\{K\}\}\(r\)\)\.

Quadratic fit andκ\\kappa\.Empirically,𝒦​\(r\)\\mathcal\{K\}\(r\)is well approximated by a quadratic𝒦​\(r\)=c0\+c1​r\+c2​r2\\mathcal\{K\}\(r\)=c\_\{0\}\+c\_\{1\}r\+c\_\{2\}r^\{2\}\. Direct calculation gives

𝒦​\(r\)−𝒦¯​\(r\)=c2​r2−c2​r=−c2​r​\(1−r\)\.\\mathcal\{K\}\(r\)\-\\bar\{\\mathcal\{K\}\}\(r\)\\;=\\;c\_\{2\}r^\{2\}\-c\_\{2\}r\\;=\\;\-c\_\{2\}\\,r\(1\-r\)\.\(10\)The deviation𝒦​\(r\)−𝒦¯​\(r\)\\mathcal\{K\}\(r\)\-\\bar\{\\mathcal\{K\}\}\(r\)is therefore controlled by the single coefficientc2c\_\{2\}\(the second derivative𝒦′′/2\\mathcal\{K\}^\{\\prime\\prime\}/2\)\. To obtain a dimensionless, hardware\-comparable scalar, we normalizec2c\_\{2\}by the pure\-decode cost𝒦​\(1\)=c0\+c1\+c2\\mathcal\{K\}\(1\)=c\_\{0\}\+c\_\{1\}\+c\_\{2\}and define the*interference convexity index*

κ=2​c2c0\+c1\+c2\.\\kappa\\;=\\;\\frac\{2c\_\{2\}\}\{c\_\{0\}\+c\_\{1\}\+c\_\{2\}\}\.\(11\)
Consequences\.Combining \([9](https://arxiv.org/html/2606.00516#A1.E9)\), \([10](https://arxiv.org/html/2606.00516#A1.E10)\), and \([11](https://arxiv.org/html/2606.00516#A1.E11)\) \(rearranged asc2=κ​𝒦​\(1\)/2c\_\{2\}=\\kappa\\mathcal\{K\}\(1\)/2\) yields

βMBe−βEBw∝−κ​𝒦​\(1\)2​r​\(1−r\),\\beta\_\{\\mathrm\{MB\}\}^\{e\}\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}\\;\\propto\\;\-\\,\\frac\{\\kappa\\,\\mathcal\{K\}\(1\)\}\{2\}\\,r\(1\-r\),from which two predictions follow:

*\(i\) More negativeκ\\kappafavors EB\.*Since𝒦​\(1\)\>0\\mathcal\{K\}\(1\)\>0andr​\(1−r\)≥0r\(1\-r\)\\geq 0on\[0,1\]\[0,1\], a more negativeκ\\kappaenlarges the LHS of \([7](https://arxiv.org/html/2606.00516#S3.E7)\), so the inequality fails at smaller mismatches and EB outperforms MB more easily\. Empirically,κ\\kappais negative on most GPUs \(the kernel\-time curve is concave\); its magnitude varies substantially across hardware\. On Qwen3\-4B with 4096 tokens we measureκ=−11\.6\\kappa=\-11\.6on the RTX PRO 6000 versusκ=−0\.69\\kappa=\-0\.69on the H200, confirming that bandwidth\-constrained hardware exhibits substantially stronger prefill–decode interference\.

*\(ii\) EB’s advantage peaks at intermediate decode ratios\.*The factorr​\(1−r\)r\(1\-r\)in \([10](https://arxiv.org/html/2606.00516#A1.E10)\) is maximized atr=1/2r=1/2and vanishes at the extremesr∈\{0,1\}r\\in\\\{0,1\\\}\. Consequently, even on hardware with strongly negativeκ\\kappa, EB yields its largest gains over MB on workloads operating at moderate decode ratios; prefill\-heavy \(r→0r\\to 0\) and decode\-heavy \(r→1r\\to 1\) workloads see diminishing returns\. This prediction is verified empirically in Section[4\.3](https://arxiv.org/html/2606.00516#S4.SS3), where on RTX PRO 6000 with Qwen3\-8B, EB\(k^∗\\hat\{k\}^\{\*\}\) improves over v1 by \+15\.3% on ShareGPT and \+11\.3% on WildChat \(bothr≈0\.5r\\approx 0\.5–0\.70\.7\) but only \+3\.7% on LongBench \(r≈0\.004r\\approx 0\.004\) and \+1\.4% on NuminaMath \(r\>0\.85r\>0\.85\)\.

### A\.4Empirical Calibration ofβMBe​\(r\)\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(r\)for EB\+

EB\+’s online switching rule in Eq\. \([8](https://arxiv.org/html/2606.00516#S3.E8)\) requires runtime evaluation ofβMBe​\(r^\)\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(\\hat\{r\}\), the mixed\-batch per\-token cost as a function of decode ratio\. UnlikeβEBw\\beta\_\{\\mathrm\{EB\}\}^\{w\}, which derives analytically from the prefill/decode kernel coefficients\(αp,βp,αd,βd\)\(\\alpha\_\{p\},\\beta\_\{p\},\\alpha\_\{d\},\\beta\_\{d\}\)already calibrated for EB\(k^∗\\hat\{k\}^\{\*\}\),βMBe​\(r\)\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(r\)captures prefill↔\\leftrightarrowdecode interference inside a mixed batch and must be measured empirically\.

When the interference convexityκ\\kappahas been profiled \(Appendix[A\.3](https://arxiv.org/html/2606.00516#A1.SS3)\), the coefficients\(c0,c1,c2\)\(c\_\{0\},c\_\{1\},c\_\{2\}\)ofβMBe​\(r\)=c0\+c1​r\+c2​r2\\beta\_\{\\mathrm\{MB\}\}^\{e\}\(r\)=c\_\{0\}\+c\_\{1\}r\+c\_\{2\}r^\{2\}follow directly fromc2=κ​𝒦​\(1\)/2c\_\{2\}=\\kappa\\,\\mathcal\{K\}\(1\)/2\. Whenκ\\kappais not available,\(c0,c1,c2\)\(c\_\{0\},c\_\{1\},c\_\{2\}\)can be obtained from a small set of synthetic v1 runs at varying decode ratios: fit1/throughputv​1​\(r\)1/\\text\{throughput\}\_\{v1\}\(r\)to a quadratic inrr\. On H200 with Qwen3\-8B, fitting v1 grid data atr∈\{0\.11,0\.50,0\.89\}r\\in\\\{0\.11,0\.50,0\.89\\\}yields coefficients in the10−5​s/tok10^\{\-5\}\\ \\mathrm\{s/tok\}range\. The priority marginδ\\deltain Eq\. \([8](https://arxiv.org/html/2606.00516#S3.E8)\) should match the observed\|LHS−RHS\|\|\\mathrm\{LHS\}\-\\mathrm\{RHS\}\|magnitude\. For the calibration above, typical\|LHS−RHS\|\|\\mathrm\{LHS\}\-\\mathrm\{RHS\}\|atN^obs∈\[200,2000\]\\hat\{N\}\_\{\\mathrm\{obs\}\}\\in\[200,2000\]is in the10−6−10−510^\{\-6\}\\\!\-\\\!10^\{\-5\}range; we setδ=10−5\\delta=10^\{\-5\}, yielding stable MB↔\\leftrightarrowEB switching across workload drift \(verified on non\-stationary workloads in Table[3](https://arxiv.org/html/2606.00516#S4.T3), bottom\)\. With this calibration, EB\+on Qwen3\-8B distribution\-shift correctly switches MB→\\toEB during decode\-rich phases and back to MB as concurrency drops near experiment end\.

## Appendix BProof of Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1): Baseline Threshold under CFR

We derive the optimal switching threshold under constant failure rate \(CFR\), where the hazard rate ish​\(t\)=p0h\(t\)=p\_\{0\}\. This corresponds to geometric output length distributions, with mean output lengthμO=1/p0\\mu\_\{O\}=1/p\_\{0\}\.

### B\.1Stochastic Model and Fluid Approximation

Exact stochastic dynamics\.LetNtN\_\{t\}denote the number of requests still decoding afterttiterations\. The process\{Nt\}t≥0\\\{N\_\{t\}\\\}\_\{t\\geq 0\}is a pure death process: at each iteration, each of theNtN\_\{t\}active requests independently completes with probabilityp0p\_\{0\}\. Thus,

Nt\+1=Nt−Dt,Dt∣Nt∼Binomial​\(Nt,p0\),N\_\{t\+1\}=N\_\{t\}\-D\_\{t\},\\quad D\_\{t\}\\mid N\_\{t\}\\sim\\mathrm\{Binomial\}\(N\_\{t\},p\_\{0\}\),whereDtD\_\{t\}is the number of completions in iterationtt\.

Fluid approximation\.For large batch sizes, we approximate the discrete stochastic process by a continuous deterministic flow\. Define the scaled processN¯t\(N\)=Nt/N\\bar\{N\}\_\{t\}^\{\(N\)\}=N\_\{t\}/N\. By the law of large numbers for density\-dependent population processes\(Ethier and Kurtz,[1986](https://arxiv.org/html/2606.00516#bib.bib1); Kurtz,[1970](https://arxiv.org/html/2606.00516#bib.bib2)\), asN→∞N\\to\\infty, the scaled process converges uniformly in probability to the solution of the ordinary differential equation:

d​nd​t=−p0⋅n​\(t\),n​\(0\)=1,\\frac\{dn\}\{dt\}=\-p\_\{0\}\\cdot n\(t\),\\quad n\(0\)=1,\(12\)wherettdenotes the*iteration index*\(not wall\-clock time\), consistent with the hazard\-rate argument used in the main text\. Specifically, for any fixedT\>0T\>0,

sup0≤t≤T\|N¯t\(N\)−n​\(t\)\|→𝑃0as​N→∞\.\\sup\_\{0\\leq t\\leq T\}\\left\|\\bar\{N\}\_\{t\}^\{\(N\)\}\-n\(t\)\\right\|\\xrightarrow\{P\}0\\quad\\text\{as \}N\\to\\infty\.The approximation error isO​\(1/N\)O\(1/\\sqrt\{N\}\), arising from the central limit theorem for the cumulative martingale differences\.

The ODE \([12](https://arxiv.org/html/2606.00516#A2.E12)\) has solution:

n​\(t\)=e−p0​t\.n\(t\)=e^\{\-p\_\{0\}t\}\.The exact discrete trajectory is\(1−p0\)t\(1\-p\_\{0\}\)^\{t\}; the continuous\-time fluid limite−p0​te^\{\-p\_\{0\}t\}used here introduces an additionalO​\(p0\)O\(p\_\{0\}\)relative error \(the gap between−p0\-p\_\{0\}andln⁡\(1−p0\)\\ln\(1\-p\_\{0\}\)\), enumerated as Source 4 in the error analysis below\.

### B\.2Decode Phase Duration

A phase switch occurs whenk=θ​Nk=\\theta Nrequests have completed, i\.e\., when the batch size drops toN​\(1−θ\)N\(1\-\\theta\)\. Under the fluid approximation, this occurs at iteration indext∗t^\{\*\}satisfyingn​\(t∗\)=1−θn\(t^\{\*\}\)=1\-\\theta:

e−p0​t∗=1−θ⟹t∗=−ln⁡\(1−θ\)p0=ζ​\(θ\)p0,e^\{\-p\_\{0\}t^\{\*\}\}=1\-\\theta\\quad\\Longrightarrow\\quad t^\{\*\}=\\frac\{\-\\ln\(1\-\\theta\)\}\{p\_\{0\}\}=\\frac\{\\zeta\(\\theta\)\}\{p\_\{0\}\},where we defineζ​\(θ\)≜−ln⁡\(1−θ\)\\zeta\(\\theta\)\\triangleq\-\\ln\(1\-\\theta\)for notational convenience\.

Wall\-clock decode time\.The wall\-clock time for the decode phase is the sum of iteration times\. Each iteration with batch sizenntakes timetd​\(n\)=αd\+βd​nt\_\{d\}\(n\)=\\alpha\_\{d\}\+\\beta\_\{d\}n, whereαd\\alpha\_\{d\}is the fixed overhead andβd\\beta\_\{d\}is the per\-request cost\. Under the fluid approximation, the expected decode time is:

𝔼​\[Td\]=∫0t∗td​\(N⋅n​\(t\)\)​𝑑t=∫0t∗\(αd\+βd​N​e−p0​t\)​𝑑t\.\\mathbb\{E\}\[T\_\{d\}\]=\\int\_\{0\}^\{t^\{\*\}\}t\_\{d\}\(N\\cdot n\(t\)\)\\,dt=\\int\_\{0\}^\{t^\{\*\}\}\\left\(\\alpha\_\{d\}\+\\beta\_\{d\}Ne^\{\-p\_\{0\}t\}\\right\)dt\.\(13\)
Evaluating the integral:

𝔼​\[Td\]\\displaystyle\\mathbb\{E\}\[T\_\{d\}\]=αd​t∗\+βd​N​∫0t∗e−p0​t​𝑑t\\displaystyle=\\alpha\_\{d\}t^\{\*\}\+\\beta\_\{d\}N\\int\_\{0\}^\{t^\{\*\}\}e^\{\-p\_\{0\}t\}\\,dt=αd⋅ζp0\+βd​N⋅1−e−p0​t∗p0\\displaystyle=\\alpha\_\{d\}\\cdot\\frac\{\\zeta\}\{p\_\{0\}\}\+\\beta\_\{d\}N\\cdot\\frac\{1\-e^\{\-p\_\{0\}t^\{\*\}\}\}\{p\_\{0\}\}=αd​ζp0\+βd​N​θp0,\\displaystyle=\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\},where the last equality uses1−e−p0​t∗=θ1\-e^\{\-p\_\{0\}t^\{\*\}\}=\\theta\.

### B\.3Approximation Error Analysis

The fluid approximation introduces several sources of error relative to the exact stochastic model\. We characterize each below and argue that they do not materially affect the optimal threshold\.

Source 1: Stochastic fluctuation\.The fluid model replaces the random completionsDt∼Binomial​\(Nt,p0\)D\_\{t\}\\sim\\mathrm\{Binomial\}\(N\_\{t\},p\_\{0\}\)with their conditional expectationp0​Ntp\_\{0\}N\_\{t\}\. By Kurtz’s theorem\(Kurtz,[1970](https://arxiv.org/html/2606.00516#bib.bib2)\), the scaled batch sizeNt/NN\_\{t\}/Nconverges uniformly to the ODE solutionn​\(t\)=e−p0​tn\(t\)=e^\{\-p\_\{0\}t\}, with fluctuations of orderO​\(1/N\)O\(1/\\sqrt\{N\}\)\. Since practical LLM serving systems operate at batch sizesN≥64N\\geq 64, this contributes at most a few percent relative error to the decode time estimate\.

Source 2: Discretization and stopping\-time mismatch\.The fluid model assumes the batch size decreases continuously and stops exactly whenkkrequests have completed\. In the actual discrete system, this cannot happen for two reasons: \(i\) iterations are discrete, so the system can only stop at integer iteration indicesM=⌈t∗⌉M=\\lceil t^\{\*\}\\rceil, not at the continuous stopping pointt∗t^\{\*\}; and \(ii\) multiple requests may complete within a single iteration, causing the cumulative completion count to*overshoot*the thresholdkkrather than hitting it exactly\. Both effects share the same root cause—the discrete system cannot stop at an arbitrary point—and produce a bounded relative error that does not vanish withNN\. The bias is more pronounced at smallθ\\theta, because the batch size at the switching point is still large \(≈N​\(1−θ\)\{\\approx\}\\,N\(1\{\-\}\\theta\)\), leading to larger per\-iteration completion variance and thus larger overshoot relative tokk\.

Source 3: Integer rounding\.The continuous thresholdθ​N\\theta Nis rounded tok=⌊θ​N⌋k=\\lfloor\\theta N\\rfloorin practice\. This contributes anO​\(1/N\)O\(1/N\)relative error and is negligible forN≥32N\\geq 32\.

Source 4: Continuous\-time fluid limit\.The ODEn˙=−p0​n\\dot\{n\}=\-p\_\{0\}nhas solutionn​\(t\)=e−p0​tn\(t\)=e^\{\-p\_\{0\}t\}, whereas the exact discrete trajectory is\(1−p0\)t=et​ln⁡\(1−p0\)\(1\-p\_\{0\}\)^\{t\}=e^\{t\\ln\(1\-p\_\{0\}\)\}\. The gap−p0−ln⁡\(1−p0\)=−p02/2\+O​\(p03\)\-p\_\{0\}\-\\ln\(1\-p\_\{0\}\)=\-p\_\{0\}^\{2\}/2\+O\(p\_\{0\}^\{3\}\)contributes an additionalO​\(p0\)O\(p\_\{0\}\)relative error to𝔼​\[Td\]\\mathbb\{E\}\[T\_\{d\}\]and toθ0\\theta\_\{0\}\. This source is independent ofNN, deterministic, and would be absorbed exactly by reparametrizingp0→−ln⁡\(1−p0\)p\_\{0\}\\to\-\\ln\(1\-p\_\{0\}\)in \([15](https://arxiv.org/html/2606.00516#A2.E15)\)\. For our operating regimep0∈\[0\.003,0\.01\]p\_\{0\}\\in\[0\.003,0\.01\], the numerical impact onθ0\\theta\_\{0\}is below1%1\\%and is dominated by theO​\(1/N\)O\(1/\\sqrt\{N\}\)stochastic fluctuation \(Source 1\)\.

Table 8:Summary of approximation error sources\.
### B\.4Expected Prefill Time

For the prefill phase, each of thek=θ​Nk=\\theta Ncompleted requests generates a new request\. The prefill time is:

𝔼​\[Tp\]=αp\+βp⋅\(θ​N\)⋅μL=αp\+βp​N​μL​θ,\\mathbb\{E\}\[T\_\{p\}\]=\\alpha\_\{p\}\+\\beta\_\{p\}\\cdot\(\\theta N\)\\cdot\\mu\_\{L\}=\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta,whereμL\\mu\_\{L\}is the mean input length\.

### B\.5Complete Cycle Time

The total cycle time combines decode and prefill phases:

𝒯​\(θ\)=𝔼​\[Td\]\+𝔼​\[Tp\]=αd​ζp0\+βd​N​θp0\+αp\+βp​N​μL​θ\.\\mathcal\{T\}\(\\theta\)=\\mathbb\{E\}\[T\_\{d\}\]\+\\mathbb\{E\}\[T\_\{p\}\]=\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta\.
Grouping terms:

𝒯​\(θ\)=αd​ζ​\(θ\)p0\+αp\+\(βd​Np0\+βp​N​μL\)​θ\.\\mathcal\{T\}\(\\theta\)=\\frac\{\\alpha\_\{d\}\\zeta\(\\theta\)\}\{p\_\{0\}\}\+\\alpha\_\{p\}\+\\left\(\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}\}\+\\beta\_\{p\}N\\mu\_\{L\}\\right\)\\theta\.

### B\.6Throughput Optimization

The normalized throughput is:

R​\(θ\)=θ𝒯​\(θ\)\.R\(\\theta\)=\\frac\{\\theta\}\{\\mathcal\{T\}\(\\theta\)\}\.
To maximizeRR, we take the derivative and set it to zero:

d​Rd​θ=𝒯​\(θ\)−θ​𝒯′​\(θ\)𝒯​\(θ\)2=0\.\\frac\{dR\}\{d\\theta\}=\\frac\{\\mathcal\{T\}\(\\theta\)\-\\theta\\mathcal\{T\}^\{\\prime\}\(\\theta\)\}\{\\mathcal\{T\}\(\\theta\)^\{2\}\}=0\.
This yields the first\-order optimality condition:

𝒯​\(θ\)=θ​𝒯′​\(θ\)\.\\mathcal\{T\}\(\\theta\)=\\theta\\mathcal\{T\}^\{\\prime\}\(\\theta\)\.\(14\)
Computing𝒯′​\(θ\)\\mathcal\{T\}^\{\\prime\}\(\\theta\)\.Sinceζ​\(θ\)=−ln⁡\(1−θ\)\\zeta\(\\theta\)=\-\\ln\(1\-\\theta\), we haveζ′​\(θ\)=11−θ\\zeta^\{\\prime\}\(\\theta\)=\\frac\{1\}\{1\-\\theta\}\. Thus:

𝒯′​\(θ\)=αdp0​\(1−θ\)\+βd​Np0\+βp​N​μL\.\\mathcal\{T\}^\{\\prime\}\(\\theta\)=\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\)\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}\}\+\\beta\_\{p\}N\\mu\_\{L\}\.
Applying the optimality condition\.Substituting into \([14](https://arxiv.org/html/2606.00516#A2.E14)\):

αd​ζp0\+βd​N​θp0\+αp\+βp​N​μL​θ=θ​\[αdp0​\(1−θ\)\+βd​Np0\+βp​N​μL\]\.\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta=\\theta\\left\[\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\)\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}\}\+\\beta\_\{p\}N\\mu\_\{L\}\\right\]\.
Expanding the right\-hand side:

αd​ζp0\+βd​N​θp0\+αp\+βp​N​μL​θ=αd​θp0​\(1−θ\)\+βd​N​θp0\+βp​N​μL​θ\.\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta=\\frac\{\\alpha\_\{d\}\\theta\}\{p\_\{0\}\(1\-\\theta\)\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta\.
The termsβd​N​θp0\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}andβp​N​μL​θ\\beta\_\{p\}N\\mu\_\{L\}\\thetaappear on both sides and cancel:

αd​ζp0\+αp=αd​θp0​\(1−θ\)\.\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\alpha\_\{p\}=\\frac\{\\alpha\_\{d\}\\theta\}\{p\_\{0\}\(1\-\\theta\)\}\.
Multiplying both sides byp0αd\\frac\{p\_\{0\}\}\{\\alpha\_\{d\}\}:

ζ\+p0​αpαd=θ1−θ\.\\zeta\+\\frac\{p\_\{0\}\\alpha\_\{p\}\}\{\\alpha\_\{d\}\}=\\frac\{\\theta\}\{1\-\\theta\}\.
Rearranging and substitutingζ=−ln⁡\(1−θ\)\\zeta=\-\\ln\(1\-\\theta\):

θ01−θ0\+ln⁡\(1−θ0\)=p0​αpαd\\boxed\{\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\+\\ln\(1\-\\theta\_\{0\}\)=\\frac\{p\_\{0\}\\alpha\_\{p\}\}\{\\alpha\_\{d\}\}\}\(15\)

### B\.7Existence and Uniqueness

###### Lemma B\.2\.

Equation \([15](https://arxiv.org/html/2606.00516#A2.E15)\) has a unique solutionθ0∈\(0,1\)\\theta\_\{0\}\\in\(0,1\)for anyγ=p0​αp/αd\>0\\gamma=p\_\{0\}\\alpha\_\{p\}/\\alpha\_\{d\}\>0\.

###### Proof\.

DefineΨ​\(θ\)=θ1−θ\+ln⁡\(1−θ\)\\Psi\(\\theta\)=\\frac\{\\theta\}\{1\-\\theta\}\+\\ln\(1\-\\theta\)\.

Step 1: Monotonicity\.Computing the derivative:

Ψ′​\(θ\)=\(1−θ\)−θ⋅\(−1\)\(1−θ\)2−11−θ=1\(1−θ\)2−11−θ=1−\(1−θ\)\(1−θ\)2=θ\(1−θ\)2\.\\Psi^\{\\prime\}\(\\theta\)=\\frac\{\(1\-\\theta\)\-\\theta\\cdot\(\-1\)\}\{\(1\-\\theta\)^\{2\}\}\-\\frac\{1\}\{1\-\\theta\}=\\frac\{1\}\{\(1\-\\theta\)^\{2\}\}\-\\frac\{1\}\{1\-\\theta\}=\\frac\{1\-\(1\-\\theta\)\}\{\(1\-\\theta\)^\{2\}\}=\\frac\{\\theta\}\{\(1\-\\theta\)^\{2\}\}\.
Sinceθ\>0\\theta\>0and\(1−θ\)2\>0\(1\-\\theta\)^\{2\}\>0forθ∈\(0,1\)\\theta\\in\(0,1\), we haveΨ′​\(θ\)\>0\\Psi^\{\\prime\}\(\\theta\)\>0, soΨ\\Psiis strictly increasing\.

Step 2: Boundary behavior\.

- •Asθ→0\+\\theta\\to 0^\{\+\}:θ1−θ→0\\frac\{\\theta\}\{1\-\\theta\}\\to 0andln⁡\(1−θ\)→0\\ln\(1\-\\theta\)\\to 0, solimθ→0\+Ψ​\(θ\)=0\\lim\_\{\\theta\\to 0^\{\+\}\}\\Psi\(\\theta\)=0\.
- •Asθ→1−\\theta\\to 1^\{\-\}:θ1−θ→\+∞\\frac\{\\theta\}\{1\-\\theta\}\\to\+\\inftyandln⁡\(1−θ\)→−∞\\ln\(1\-\\theta\)\\to\-\\infty, but the first term dominates \(it grows like1/\(1−θ\)1/\(1\-\\theta\)while the second decreases likeln⁡\(1−θ\)\\ln\(1\-\\theta\)\), solimθ→1−Ψ​\(θ\)=\+∞\\lim\_\{\\theta\\to 1^\{\-\}\}\\Psi\(\\theta\)=\+\\infty\.

Step 3: Existence and uniqueness\.SinceΨ\\Psiis continuous and strictly increasing on\(0,1\)\(0,1\)withΨ​\(0\+\)=0\\Psi\(0^\{\+\}\)=0andΨ​\(1−\)=\+∞\\Psi\(1^\{\-\}\)=\+\\infty, by the intermediate value theorem, for anyγ\>0\\gamma\>0, there exists a uniqueθ0∈\(0,1\)\\theta\_\{0\}\\in\(0,1\)such thatΨ​\(θ0\)=γ\\Psi\(\\theta\_\{0\}\)=\\gamma\. ∎

### B\.8Parameter Independence

###### Corollary B\.3\.

The optimal thresholdθ0\\theta\_\{0\}depends only on the ratiop0​αp/αdp\_\{0\}\\alpha\_\{p\}/\\alpha\_\{d\}and is independent ofβd\\beta\_\{d\},βp\\beta\_\{p\},NN, andμL\\mu\_\{L\}\.

###### Proof\.

This follows directly from the cancellation observed in the derivation: the terms involvingβd\\beta\_\{d\},βp\\beta\_\{p\},NN, andμL\\mu\_\{L\}appear linearly inθ\\thetaon both sides of the optimality condition and cancel exactly\.

Intuitively, this occurs because:

1. 1\.The per\-token decode costβd​N​θ/p0\\beta\_\{d\}N\\theta/p\_\{0\}depends on the*number*of requests that complete, which equalsθ​N\\theta Nregardless of*when*they complete\.
2. 2\.The prefill costβp​N​μL​θ\\beta\_\{p\}N\\mu\_\{L\}\\thetasimilarly depends only on the total tokens prefilled, not the timing\.

Thus, the marginal cost\-benefit trade\-off at the optimal threshold depends only on the timing\-related parameters\(αd,αp,p0\)\(\\alpha\_\{d\},\\alpha\_\{p\},p\_\{0\}\)\. ∎

### B\.9Second\-Order Condition

To confirmθ0\\theta\_\{0\}is a maximum \(not a minimum\), we verify the second\-order condition\.

###### Lemma B\.4\.

The critical pointθ0\\theta\_\{0\}satisfying \([15](https://arxiv.org/html/2606.00516#A2.E15)\) is a global maximum ofR​\(θ\)R\(\\theta\)\.

###### Proof\.

DefineΦ​\(θ\)=𝒯​\(θ\)−θ​𝒯′​\(θ\)\\Phi\(\\theta\)=\\mathcal\{T\}\(\\theta\)\-\\theta\\mathcal\{T\}^\{\\prime\}\(\\theta\)\. Then:

Φ′​\(θ\)=𝒯′​\(θ\)−𝒯′​\(θ\)−θ​𝒯′′​\(θ\)=−θ​𝒯′′​\(θ\)\.\\Phi^\{\\prime\}\(\\theta\)=\\mathcal\{T\}^\{\\prime\}\(\\theta\)\-\\mathcal\{T\}^\{\\prime\}\(\\theta\)\-\\theta\\mathcal\{T\}^\{\\prime\\prime\}\(\\theta\)=\-\\theta\\mathcal\{T\}^\{\\prime\\prime\}\(\\theta\)\.
Since𝒯′′​\(θ\)=αdp0​\(1−θ\)2\>0\\mathcal\{T\}^\{\\prime\\prime\}\(\\theta\)=\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\)^\{2\}\}\>0, we haveΦ′​\(θ\)<0\\Phi^\{\\prime\}\(\\theta\)<0forθ\>0\\theta\>0, soΦ\\Phiis strictly decreasing\.

Checking boundary values:

- •Φ​\(0\+\)=𝒯​\(0\)−0=αp\>0\\Phi\(0^\{\+\}\)=\\mathcal\{T\}\(0\)\-0=\\alpha\_\{p\}\>0\.
- •Φ​\(1−\)→−∞\\Phi\(1^\{\-\}\)\\to\-\\infty\(since𝒯′​\(θ\)→\+∞\\mathcal\{T\}^\{\\prime\}\(\\theta\)\\to\+\\inftyasθ→1−\\theta\\to 1^\{\-\}\)\.

ThusΦ\\Phihas a unique zero atθ0\\theta\_\{0\}, withΦ​\(θ\)\>0\\Phi\(\\theta\)\>0forθ<θ0\\theta<\\theta\_\{0\}andΦ​\(θ\)<0\\Phi\(\\theta\)<0forθ\>θ0\\theta\>\\theta\_\{0\}\. Sinced​Rd​θ=Φ​\(θ\)/𝒯​\(θ\)2\\frac\{dR\}\{d\\theta\}=\\Phi\(\\theta\)/\\mathcal\{T\}\(\\theta\)^\{2\}, this confirmsRRis increasing forθ<θ0\\theta<\\theta\_\{0\}and decreasing forθ\>θ0\\theta\>\\theta\_\{0\}, soθ0\\theta\_\{0\}is the unique global maximum\. ∎

This completes the proof of Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1)\. ∎

## Appendix CProof of Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2): IFR Threshold Correction

We now extend the analysis to the increasing failure rate \(IFR\) case with linear hazardh​\(t\)=p0\+η​th\(t\)=p\_\{0\}\+\\eta t, whereη≥0\\eta\\geq 0captures the degree of IFR behavior\. The proof proceeds by: \(1\) deriving the modified batch dynamics, \(2\) computing cycle time as a perturbation expansion inη\\eta, and \(3\) applying the implicit function theorem to obtain the threshold correction\.

Scope of the analysis\.The argument below operates on the fluid \(deterministic\) model, treating the decode batch as if every request evolved underh​\(t\)=p0\+η​th\(t\)=p\_\{0\}\+\\eta tmeasured from the start of the current phase\. Under CFR \(memoryless\), this is exact because hazards are state\-independent; under IFR, continuing requests carry heterogeneous ages across cycles, so successive cycles are no longer i\.i\.d\. and the renewal\-reward foundation does not apply directly\. The fluid argument below does not require i\.i\.d\. cycles; the finite\-NNstochastic analog would require a mixing\-time argument for the age\-dependent Markov chain, which we leave to future work\. Empirically,Δ​θ\>0\\Delta\\theta\>0is observed on both synthetic IFR workloads \(Appendix[E\.3](https://arxiv.org/html/2606.00516#A5.SS3)\) and real LLM workloads \(Appendix[E\.4](https://arxiv.org/html/2606.00516#A5.SS4)\)\.

### C\.1Batch Size Evolution under IFR

Under the fluid approximation with linear hazard rate, the batch size evolves according to \(for algebraic convenience we usen​\(t\)n\(t\)here as the unnormalized batch size withn​\(0\)=Nn\(0\)=N, in contrast to the normalized conventionn​\(0\)=1n\(0\)=1adopted in Appendix[B](https://arxiv.org/html/2606.00516#A2); the two are related by an overall factor ofNN\):

d​nd​t=−h​\(t\)⋅n​\(t\)=−\(p0\+η​t\)​n​\(t\),n​\(0\)=N\.\\frac\{dn\}\{dt\}=\-h\(t\)\\cdot n\(t\)=\-\(p\_\{0\}\+\\eta t\)\\,n\(t\),\\quad n\(0\)=N\.
This is a separable ODE\. Separating variables:

d​nn=−\(p0\+η​t\)​d​t\.\\frac\{dn\}\{n\}=\-\(p\_\{0\}\+\\eta t\)\\,dt\.
Integrating both sides:

ln⁡n​\(t\)−ln⁡N=−p0​t−η​t22\.\\ln n\(t\)\-\\ln N=\-p\_\{0\}t\-\\frac\{\\eta t^\{2\}\}\{2\}\.
Exponentiating:

n​\(t\)=N​exp⁡\(−p0​t−η​t22\)\.n\(t\)=N\\exp\\left\(\-p\_\{0\}t\-\\frac\{\\eta t^\{2\}\}\{2\}\\right\)\.\(16\)
Verification\.Atη=0\\eta=0, this reduces ton​\(t\)=N​e−p0​tn\(t\)=Ne^\{\-p\_\{0\}t\}, recovering the CFR solution\.

### C\.2Decode Phase Duration

A phase switch occurs whenk=θ​Nk=\\theta Nrequests have completed, i\.e\., whenn​\(T\)=N​\(1−θ\)n\(T\)=N\(1\-\\theta\)\. Substituting into \([16](https://arxiv.org/html/2606.00516#A3.E16)\):

N​\(1−θ\)=N​exp⁡\(−p0​T−η​T22\)\.N\(1\-\\theta\)=N\\exp\\left\(\-p\_\{0\}T\-\\frac\{\\eta T^\{2\}\}\{2\}\\right\)\.
Taking logarithms:

p0​T\+η​T22=−ln⁡\(1−θ\)=ζ​\(θ\)\.p\_\{0\}T\+\\frac\{\\eta T^\{2\}\}\{2\}=\-\\ln\(1\-\\theta\)=\\zeta\(\\theta\)\.
This is a quadratic equation inTT\. Solving for the positive root:

T​\(θ\)=−p0\+p02\+2​η​ζ​\(θ\)η\.T\(\\theta\)=\\frac\{\-p\_\{0\}\+\\sqrt\{p\_\{0\}^\{2\}\+2\\eta\\zeta\(\\theta\)\}\}\{\\eta\}\.
Perturbation expansion\.For smallη\\eta, we expandTTto first order\. Defineξ=η​ζ/p02\\xi=\\eta\\zeta/p\_\{0\}^\{2\}\(dimensionless\)\. Then:

T=p0η​\(−1\+1\+2​ξ\)=p0η​\(−1\+1\+ξ−ξ22\+O​\(ξ3\)\)=p0​ξη​\(1−ξ2\+O​\(ξ2\)\)\.T=\\frac\{p\_\{0\}\}\{\\eta\}\\left\(\-1\+\\sqrt\{1\+2\\xi\}\\right\)=\\frac\{p\_\{0\}\}\{\\eta\}\\left\(\-1\+1\+\\xi\-\\frac\{\\xi^\{2\}\}\{2\}\+O\(\\xi^\{3\}\)\\right\)=\\frac\{p\_\{0\}\\xi\}\{\\eta\}\\left\(1\-\\frac\{\\xi\}\{2\}\+O\(\\xi^\{2\}\)\\right\)\.
Substitutingξ=η​ζ/p02\\xi=\\eta\\zeta/p\_\{0\}^\{2\}:

T​\(θ\)=ζp0−η​ζ22​p03\+O​\(η2\)≜T0\+η​T1\+O​\(η2\),T\(\\theta\)=\\frac\{\\zeta\}\{p\_\{0\}\}\-\\frac\{\\eta\\zeta^\{2\}\}\{2p\_\{0\}^\{3\}\}\+O\(\\eta^\{2\}\)\\triangleq T\_\{0\}\+\\eta T\_\{1\}\+O\(\\eta^\{2\}\),\(17\)whereT0=ζ/p0T\_\{0\}=\\zeta/p\_\{0\}is the CFR decode time andT1=−ζ2/\(2​p03\)<0T\_\{1\}=\-\\zeta^\{2\}/\(2p\_\{0\}^\{3\}\)<0is the first\-order correction \(negative because IFR reduces decode time\)\.

### C\.3Cumulative Batch Size

Define the cumulative batch size integral:

I​\(θ\)=∫0T​\(θ\)n​\(s\)​𝑑s=∫0TN​exp⁡\(−p0​s−η​s22\)​𝑑s\.I\(\\theta\)=\\int\_\{0\}^\{T\(\\theta\)\}n\(s\)\\,ds=\\int\_\{0\}^\{T\}N\\exp\\left\(\-p\_\{0\}s\-\\frac\{\\eta s^\{2\}\}\{2\}\\right\)ds\.
Zeroth\-order term\.Atη=0\\eta=0:

I0=N​∫0T0e−p0​s​𝑑s=Np0​\(1−e−p0​T0\)=Np0​\(1−e−ζ\)=Np0​\(1−\(1−θ\)\)=N​θp0\.I\_\{0\}=N\\int\_\{0\}^\{T\_\{0\}\}e^\{\-p\_\{0\}s\}ds=\\frac\{N\}\{p\_\{0\}\}\\left\(1\-e^\{\-p\_\{0\}T\_\{0\}\}\\right\)=\\frac\{N\}\{p\_\{0\}\}\\left\(1\-e^\{\-\\zeta\}\\right\)=\\frac\{N\}\{p\_\{0\}\}\(1\-\(1\-\\theta\)\)=\\frac\{N\\theta\}\{p\_\{0\}\}\.
First\-order correction\.To compute theO​\(η\)O\(\\eta\)correction, we use:

I​\(θ\)=∫0T0\+η​T1N​e−p0​s⋅e−η​s2/2​𝑑s\+O​\(η2\)\.I\(\\theta\)=\\int\_\{0\}^\{T\_\{0\}\+\\eta T\_\{1\}\}Ne^\{\-p\_\{0\}s\}\\cdot e^\{\-\\eta s^\{2\}/2\}ds\+O\(\\eta^\{2\}\)\.
Expandinge−η​s2/2≈1−η​s2/2e^\{\-\\eta s^\{2\}/2\}\\approx 1\-\\eta s^\{2\}/2and accounting for the change in upper limit:

I=I0\+η​\[N​e−p0​T0​T1−N2​∫0T0s2​e−p0​s​𝑑s\]\+O​\(η2\)\.I=I\_\{0\}\+\\eta\\left\[Ne^\{\-p\_\{0\}T\_\{0\}\}T\_\{1\}\-\\frac\{N\}\{2\}\\int\_\{0\}^\{T\_\{0\}\}s^\{2\}e^\{\-p\_\{0\}s\}ds\\right\]\+O\(\\eta^\{2\}\)\.
Computing the integral∫0T0s2​e−p0​s​𝑑s\\int\_\{0\}^\{T\_\{0\}\}s^\{2\}e^\{\-p\_\{0\}s\}dsby parts \(twice\) and simplifying:

I​\(θ\)=N​θp0\+η​Np03​\[\(1−θ\)​ζ−θ\]\+O​\(η2\)\.I\(\\theta\)=\\frac\{N\\theta\}\{p\_\{0\}\}\+\\frac\{\\eta N\}\{p\_\{0\}^\{3\}\}\\left\[\(1\-\\theta\)\\zeta\-\\theta\\right\]\+O\(\\eta^\{2\}\)\.\(18\)
Verification\.The correction term\(1−θ\)​ζ−θ=\(1−θ\)​\(−ln⁡\(1−θ\)\)−θ\(1\-\\theta\)\\zeta\-\\theta=\(1\-\\theta\)\(\-\\ln\(1\-\\theta\)\)\-\\theta\. For smallθ\\theta,ζ≈θ\+θ2/2\\zeta\\approx\\theta\+\\theta^\{2\}/2, so the correction≈\(1−θ\)​\(θ\+θ2/2\)−θ=−θ2/2\+O​\(θ3\)<0\\approx\(1\-\\theta\)\(\\theta\+\\theta^\{2\}/2\)\-\\theta=\-\\theta^\{2\}/2\+O\(\\theta^\{3\}\)<0, confirming that IFR reduces cumulative batch size\.

### C\.4Complete Cycle Time Expansion

The total cycle time is:

𝒯​\(θ\)=𝔼​\[Td\]\+𝔼​\[Tp\]=αd​T\+βd​I\+αp\+βp​N​μL​θ\.\\mathcal\{T\}\(\\theta\)=\\mathbb\{E\}\[T\_\{d\}\]\+\\mathbb\{E\}\[T\_\{p\}\]=\\alpha\_\{d\}T\+\\beta\_\{d\}I\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta\.
Substituting the expansions \([17](https://arxiv.org/html/2606.00516#A3.E17)\) and \([18](https://arxiv.org/html/2606.00516#A3.E18)\):

𝒯​\(θ\)\\displaystyle\\mathcal\{T\}\(\\theta\)=αd​\(ζp0−η​ζ22​p03\)\+βd​\(N​θp0\+η​Np03​\[\(1−θ\)​ζ−θ\]\)\+αp\+βp​N​μL​θ\+O​\(η2\)\\displaystyle=\\alpha\_\{d\}\\left\(\\frac\{\\zeta\}\{p\_\{0\}\}\-\\frac\{\\eta\\zeta^\{2\}\}\{2p\_\{0\}^\{3\}\}\\right\)\+\\beta\_\{d\}\\left\(\\frac\{N\\theta\}\{p\_\{0\}\}\+\\frac\{\\eta N\}\{p\_\{0\}^\{3\}\}\\left\[\(1\-\\theta\)\\zeta\-\\theta\\right\]\\right\)\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta\+O\(\\eta^\{2\}\)=αd​ζp0\+βd​N​θp0\+αp\+βp​N​μL​θ⏟𝒯0​\(θ\)\+η​\[−αd​ζ22​p03\+βd​Np03​\(\(1−θ\)​ζ−θ\)\]⏟𝒯1​\(θ\)\+O​\(η2\)\.\\displaystyle=\\underbrace\{\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}\}\+\\frac\{\\beta\_\{d\}N\\theta\}\{p\_\{0\}\}\+\\alpha\_\{p\}\+\\beta\_\{p\}N\\mu\_\{L\}\\theta\}\_\{\\mathcal\{T\}\_\{0\}\(\\theta\)\}\+\\eta\\underbrace\{\\left\[\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2p\_\{0\}^\{3\}\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}^\{3\}\}\\left\(\(1\-\\theta\)\\zeta\-\\theta\\right\)\\right\]\}\_\{\\mathcal\{T\}\_\{1\}\(\\theta\)\}\+O\(\\eta^\{2\}\)\.
We write this as:

𝒯​\(θ\)=𝒯0​\(θ\)\+η​𝒯1​\(θ\)\+O​\(η2\),\\mathcal\{T\}\(\\theta\)=\\mathcal\{T\}\_\{0\}\(\\theta\)\+\\eta\\,\\mathcal\{T\}\_\{1\}\(\\theta\)\+O\(\\eta^\{2\}\),\(19\)where𝒯0\\mathcal\{T\}\_\{0\}is the CFR cycle time and𝒯1​\(θ\)\\mathcal\{T\}\_\{1\}\(\\theta\)is the first\-order IFR correction\.

### C\.5Optimal Threshold via Implicit Function Theorem

Setup\.Define the residual function \(extendingΦ\\Phifrom the CFR proof to depend onη\\eta\):

Φ​\(θ,η\)=𝒯​\(θ,η\)−θ⋅∂𝒯∂θ​\(θ,η\)\.\\Phi\(\\theta,\\eta\)=\\mathcal\{T\}\(\\theta,\\eta\)\-\\theta\\cdot\\frac\{\\partial\\mathcal\{T\}\}\{\\partial\\theta\}\(\\theta,\\eta\)\.
The optimal thresholdθ∗​\(η\)\\theta^\{\*\}\(\\eta\)satisfies the first\-order conditionΦ​\(θ∗,η\)=0\\Phi\(\\theta^\{\*\},\\eta\)=0\. Atη=0\\eta=0, we haveθ∗​\(0\)=θ0\\theta^\{\*\}\(0\)=\\theta\_\{0\}, the CFR baseline from Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1)\.

Implicit differentiation\.By the implicit function theorem, if∂Φ/∂θ≠0\\partial\\Phi/\\partial\\theta\\neq 0at\(θ0,0\)\(\\theta\_\{0\},0\), then:

d​θ∗d​η\|η=0=−∂Φ/∂η∂Φ/∂θ\|\(θ0,0\)\.\\frac\{d\\theta^\{\*\}\}\{d\\eta\}\\bigg\|\_\{\\eta=0\}=\-\\frac\{\\partial\\Phi/\\partial\\eta\}\{\\partial\\Phi/\\partial\\theta\}\\bigg\|\_\{\(\\theta\_\{0\},0\)\}\.\(20\)
Thus the first\-order correction is:

Δ​θ=θ∗​\(η\)−θ0=−∂Φ/∂η∂Φ/∂θ\|\(θ0,0\)⋅η\+O​\(η2\)\.\\Delta\\theta=\\theta^\{\*\}\(\\eta\)\-\\theta\_\{0\}=\-\\frac\{\\partial\\Phi/\\partial\\eta\}\{\\partial\\Phi/\\partial\\theta\}\\bigg\|\_\{\(\\theta\_\{0\},0\)\}\\cdot\\eta\+O\(\\eta^\{2\}\)\.
Computing∂Φ/∂θ\\partial\\Phi/\\partial\\thetaatη=0\\eta=0\.

FromΦ=𝒯−θ​𝒯′\\Phi=\\mathcal\{T\}\-\\theta\\mathcal\{T\}^\{\\prime\}:

∂Φ∂θ=𝒯′−𝒯′−θ​𝒯′′=−θ​𝒯′′\.\\frac\{\\partial\\Phi\}\{\\partial\\theta\}=\\mathcal\{T\}^\{\\prime\}\-\\mathcal\{T\}^\{\\prime\}\-\\theta\\mathcal\{T\}^\{\\prime\\prime\}=\-\\theta\\mathcal\{T\}^\{\\prime\\prime\}\.
Atη=0\\eta=0:

𝒯0′′​\(θ\)=dd​θ​\[αdp0​\(1−θ\)\+βd​Np0\+βp​N​μL\]=αdp0​\(1−θ\)2\.\\mathcal\{T\}\_\{0\}^\{\\prime\\prime\}\(\\theta\)=\\frac\{d\}\{d\\theta\}\\left\[\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\)\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}\}\+\\beta\_\{p\}N\\mu\_\{L\}\\right\]=\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\)^\{2\}\}\.
Thus:

∂Φ∂θ\|\(θ0,0\)=−θ0⋅αdp0​\(1−θ0\)2=−θ0​αdp0​\(1−θ0\)2\.\\frac\{\\partial\\Phi\}\{\\partial\\theta\}\\bigg\|\_\{\(\\theta\_\{0\},0\)\}=\-\\theta\_\{0\}\\cdot\\frac\{\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\_\{0\}\)^\{2\}\}=\-\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\_\{0\}\)^\{2\}\}\.
Computing∂Φ/∂η\\partial\\Phi/\\partial\\etaat\(θ0,0\)\(\\theta\_\{0\},0\)\.

From𝒯=𝒯0\+η​𝒯1\+O​\(η2\)\\mathcal\{T\}=\\mathcal\{T\}\_\{0\}\+\\eta\\,\\mathcal\{T\}\_\{1\}\+O\(\\eta^\{2\}\):

∂𝒯∂η\|η=0=𝒯1​\(θ\)\.\\frac\{\\partial\\mathcal\{T\}\}\{\\partial\\eta\}\\bigg\|\_\{\\eta=0\}=\\mathcal\{T\}\_\{1\}\(\\theta\)\.
Similarly,𝒯′=𝒯0′\+η​𝒯1′\+O​\(η2\)\\mathcal\{T\}^\{\\prime\}=\\mathcal\{T\}\_\{0\}^\{\\prime\}\+\\eta\\,\\mathcal\{T\}\_\{1\}^\{\\prime\}\+O\(\\eta^\{2\}\), so:

∂∂η​\(θ​𝒯′\)\|η=0=θ​𝒯1′​\(θ\)\.\\frac\{\\partial\}\{\\partial\\eta\}\(\\theta\\mathcal\{T\}^\{\\prime\}\)\\bigg\|\_\{\\eta=0\}=\\theta\\,\\mathcal\{T\}\_\{1\}^\{\\prime\}\(\\theta\)\.
Thus:

∂Φ∂η\|\(θ0,0\)=𝒯1​\(θ0\)−θ0​𝒯1′​\(θ0\)\.\\frac\{\\partial\\Phi\}\{\\partial\\eta\}\\bigg\|\_\{\(\\theta\_\{0\},0\)\}=\\mathcal\{T\}\_\{1\}\(\\theta\_\{0\}\)\-\\theta\_\{0\}\\,\\mathcal\{T\}\_\{1\}^\{\\prime\}\(\\theta\_\{0\}\)\.
Computing𝒯1\\mathcal\{T\}\_\{1\}and𝒯1′\\mathcal\{T\}\_\{1\}^\{\\prime\}\.From \([19](https://arxiv.org/html/2606.00516#A3.E19)\):

𝒯1​\(θ\)=−αd​ζ22​p03\+βd​Np03​\[\(1−θ\)​ζ−θ\]\.\\mathcal\{T\}\_\{1\}\(\\theta\)=\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2p\_\{0\}^\{3\}\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}^\{3\}\}\\left\[\(1\-\\theta\)\\zeta\-\\theta\\right\]\.
Taking the derivative usingζ′=1/\(1−θ\)\\zeta^\{\\prime\}=1/\(1\-\\theta\):

𝒯1′​\(θ\)=−αd⋅2​ζ⋅ζ′2​p03\+βd​Np03​\[−ζ\+\(1−θ\)​ζ′−1\]=−αd​ζp03​\(1−θ\)\+βd​Np03​\[−ζ\+1−1\]\.\\mathcal\{T\}\_\{1\}^\{\\prime\}\(\\theta\)=\-\\frac\{\\alpha\_\{d\}\\cdot 2\\zeta\\cdot\\zeta^\{\\prime\}\}\{2p\_\{0\}^\{3\}\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}^\{3\}\}\\left\[\-\\zeta\+\(1\-\\theta\)\\zeta^\{\\prime\}\-1\\right\]=\-\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}^\{3\}\(1\-\\theta\)\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}^\{3\}\}\\left\[\-\\zeta\+1\-1\\right\]\.
Simplifying:

𝒯1′​\(θ\)=−αd​ζp03​\(1−θ\)−βd​N​ζp03\.\\mathcal\{T\}\_\{1\}^\{\\prime\}\(\\theta\)=\-\\frac\{\\alpha\_\{d\}\\zeta\}\{p\_\{0\}^\{3\}\(1\-\\theta\)\}\-\\frac\{\\beta\_\{d\}N\\zeta\}\{p\_\{0\}^\{3\}\}\.
Now compute𝒯1−θ​𝒯1′\\mathcal\{T\}\_\{1\}\-\\theta\\,\\mathcal\{T\}\_\{1\}^\{\\prime\}:

𝒯1​\(θ0\)−θ0​𝒯1′​\(θ0\)\\displaystyle\\mathcal\{T\}\_\{1\}\(\\theta\_\{0\}\)\-\\theta\_\{0\}\\,\\mathcal\{T\}\_\{1\}^\{\\prime\}\(\\theta\_\{0\}\)=−αd​ζ22​p03\+βd​Np03​\[\(1−θ0\)​ζ−θ0\]\+θ0​αd​ζp03​\(1−θ0\)\+θ0​βd​N​ζp03\\displaystyle=\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2p\_\{0\}^\{3\}\}\+\\frac\{\\beta\_\{d\}N\}\{p\_\{0\}^\{3\}\}\[\(1\-\\theta\_\{0\}\)\\zeta\-\\theta\_\{0\}\]\+\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\\zeta\}\{p\_\{0\}^\{3\}\(1\-\\theta\_\{0\}\)\}\+\\frac\{\\theta\_\{0\}\\beta\_\{d\}N\\zeta\}\{p\_\{0\}^\{3\}\}=1p03​\[−αd​ζ22\+θ0​αd​ζ1−θ0\+βd​N​\(\(1−θ0\)​ζ−θ0\+θ0​ζ\)\]\\displaystyle=\\frac\{1\}\{p\_\{0\}^\{3\}\}\\left\[\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2\}\+\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\\zeta\}\{1\-\\theta\_\{0\}\}\+\\beta\_\{d\}N\\left\(\(1\-\\theta\_\{0\}\)\\zeta\-\\theta\_\{0\}\+\\theta\_\{0\}\\zeta\\right\)\\right\]=1p03​\[−αd​ζ22\+θ0​αd​ζ1−θ0\+βd​N​\(ζ−θ0\)\]\.\\displaystyle=\\frac\{1\}\{p\_\{0\}^\{3\}\}\\left\[\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2\}\+\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\\zeta\}\{1\-\\theta\_\{0\}\}\+\\beta\_\{d\}N\(\\zeta\-\\theta\_\{0\}\)\\right\]\.
Final expression\.Substituting into \([20](https://arxiv.org/html/2606.00516#A3.E20)\):

Δ​θ=−1p03​\[−αd​ζ22\+θ0​αd​ζ1−θ0\+βd​N​\(ζ−θ0\)\]−θ0​αdp0​\(1−θ0\)2⋅η\.\\Delta\\theta=\-\\frac\{\\frac\{1\}\{p\_\{0\}^\{3\}\}\\left\[\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2\}\+\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\\zeta\}\{1\-\\theta\_\{0\}\}\+\\beta\_\{d\}N\(\\zeta\-\\theta\_\{0\}\)\\right\]\}\{\-\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\}\{p\_\{0\}\(1\-\\theta\_\{0\}\)^\{2\}\}\}\\cdot\\eta\.
Simplifying:

Δ​θ=\(1−θ0\)2p02​θ0​αd​\[−αd​ζ22\+θ0​αd​ζ1−θ0\+βd​N​\(ζ−θ0\)\]⋅η\.\\Delta\\theta=\\frac\{\(1\-\\theta\_\{0\}\)^\{2\}\}\{p\_\{0\}^\{2\}\\theta\_\{0\}\\alpha\_\{d\}\}\\left\[\-\\frac\{\\alpha\_\{d\}\\zeta^\{2\}\}\{2\}\+\\frac\{\\theta\_\{0\}\\alpha\_\{d\}\\zeta\}\{1\-\\theta\_\{0\}\}\+\\beta\_\{d\}N\(\\zeta\-\\theta\_\{0\}\)\\right\]\\cdot\\eta\.
Factoring outαd\\alpha\_\{d\}from the first two terms:

Δ​θ=η​\(1−θ0\)2p02​θ0​\[ζ​\(θ01−θ0−ζ2\)\+βd​Nαd​\(ζ−θ0\)\]\\boxed\{\\Delta\\theta=\\frac\{\\eta\(1\-\\theta\_\{0\}\)^\{2\}\}\{p\_\{0\}^\{2\}\\theta\_\{0\}\}\\left\[\\zeta\\left\(\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}\\right\)\+\\frac\{\\beta\_\{d\}N\}\{\\alpha\_\{d\}\}\(\\zeta\-\\theta\_\{0\}\)\\right\]\}\(21\)
This is Eq\. \([5](https://arxiv.org/html/2606.00516#S3.E5)\) in the main text\.

### C\.6Positivity of the Correction

###### Proposition C\.1\.

Δ​θ\>0\\Delta\\theta\>0for allθ0∈\(0,1\)\\theta\_\{0\}\\in\(0,1\)andη\>0\\eta\>0\.

###### Proof\.

The prefactorη​\(1−θ0\)2p02​θ0\>0\\frac\{\\eta\(1\-\\theta\_\{0\}\)^\{2\}\}\{p\_\{0\}^\{2\}\\theta\_\{0\}\}\>0for all valid parameters\.

For the bracketed term, we show each component is positive:

Term 1:ζ​\(θ01−θ0−ζ2\)\\zeta\\left\(\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}\\right\)\.

Using the baseline condition \([15](https://arxiv.org/html/2606.00516#A2.E15)\), we have:

θ01−θ0=γ\+ζ,where​γ=p0​αpαd\>0\.\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}=\\gamma\+\\zeta,\\quad\\text\{where \}\\gamma=\\frac\{p\_\{0\}\\alpha\_\{p\}\}\{\\alpha\_\{d\}\}\>0\.
Thus:

θ01−θ0−ζ2=γ\+ζ−ζ2=γ\+ζ2\>0\.\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}=\\gamma\+\\zeta\-\\frac\{\\zeta\}\{2\}=\\gamma\+\\frac\{\\zeta\}\{2\}\>0\.
Sinceζ=−ln⁡\(1−θ0\)\>0\\zeta=\-\\ln\(1\-\\theta\_\{0\}\)\>0forθ0∈\(0,1\)\\theta\_\{0\}\\in\(0,1\):

ζ​\(θ01−θ0−ζ2\)=ζ​\(γ\+ζ2\)\>0\.\\zeta\\left\(\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}\\right\)=\\zeta\\left\(\\gamma\+\\frac\{\\zeta\}\{2\}\\right\)\>0\.
Term 2:βd​Nαd​\(ζ−θ0\)\\frac\{\\beta\_\{d\}N\}\{\\alpha\_\{d\}\}\(\\zeta\-\\theta\_\{0\}\)\.

We need to showζ\>θ0\\zeta\>\\theta\_\{0\}, i\.e\.,−ln⁡\(1−θ0\)\>θ0\-\\ln\(1\-\\theta\_\{0\}\)\>\\theta\_\{0\}\.

Defineq​\(x\)=−ln⁡\(1−x\)−xq\(x\)=\-\\ln\(1\-x\)\-xforx∈\[0,1\)x\\in\[0,1\)\. Then:

- •q​\(0\)=0q\(0\)=0\.
- •q′​\(x\)=11−x−1=x1−x\>0q^\{\\prime\}\(x\)=\\frac\{1\}\{1\-x\}\-1=\\frac\{x\}\{1\-x\}\>0forx∈\(0,1\)x\\in\(0,1\)\.

Sinceqqis strictly increasing withq​\(0\)=0q\(0\)=0, we haveq​\(θ0\)\>0q\(\\theta\_\{0\}\)\>0for allθ0∈\(0,1\)\\theta\_\{0\}\\in\(0,1\), i\.e\.,ζ\>θ0\\zeta\>\\theta\_\{0\}\.

Sinceβd,N,αd\>0\\beta\_\{d\},N,\\alpha\_\{d\}\>0, the second term is positive \(or zero ifβd=0\\beta\_\{d\}=0\)\.

Conclusion\.Both terms are non\-negative with the first strictly positive, henceΔ​θ\>0\\Delta\\theta\>0\. ∎

### C\.7Physical Interpretation

The correction \([21](https://arxiv.org/html/2606.00516#A3.E21)\) decomposes into two distinct effects:

Duration effect \(first term\)\.Under IFR, the hazard rate increases over time, meaning later completions occur faster than earlier ones\. This reduces the expected time to accumulate additional completions beyond the CFR prediction, making it beneficial to wait for more completions\.

Mathematically, the termζ​\(θ01−θ0−ζ2\)\\zeta\\left\(\\frac\{\\theta\_\{0\}\}\{1\-\\theta\_\{0\}\}\-\\frac\{\\zeta\}\{2\}\\right\)captures the reduction in decode phase duration per additional completion achieved\.

Per\-token cost effect \(second term\)\.Under IFR, the batch size decays faster than exponential \(the Gaussian factore−η​t2/2e^\{\-\\eta t^\{2\}/2\}accelerates the decay\)\. This reduces the cumulative per\-token costβd​∫n​\(s\)​𝑑s\\beta\_\{d\}\\int n\(s\)dsfor a given number of completions\.

This benefit scales withρ=βd​N/αd\\rho=\\beta\_\{d\}N/\\alpha\_\{d\}, the ratio of total per\-token overhead to fixed overhead\. For systems with large batches and significant per\-token costs \(ρ≫1\\rho\\gg 1\), this effect dominates\.

Special case:βd=0\\beta\_\{d\}=0\.When there is no per\-token decode cost \(pure fixed overhead\), only the duration effect contributes:

Δ​θ\|βd=0=η​\(1−θ0\)2p02​θ0⋅ζ​\(γ\+ζ2\)\.\\Delta\\theta\\big\|\_\{\\beta\_\{d\}=0\}=\\frac\{\\eta\(1\-\\theta\_\{0\}\)^\{2\}\}\{p\_\{0\}^\{2\}\\theta\_\{0\}\}\\cdot\\zeta\\left\(\\gamma\+\\frac\{\\zeta\}\{2\}\\right\)\.

### C\.8Second\-Order Verification

###### Lemma C\.2\.

The corrected thresholdθ∗=θ0\+Δ​θ\+O​\(η2\)\\theta^\{\*\}=\\theta\_\{0\}\+\\Delta\\theta\+O\(\\eta^\{2\}\)remains a maximum of the throughput function for sufficiently smallη\>0\\eta\>0\.

###### Proof\.

The second\-order condition requires∂2R/∂θ2<0\\partial^\{2\}R/\\partial\\theta^\{2\}<0at the optimum\.

Atη=0\\eta=0, we showed in Appendix[B](https://arxiv.org/html/2606.00516#A2)thatθ0\\theta\_\{0\}is a strict local maximum withΦ′​\(θ0\)=−θ0​𝒯0′′​\(θ0\)<0\\Phi^\{\\prime\}\(\\theta\_\{0\}\)=\-\\theta\_\{0\}\\mathcal\{T\}\_\{0\}^\{\\prime\\prime\}\(\\theta\_\{0\}\)<0\.

By continuity, for sufficiently smallη\\eta, the second derivative remains negative in a neighborhood ofθ∗​\(η\)\\theta^\{\*\}\(\\eta\), confirming it remains a local maximum\.

To verify it is the global maximum, observe that the boundary behavior \(R​\(0\+\)=0R\(0^\{\+\}\)=0andR​\(1−\)R\(1^\{\-\}\)bounded\) is preserved under small perturbations, ensuring the interior maximum is global\. ∎

This completes the proof of Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)\. ∎

## Appendix DProof of Proposition[3\.3](https://arxiv.org/html/2606.00516#S3.Thmtheorem3): Memory\-Safe Batch Size

The proof proceeds in four steps: \(1\) steady\-state analysis of request ages, \(2\) expected initial memory, \(3\) variance of memory increments, and \(4\) supremum bound\. We model the memory process as a random walk with i\.i\.d\. increments and apply an infinite\-horizon supremum bound—a standard heuristic\-conservative approximation; a fully rigorous finite\-horizon martingale concentration analysis is left to future work\.

### D\.1Steady\-State Request Age Distribution

Define the*age*AAof a request as the number of decode phases it has participated in\. In each cycle,kkout ofNNrequests complete and are replaced\. At steady state,AAfollows a geometric distribution:

Pr⁡\(A=a\)=θ​\(1−θ\)a−1,a=1,2,…\\Pr\(A=a\)=\\theta\(1\-\\theta\)^\{a\-1\},\\quad a=1,2,\\ldotswith mean𝔼​\[A\]=1/θ\\mathbb\{E\}\[A\]=1/\\theta, whereθ=k/N\\theta=k/N\.

At decode start, the batch containskknew requests \(ageA=1A=1\) and\(N−k\)\(N\-k\)continuing requests \(ageA≥2A\\geq 2\)\. For continuing requests, the memoryless property gives:

𝔼​\[A−1∣A≥2\]=𝔼​\[A\]=1θ\.\\mathbb\{E\}\[A\-1\\mid A\\geq 2\]=\\mathbb\{E\}\[A\]=\\frac\{1\}\{\\theta\}\.

### D\.2Expected Tokens Generated by Continuing Requests

Under the fluid approximation, starting withNNrequests and waiting forkkcompletions, the expected decode phase duration \(in iterations\) is

t∗=1p0​ln⁡11−θ,t^\{\*\}=\\frac\{1\}\{p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\},matching the CFR stopping time from Appendix[B](https://arxiv.org/html/2606.00516#A2)\.

A continuing request has survived\(A−1\)\(A\-1\)previous phases, generating tokens in each\. Thus:

𝔼​\[G∣continuing\]=𝔼​\[A−1∣A≥2\]⋅t∗=1θ⋅1p0​ln⁡11−θ=1θ​p0​ln⁡11−θ\.\\mathbb\{E\}\[G\\mid\\text\{continuing\}\]=\\mathbb\{E\}\[A\-1\\mid A\\geq 2\]\\cdot t^\{\*\}=\\frac\{1\}\{\\theta\}\\cdot\\frac\{1\}\{p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}=\\frac\{1\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\.
The expected initial memory is:

𝔼​\[X0\]=k⋅μL\+\(N−k\)⋅\(μL\+𝔼​\[G\]\)=N​μL\+N​\(1−θ\)θ​p0​ln⁡11−θ\.\\mathbb\{E\}\[X\_\{0\}\]=k\\cdot\\mu\_\{L\}\+\(N\-k\)\\cdot\(\\mu\_\{L\}\+\\mathbb\{E\}\[G\]\)=N\\mu\_\{L\}\+\\frac\{N\(1\-\\theta\)\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\.

### D\.3Variance of Memory Increments

At each decode iteration withnnactive requests, the memory change is:

Δ​X=n−∑i=1nZi​Si,\\Delta X=n\-\\sum\_\{i=1\}^\{n\}Z\_\{i\}S\_\{i\},whereZi∼Bernoulli​\(p0\)Z\_\{i\}\\sim\\mathrm\{Bernoulli\}\(p\_\{0\}\)indicates completion andSi=Li\+OiS\_\{i\}=L\_\{i\}\+O\_\{i\}is the total sequence length released upon completion\. LetJ=∑i=1nZi∼Binomial​\(n,p0\)J=\\sum\_\{i=1\}^\{n\}Z\_\{i\}\\sim\\mathrm\{Binomial\}\(n,p\_\{0\}\)\.

Expected change\.Since completed requests have𝔼​\[S∣complete\]=μL\+1/p0\\mathbb\{E\}\[S\\mid\\text\{complete\}\]=\\mu\_\{L\}\+1/p\_\{0\}:

𝔼​\[Δ​X\]=n−n​p0​\(μL\+1/p0\)=n−n​p0​μL−n=−n​p0​μL=dn\.\\mathbb\{E\}\[\\Delta X\]=n\-np\_\{0\}\(\\mu\_\{L\}\+1/p\_\{0\}\)=n\-np\_\{0\}\\mu\_\{L\}\-n=\-np\_\{0\}\\mu\_\{L\}=d\_\{n\}\.
Variance via law of total variance\.We computeVar​\(Δ​X\)=𝔼​\[Var​\(Δ​X∣J\)\]\+Var​\(𝔼​\[Δ​X∣J\]\)\\mathrm\{Var\}\(\\Delta X\)=\\mathbb\{E\}\[\\mathrm\{Var\}\(\\Delta X\\mid J\)\]\+\\mathrm\{Var\}\(\\mathbb\{E\}\[\\Delta X\\mid J\]\)\.

For the first term: givenJ=jJ=jcompletions, the released memory is∑i=1jSi\\sum\_\{i=1\}^\{j\}S\_\{i\}whereSiS\_\{i\}are i\.i\.d\. ThusVar​\(Δ​X∣J=j\)=j⋅Var​\(S\)\\mathrm\{Var\}\(\\Delta X\\mid J=j\)=j\\cdot\\mathrm\{Var\}\(S\), and:

𝔼​\[Var​\(Δ​X∣J\)\]=𝔼​\[J\]⋅Var​\(S\)=n​p0⋅Var​\(S\)\.\\mathbb\{E\}\[\\mathrm\{Var\}\(\\Delta X\\mid J\)\]=\\mathbb\{E\}\[J\]\\cdot\\mathrm\{Var\}\(S\)=np\_\{0\}\\cdot\\mathrm\{Var\}\(S\)\.
For the second term:𝔼​\[Δ​X∣J\]=n−J⋅𝔼​\[S\]\\mathbb\{E\}\[\\Delta X\\mid J\]=n\-J\\cdot\\mathbb\{E\}\[S\], so:

Var​\(𝔼​\[Δ​X∣J\]\)=𝔼​\[S\]2⋅Var​\(J\)=\(μL\+1/p0\)2⋅n​p0​\(1−p0\)\.\\mathrm\{Var\}\(\\mathbb\{E\}\[\\Delta X\\mid J\]\)=\\mathbb\{E\}\[S\]^\{2\}\\cdot\\mathrm\{Var\}\(J\)=\(\\mu\_\{L\}\+1/p\_\{0\}\)^\{2\}\\cdot np\_\{0\}\(1\-p\_\{0\}\)\.
Combining and usingVar​\(S\)=σL2\+\(1−p0\)/p02\\mathrm\{Var\}\(S\)=\\sigma\_\{L\}^\{2\}\+\(1\-p\_\{0\}\)/p\_\{0\}^\{2\}:

vn=n​p0​\[σL2\+1−p0p02\]\+n​p0​\(1−p0\)​\(μL\+1p0\)2\.v\_\{n\}=np\_\{0\}\\left\[\\sigma\_\{L\}^\{2\}\+\\frac\{1\-p\_\{0\}\}\{p\_\{0\}^\{2\}\}\\right\]\+np\_\{0\}\(1\-p\_\{0\}\)\\left\(\\mu\_\{L\}\+\\frac\{1\}\{p\_\{0\}\}\\right\)^\{2\}\.
Forp0≪1p\_\{0\}\\ll 1and outputs dominating inputs \(μL≪1/p0\\mu\_\{L\}\\ll 1/p\_\{0\}\), the dominant terms are:

vn≈n​p0⋅1p02\+n​p0⋅1p02=2​np0\.v\_\{n\}\\approx np\_\{0\}\\cdot\\frac\{1\}\{p\_\{0\}^\{2\}\}\+np\_\{0\}\\cdot\\frac\{1\}\{p\_\{0\}^\{2\}\}=\\frac\{2n\}\{p\_\{0\}\}\.

### D\.4Supremum Bound

The memory processYt=Xt−X0Y\_\{t\}=X\_\{t\}\-X\_\{0\}resembles a random walk with negative driftdn<0d\_\{n\}<0and per\-step variancevnv\_\{n\}\. For such walks, the all\-time supremum satisfies\(Feller,[1971](https://arxiv.org/html/2606.00516#bib.bib3)\):

𝔼​\[supt≥0Yt\]=v2​\|d\|,Pr⁡\(supt≥0Yt\>x\)≤exp⁡\(−2​\|d\|​xv\)\.\\mathbb\{E\}\\left\[\\sup\_\{t\\geq 0\}Y\_\{t\}\\right\]=\\frac\{v\}\{2\|d\|\},\\quad\\Pr\\\!\\left\(\\sup\_\{t\\geq 0\}Y\_\{t\}\>x\\right\)\\leq\\exp\\left\(\-\\frac\{2\|d\|x\}\{v\}\\right\)\.\(22\)
Since the batch sizenndecreases during decode \(fromNNtoN−kN\-k\), increments are not strictly i\.i\.d\. Usingn=Nn=Nas an upper bound yields conservative estimates:

vN≈2​Np0,\|dN\|=N​p0​μL\.v\_\{N\}\\approx\\frac\{2N\}\{p\_\{0\}\},\\quad\|d\_\{N\}\|=Np\_\{0\}\\mu\_\{L\}\.
The expected supremum is:

𝔼​\[supt≥0Yt\]≈vN2​\|dN\|=2​N/p02​N​p0​μL=1p02​μL≜v¯\.\\mathbb\{E\}\\left\[\\sup\_\{t\\geq 0\}Y\_\{t\}\\right\]\\approx\\frac\{v\_\{N\}\}\{2\|d\_\{N\}\|\}=\\frac\{2N/p\_\{0\}\}\{2Np\_\{0\}\\mu\_\{L\}\}=\\frac\{1\}\{p\_\{0\}^\{2\}\\mu\_\{L\}\}\\triangleq\\bar\{v\}\.
We introducev¯=\(p02​μL\)−1\\bar\{v\}=\(p\_\{0\}^\{2\}\\mu\_\{L\}\)^\{\-1\}as shorthand for the remainder of this proof; it characterizes the memory volatility arising from stochastic completion timing, and is notably independent ofNN\.

Probabilistic bound\.For OOM probability at mostϵ\\epsilon, we requirePr⁡\(supYt\>C−𝔼​\[X0\]\)≤ϵ\\Pr\(\\sup Y\_\{t\}\>C\-\\mathbb\{E\}\[X\_\{0\}\]\)\\leq\\epsilon\. From \([22](https://arxiv.org/html/2606.00516#A4.E22)\):

Pr⁡\(supYt\>x\)≤exp⁡\(−2​\|dN\|​xvN\)=exp⁡\(−xv¯\)\.\\Pr\\\!\\left\(\\sup Y\_\{t\}\>x\\right\)\\leq\\exp\\left\(\-\\frac\{2\|d\_\{N\}\|x\}\{v\_\{N\}\}\\right\)=\\exp\\left\(\-\\frac\{x\}\{\\bar\{v\}\}\\right\)\.
Setting this toϵ\\epsilonand solving:xϵ=v¯​ln⁡\(1/ϵ\)x\_\{\\epsilon\}=\\bar\{v\}\\ln\(1/\\epsilon\)\.

The constraint𝔼​\[X0\]\+xϵ≤C\\mathbb\{E\}\[X\_\{0\}\]\+x\_\{\\epsilon\}\\leq Cyields:

N​μL\+N​\(1−θ\)θ​p0​ln⁡11−θ\+v¯​ln⁡1ϵ≤C\.N\\mu\_\{L\}\+\\frac\{N\(1\-\\theta\)\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\+\\bar\{v\}\\ln\\frac\{1\}\{\\epsilon\}\\leq C\.
Solving forNNgives \([6](https://arxiv.org/html/2606.00516#S3.E6)\)\. ∎

### D\.5Comparison of Batch Size Bounds

We present three batch size bounds of increasing conservatism:

Static bound\(ignores decode dynamics\):

Nstatic∗=⌊CμL\+1−θθ​p0​ln⁡11−θ⌋\.N^\{\*\}\_\{\\mathrm\{static\}\}=\\left\\lfloor\\frac\{C\}\{\\mu\_\{L\}\+\\frac\{1\-\\theta\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\}\\right\\rfloor\.
Expected peak bound\(uses𝔼​\[supYt\]=v¯\\mathbb\{E\}\[\\sup Y\_\{t\}\]=\\bar\{v\}\):

Nexpected∗=⌊C−v¯μL\+1−θθ​p0​ln⁡11−θ⌋\.N^\{\*\}\_\{\\mathrm\{expected\}\}=\\left\\lfloor\\frac\{C\-\\bar\{v\}\}\{\\mu\_\{L\}\+\\frac\{1\-\\theta\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\}\\right\\rfloor\.
Probabilistic bound\(OOM probability≤ϵ\\leq\\epsilon\):

N∗=⌊C−v¯​ln⁡\(1/ϵ\)μL\+1−θθ​p0​ln⁡11−θ⌋\.N^\{\*\}=\\left\\lfloor\\frac\{C\-\\bar\{v\}\\ln\(1/\\epsilon\)\}\{\\mu\_\{L\}\+\\frac\{1\-\\theta\}\{\\theta p\_\{0\}\}\\ln\\frac\{1\}\{1\-\\theta\}\}\\right\\rfloor\.
These satisfyN∗≤Nexpected∗≤Nstatic∗N^\{\*\}\\leq N^\{\*\}\_\{\\mathrm\{expected\}\}\\leq N^\{\*\}\_\{\\mathrm\{static\}\}\. The gap depends onv¯​ln⁡\(1/ϵ\)\\bar\{v\}\\ln\(1/\\epsilon\), which is typically small relative toCCfor practical workloads\.

### D\.6Runtime Refinement: KV\-aware Feasibility Gate

The design\-time boundN∗N^\{\*\}above guarantees*expected\-value*memory safety: withN≤N∗N\\leq N^\{\*\}, the supremum of the memory process exceeds the KV capacity with probability at mostϵ\\epsilonunder the model assumptions\. Under high output\-length variance \(σO\\sigma\_\{O\}large\), the realized peak can transiently deviate from𝔼​\[supYt\]\\mathbb\{E\}\[\\sup Y\_\{t\}\]beyond theϵ\\epsilon\-tolerance, especially for MoE models where weight activations leave less headroom \(e\.g\., Qwen3\-30B\-A3B withσO∈\{153,245\}\\sigma\_\{O\}\\in\\\{153,245\\\}on ShareGPT/WildChat\)\.

To handle this at runtime, our scheduler augments the ratio\-based phase 1→\\to2 transition condition with an instantaneous KV\-occupancy check\. The original integer\-arithmetic conditionfillable⋅\(N−k∗\)≥n⋅k∗\\text\{fillable\}\\cdot\(N\-k^\{\*\}\)\\geq n\\cdot k^\{\*\}is gated by

free\_blocks≥fK​V⋅total\_blocks,fK​V=min⁡\(0\.6,max⁡\(0\.05,N⋅μO⋅sK​Vb⋅total\_blocks\+f0\)\),\\text\{free\\\_blocks\}\\;\\geq\\;f\_\{KV\}\\cdot\\text\{total\\\_blocks\},\\quad f\_\{KV\}=\\min\\\!\\left\(0\.6,\\;\\max\\\!\\left\(0\.05,\\;\\frac\{N\\cdot\\mu\_\{O\}\\cdot s\_\{KV\}\}\{b\\cdot\\text\{total\\\_blocks\}\}\+f\_\{0\}\\right\)\\right\),\(23\)wherebbis the block size,sK​Vs\_\{KV\}a safety multiplier onμO\\mu\_\{O\}\(defaultsK​V=0\.5s\_\{KV\}=0\.5\), andf0f\_\{0\}a base reserve \(default0\)\. The thresholdfK​Vf\_\{KV\}scales with the expected per\-batch decode KV usage and is clipped into\[0\.05,0\.6\]\[0\.05,0\.6\]for numerical stability\.

When the gate fires, phase 2 is deferred and the scheduler stays in phase 1, letting existing decodes free KV via natural completion before attempting refill\. This prevents the DECODE↔\\leftrightarrowREFILL\_PREFILL oscillation triggered when phase 2 ratio\-eligibility coincides with insufficient KV for prefill block allocation \(which would otherwise bounce back through the vLLM preemption path\)\. The gate is orthogonal toN∗N^\{\*\}:N∗N^\{\*\}bounds the static batch parameter, while the gate adds a runtime feasibility check at the moment of switching, providing defense\-in\-depth against transient pressure\. Empirically \(Table[2](https://arxiv.org/html/2606.00516#S4.T2), H200 block\), this reduces the EB\(k^∗\\hat\{k\}^\{\*\}\) vs\. v1 gap on Qwen3\-30B\-A3B ShareGPT from−12\.0%\-12\.0\\%to−2\.5%\-2\.5\\%and on NuminaMath from−11\.4%\-11\.4\\%to−8\.2%\-8\.2\\%\.

### D\.7Online Adaptive Algorithm: Pseudo\-code

Algorithm[1](https://arxiv.org/html/2606.00516#alg1)provides the pseudo\-code for the online controller described in Section[3\.4](https://arxiv.org/html/2606.00516#S3.SS4), which jointly updates\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)from sliding\-window estimates of the workload parameters\.

Algorithm 1Online Joint Adaptation of\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)0:System parameters

\(αp,αd,βd\)\(\\alpha\_\{p\},\\alpha\_\{d\},\\beta\_\{d\}\); cache budget

CC; risk level

ϵ\\epsilon; window size

WW; minimum window

WminW\_\{\\min\}; update interval

TupdT\_\{\\mathrm\{upd\}\}
0:Continuously updated

\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)
1:Initialize

𝒲O,𝒲L←∅\\mathcal\{W\}\_\{O\},\\mathcal\{W\}\_\{L\}\\leftarrow\\emptyset,

j←0j\\leftarrow 0
2:Initialize

\(k^∗,N^∗\)←\(⌊θdefault​Ndefault⌋,Ndefault\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\)\\leftarrow\(\\lfloor\\theta\_\{\\mathrm\{default\}\}N\_\{\\mathrm\{default\}\}\\rfloor,N\_\{\\mathrm\{default\}\}\)
3:foreach completed request with input length

LLand output length

OOdo

4:

𝒲O\.append​\(O\)\\mathcal\{W\}\_\{O\}\.\\mathrm\{append\}\(O\);

𝒲L\.append​\(L\)\\mathcal\{W\}\_\{L\}\.\\mathrm\{append\}\(L\);if

\|𝒲O\|\>W\|\\mathcal\{W\}\_\{O\}\|\>Wthenpop oldest \(same for

𝒲L\\mathcal\{W\}\_\{L\}\)

5:

j←j\+1j\\leftarrow j\+1
6:if

j≥Tupdj\\geq T\_\{\\mathrm\{upd\}\}and

\|𝒲O\|≥Wmin\|\\mathcal\{W\}\_\{O\}\|\\geq W\_\{\\min\}then

7:

\(p^0,η^\)←FitLinearHazard​\(𝒲O\)\(\\hat\{p\}\_\{0\},\\hat\{\\eta\}\)\\leftarrow\\mathrm\{FitLinearHazard\}\(\\mathcal\{W\}\_\{O\}\)
8:

μ^L←Mean​\(𝒲L\)\\hat\{\\mu\}\_\{L\}\\leftarrow\\mathrm\{Mean\}\(\\mathcal\{W\}\_\{L\}\)
9:

θ^0←SolveCFR​\(p^0\)\\hat\{\\theta\}\_\{0\}\\leftarrow\\mathrm\{SolveCFR\}\(\\hat\{p\}\_\{0\}\)⊳\\trianglerightEq\. \([3](https://arxiv.org/html/2606.00516#S3.E3)\)

10:

θ^∗←Clip​\(θ^0\+Δ​θ^,θmin,θmax\)\\hat\{\\theta\}^\{\*\}\\leftarrow\\mathrm\{Clip\}\(\\hat\{\\theta\}\_\{0\}\+\\widehat\{\\Delta\\theta\},\\theta\_\{\\min\},\\theta\_\{\\max\}\)⊳\\trianglerightEq\. \([5](https://arxiv.org/html/2606.00516#S3.E5)\)

11:

N^∗←MemSafeBatch​\(θ^∗,p^0,μ^L\)\\hat\{N\}^\{\*\}\\leftarrow\\mathrm\{MemSafeBatch\}\(\\hat\{\\theta\}^\{\*\},\\hat\{p\}\_\{0\},\\hat\{\\mu\}\_\{L\}\)⊳\\trianglerightEq\. \([6](https://arxiv.org/html/2606.00516#S3.E6)\)

12:

k^∗←⌊θ^∗​N^∗⌋\\hat\{k\}^\{\*\}\\leftarrow\\lfloor\\hat\{\\theta\}^\{\*\}\\hat\{N\}^\{\*\}\\rfloor; apply

\(k^∗,N^∗\)\(\\hat\{k\}^\{\*\},\\hat\{N\}^\{\*\}\);

j←0j\\leftarrow 0
13:endif

14:endfor

## Appendix EWorkload Characterization and IFR Validation

### E\.1Workload Distribution Analysis

Table[9](https://arxiv.org/html/2606.00516#A5.T9)reports basic input and output\-length statistics and whether the workload exhibits increasing failure rate \(IFR\) in this reliable region\. Note that WildChat comprises 3000 multi\-turn conversations rather than independent requests; with an average of 9\.3 turns per conversation \(range\[6,55\]\[6,55\]\), this yields approximately 27,900 total requests, providing comparable scale to the other workloads\. Figure[11](https://arxiv.org/html/2606.00516#A5.F11)summarizes output\-length distributions and empirical hazard rates for the real workloads used in our evaluation\. We estimate the hazard rate ash^​\(t\)=\#​\{Oi=t\}/\#​\{Oi≥t\}\\hat\{h\}\(t\)=\\\#\\\{O\_\{i\}=t\\\}/\\\#\\\{O\_\{i\}\\geq t\\\}, and only interpreth^​\(t\)\\hat\{h\}\(t\)up to the 95th percentile due to tail sparsity\.

Table 9:Input and output length statistics for experimental workloads\.InputOutputWorkloadSamplesRangeμL\\mu\_\{L\}RangeμO\\mu\_\{O\}σO\\sigma\_\{O\}IFRShareGPT4000\[1,20448\]\[1,20448\]105\[2,500\]\[2,500\]280153✓LongBench4000\[1000,4000\]\[1000,4000\]2492\[1,19\]\[1,19\]114✓NuminaMath4000\[16,1395\]\[16,1395\]122\[800,3149\]\[800,3149\]1039228✓WildChat3000†\[6,55\]‡\[6,55\]^\{\\ddagger\}9\.3‡\[1,1934\]\[1,1934\]310245✓†Conversations\.‡Turns per conversation\.![Refer to caption](https://arxiv.org/html/2606.00516v1/x13.png)Figure 11:Output length distributions and hazard rates of real workloads \(Gemma\-3\-1B\-IT on NVIDIA RTX PRO 6000\)\. \(a\) Output length histograms\. \(b\) Empirical hazard rateh^​\(t\)=\#​\{Oi=t\}/\#​\{Oi≥t\}\\hat\{h\}\(t\)=\\\#\\\{O\_\{i\}=t\\\}/\\\#\\\{O\_\{i\}\\geq t\\\}\. Dashed lines mark the 95th percentile; beyond this point estimates become unreliable due to data sparsity\.
### E\.2Baseline Threshold Validation

This appendix validates the structural properties of the CFR baseline normalized thresholdθ0\\theta\_\{0\}implied by Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1):

- •\(P1\) Scale invariance:θ0\\theta\_\{0\}is invariant to batch sizeNN\.
- •\(P2\) Input\-length independence:θ0\\theta\_\{0\}is insensitive to mean input lengthμL\\mu\_\{L\}\.
- •\(P3\) Output\-length dependence:θ0\\theta\_\{0\}decreases with mean output lengthμO\\mu\_\{O\}\.

We empirically study how the optimalθ^∗\\hat\{\\theta\}^\{\*\}varies with these parameters under geometric \(CFR\) output lengths\. All experiments use Qwen3\-8B on H200 with 4,000 requests per configuration \(including a 100\-request warmup\)\.

#### E\.2\.1Scale Invariance and Output\-Length Dependence \(P1, P3\)

Setup\.We sweepN∈\{64,128,256,512\}N\\in\\\{64,128,256,512\\\}andμO∈\{64,128,192,256,320,384\}\\mu\_\{O\}\\in\\\{64,128,192,256,320,384\\\}withμL=1\\mu\_\{L\}=1\. For each configuration, we run a threshold sweep \(3 runs\) and report the empirically optimalθ^∗=k^∗/N\\hat\{\\theta\}^\{\*\}=\\hat\{k\}^\{\*\}/Nin Table[10](https://arxiv.org/html/2606.00516#A5.T10)\.

Results\.Consistent with Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1):

- •\(P1\)For fixedμO\\mu\_\{O\},θ^∗\\hat\{\\theta\}^\{\*\}shows no systematic dependence onNN; the row summariesθ¯∗±std\\bar\{\\theta\}^\{\*\}\\pm\\mathrm\{std\}exhibit no trend withNN\.
- •\(P3\)The row means decrease from0\.460\.46atμO=64\\mu\_\{O\}=64to0\.230\.23atμO=384\\mu\_\{O\}=384, supporting the predicted monotone relationship\.

The remaining variance is expected due to discretekkchoices and measurement noise\.

Table 10:Empirical optimal thresholdθ^∗\\hat\{\\theta\}^\{\*\}across batch sizes and output lengths \(geometric outputs,μL=1\\mu\_\{L\}=1\)\.
#### E\.2\.2Input\-Length Independence \(P2\)

Setup\.To probe dependence onμL\\mu\_\{L\}, we fixN=128N=128andμO=64\\mu\_\{O\}=64, varyμL∈\{1,32,64,128\}\\mu\_\{L\}\\in\\\{1,32,64,128\\\}, and sweepkksubject tok​μL≤16384k\\mu\_\{L\}\\leq 16384\. Table[11](https://arxiv.org/html/2606.00516#A5.T11)reports throughput for eachkkand highlights the bestk∗k^\{\*\}perμL\\mu\_\{L\}\.

Results\.While the optimal discretek∗k^\{\*\}shifts withμL\\mu\_\{L\}under the fixed token\-budget constraint, the corresponding normalized optimumθ∗\\theta^\{\*\}remains in a comparable range \(0\.438–0\.625\), indicating weak sensitivity toμL\\mu\_\{L\}as predicted by Proposition[3\.1](https://arxiv.org/html/2606.00516#S3.Thmtheorem1)\.

Table 11:Throughput \(tokens/s\) for varyingk∗k^\{\*\}andμL\\mu\_\{L\}withN=128N=128,μO=64\\mu\_\{O\}=64\. Bold indicates optimalk∗k^\{\*\}for eachμL\\mu\_\{L\}\. All experiments satisfyk×μL≤16384k\\times\\mu\_\{L\}\\leq 16384\.

### E\.3Empirical Evidence forΔ​θ\>0\\Delta\\theta\>0under IFR

Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)predicts that the IFR correction satisfiesΔ​θ\>0\\Delta\\theta\>0, i\.e\., IFR workloads favor later switching than the CFR baseline\. We empirically validate this prediction through controlled experiments\.

Experimental Setup\.We generate synthetic output lengths from Gamma distributions with varying shape parameters to simulate different hazard rate behaviors\. Specifically, we useO∼Gamma​\(a,λ\)O\\sim\\text\{Gamma\}\(a,\\lambda\)where:

- •CFR case\(a=1a=1\): The Gamma distribution reduces to an exponential distribution, which exhibits a constant hazard rateh​\(t\)=λh\(t\)=\\lambda\.
- •IFR case\(a=2a=2\): The Gamma distribution with shape parametera\>1a\>1exhibits an increasing failure rate, where the hazard rateh​\(t\)h\(t\)increases monotonically withtt\.

To ensure a fair comparison, we adjust the rate parameterλ\\lambdasuch that the mean output lengthμO=a/λ\\mu\_\{O\}=a/\\lambdaremains constant across both settings\. This isolates the effect of the hazard rate structure from differences in average workload characteristics\. For each configuration, we sweep the normalized thresholdθ=k∗/N\\theta=k^\{\*\}/Nfrom 0 to 1 and measure the resulting throughput\. All experiments are repeated three times with different random seeds to assess variability\.

Results\.Figure[12](https://arxiv.org/html/2606.00516#A5.F12)illustrates the throughput as a function ofθ\\thetaunder both CFR and IFR output\-length distributions\. The shaded regions represent 95% confidence intervals across the three runs\. Under the CFR setting \(left panel,a=1a=1\), the optimal threshold isθ∗=0\.16\\theta^\{\*\}=0\.16\. Under the IFR setting \(right panel,a=2a=2\), the optimal threshold shifts toθ∗=0\.36\\theta^\{\*\}=0\.36, representing a substantial increase ofΔ​θ=0\.20\\Delta\\theta=0\.20\. This empirical observation confirms the theoretical prediction that IFR workloads benefit from higher switching thresholds, as later switching allows the system to exploit the accelerating completion rate characteristic of IFR distributions\.

![Refer to caption](https://arxiv.org/html/2606.00516v1/x14.png)Figure 12:Throughput vs\. switching thresholdθ\\thetaunder CFR \(a=1a=1, left\) and IFR \(a=2a=2, right\) output\-length distributions modeled by Gamma distributions with identical mean output lengths\. The shaded regions indicate 95% confidence intervals over three runs\. The optimalθ∗\\theta^\{\*\}shifts from 0\.16 \(CFR\) to 0\.36 \(IFR\), consistent withΔ​θ\>0\\Delta\\theta\>0in Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)\.
### E\.4Statistical Tests for IFR in Real Workloads

Section[3\.2](https://arxiv.org/html/2606.00516#S3.SS2)\(Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)\) and Appendix[E\.1](https://arxiv.org/html/2606.00516#A5.SS1)report empirical hazard ratesh^​\(t\)\\hat\{h\}\(t\)that visually appear monotonically increasing for our real\-world workloads\. Here we provide formal statistical tests of CFR vs\. IFR\. We apply two complementary tests on the empiricalh^​\(t\)\\hat\{h\}\(t\)for each workload, restricted to the reliable regiont∈\[1,t95\]t\\in\[1,t\_\{95\}\]wheret95t\_\{95\}is the 95th percentile of output length: \(i\) theMann–Kendallnon\-parametric trend test \(H0H\_\{0\}: no monotonic trend\); \(ii\) a parametriclinear regressionh​\(t\)=p0\+η​th\(t\)=p\_\{0\}\+\\eta t\(H0H\_\{0\}:η=0\\eta=0\)\.

Table 12:Statistical tests for IFR on real\-world workloads\. All three reject CFR \(H0H\_\{0\}:η=0\\eta=0or no monotonic trend\) atp<10−5p<10^\{\-5\}under both tests, supporting Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)’s IFR assumption\. LongBench is excluded asμO≈12\\mu\_\{O\}\\approx 12is too short for hazard\-rate analysis\.
### E\.5Sensitivity to Hazard\-Rate Estimation Accuracy

Theorem[3\.2](https://arxiv.org/html/2606.00516#S3.Thmtheorem2)and Algorithm[1](https://arxiv.org/html/2606.00516#alg1)estimatep^0\\hat\{p\}\_\{0\}online from a sliding window of recent completions; if the estimate drifts during transient periods, the resultingθ^∗\\hat\{\\theta\}^\{\*\}may move off the true optimum\. We characterize this sensitivity by sweeping fixedθ∗∈\[0\.1,0\.9\]\\theta^\{\*\}\\in\[0\.1,0\.9\]and comparing against the adaptive controller on Qwen3\-8B, RTX PRO 6000, with 3,000 requests per configuration\.

Table[13](https://arxiv.org/html/2606.00516#A5.T13)reports throughput under \(i\) a synthetic Gamma\(a=2a\{=\}2\) workload withμO=256\\mu\_\{O\}=256, and \(ii\) ShareGPT \(μO≈280\\mu\_\{O\}\\approx 280\)\. Two observations support robustness to estimation error:\(1\) Wide throughput plateau\.The plateau spansθ∗∈\[0\.6,0\.9\]\\theta^\{\*\}\\in\[0\.6,0\.9\]on Gamma \(≥87%\\geq 87\\%of peak\) and\[0\.5,0\.8\]\[0\.5,0\.8\]on ShareGPT \(≥95%\\geq 95\\%\)\. The optimalθ∗\\theta^\{\*\}shifts across workloads \(0\.7 vs\. 0\.6\), but any choice within the plateau yields near\-peak throughput, so moderate estimation error has limited impact\.\(2\) Adaptive estimator stays in the plateau\.The online controller achieves98% of peakon both workloads\. Onlinep^0\\hat\{p\}\_\{0\}shows2\.8×10−42\.8\\times 10^\{\-4\}absolute error \(15% relative\) on Gamma versus7×10−67\\times 10^\{\-6\}\(0\.5% relative\) on ShareGPT—a30×30\\timesgap in relative error—yet both attain98%98\\%peak\. Combined with the threshold clippingθ^∗∈\[θmin,θmax\]\\hat\{\\theta\}^\{\*\}\\in\[\\theta\_\{\\min\},\\theta\_\{\\max\}\]and vLLM’s preemption mechanism, this provides defense\-in\-depth against transient mis\-estimation\.

Table 13:Sensitivity to hazard\-rate estimation accuracy\. “Adaptive” uses the online controller \(Algorithm[1](https://arxiv.org/html/2606.00516#alg1)\); other columns sweep fixedθ∗\\theta^\{\*\}\. Throughput in tok/s; “% of peak” below each row\.

## Appendix FAdditional End\-to\-End Comparisons

### F\.1Sensitivity Analysis Across Workloads and Platforms

This appendix provides comprehensive sensitivity analyses for all workloads \(ShareGPT, LongBench, WildChat, NuminaMath\) across H200 and RTX PRO 6000 platforms\. For each workload\-platform combination, we perform a full grid search over token budgetBBand batch sizeNNfor v0, v1, and EB\(k^∗\\hat\{k\}^\{\*\}\) schedulers\. Tables[14](https://arxiv.org/html/2606.00516#A6.T14)and[15](https://arxiv.org/html/2606.00516#A6.T15)below report four robustness metrics: coefficient of variation \(CV\), range ratio,BB\-Sens, andNN\-Sens, with lower values indicating better stability across configurations\.

#### F\.1\.1Parameter Sensitivity Metrics

We define the following metrics to quantify scheduling robustness across deployment configurations\. LetTP​\(B,N\)\\mathrm\{TP\}\(B,N\)denote the measured throughput \(requests/second\) for a scheduler under token budgetBBand batch sizeNN, and letℬ\\mathcal\{B\}and𝒩\\mathcal\{N\}be the sets of evaluated token budgets and batch sizes, respectively\.

Coefficient of Variation \(CV\)\.The CV measures overall throughput variability across all configurations:

CV=std\(TP\(B,N\):B∈ℬ,N∈𝒩\)mean\(TP\(B,N\):B∈ℬ,N∈𝒩\)\\text\{CV\}=\\frac\{\\mathrm\{std\}\(\\mathrm\{TP\}\(B,N\):B\\in\\mathcal\{B\},N\\in\\mathcal\{N\}\)\}\{\\mathrm\{mean\}\(\\mathrm\{TP\}\(B,N\):B\\in\\mathcal\{B\},N\\in\\mathcal\{N\}\)\}wherestd​\(⋅\)\\mathrm\{std\}\(\\cdot\)andmean​\(⋅\)\\mathrm\{mean\}\(\\cdot\)denote standard deviation and mean, respectively\. Lower values indicate greater stability\.

Range Ratio\.The range ratio captures the extremes of performance variation:

Range=maxB,N⁡TP​\(B,N\)minB,N⁡TP​\(B,N\)\\text\{Range\}=\\frac\{\\max\_\{B,N\}\\mathrm\{TP\}\(B,N\)\}\{\\min\_\{B,N\}\\mathrm\{TP\}\(B,N\)\}Values closer to 1\.0 indicate more consistent performance across configurations\.

Token Budget Sensitivity \(BB\-Sens\)\.This metric isolates the effect of varyingBBwhile holdingNNfixed:

B​\-Sens=1\|𝒩\|​∑N∈𝒩std\(TP\(B,N\):B∈ℬ\)mean\(TP\(B,N\):B∈ℬ\)B\\text\{\-Sens\}=\\frac\{1\}\{\|\\mathcal\{N\}\|\}\\sum\_\{N\\in\\mathcal\{N\}\}\\frac\{\\mathrm\{std\}\(\\mathrm\{TP\}\(B,N\):B\\in\\mathcal\{B\}\)\}\{\\mathrm\{mean\}\(\\mathrm\{TP\}\(B,N\):B\\in\\mathcal\{B\}\)\}
Batch Size Sensitivity \(NN\-Sens\)\.This metric isolates the effect of varyingNNwhile holdingBBfixed:

N​\-Sens=1\|ℬ\|​∑B∈ℬstd\(TP\(B,N\):N∈𝒩\)mean\(TP\(B,N\):N∈𝒩\)N\\text\{\-Sens\}=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{B\\in\\mathcal\{B\}\}\\frac\{\\mathrm\{std\}\(\\mathrm\{TP\}\(B,N\):N\\in\\mathcal\{N\}\)\}\{\\mathrm\{mean\}\(\\mathrm\{TP\}\(B,N\):N\\in\\mathcal\{N\}\)\}

#### F\.1\.2Optimal Configuration Details

Tables[14](https://arxiv.org/html/2606.00516#A6.T14)and[15](https://arxiv.org/html/2606.00516#A6.T15)report the optimal token budget \(BB\) and batch size \(NN\) for each scheduler across all workloads on RTX PRO 6000 and H200 respectively\. These configurations were selected via grid search to maximize throughput\.

Table 14:Optimal configurations and sensitivity metrics for each scheduler across workloads on RTX PRO 6000\. Lower sensitivity values indicate greater robustness\.Table 15:Optimal configurations and sensitivity metrics for each scheduler across workloads on H200\. Lower sensitivity values indicate greater robustness\.Configuration variability\.Optimal configurations vary substantially across workloads, models, and hardware\. Token budgets range from 4,096 to 18,432, while batch sizes span 256 to 2,048\. No single configuration universally dominates, highlighting the importance of workload\-aware tuning\.

Sensitivity patterns\.Prefill\-heavy workloads \(LongBench\) exhibit the lowest sensitivity \(CV<<5%, range ratio<<1\.2×\\times\), as throughput is dominated by prefill computation with limited scheduling flexibility\. In contrast, balanced workloads \(ShareGPT\) show higher sensitivity, particularly for larger models \(Qwen3\-30B\-A3B: CV up to 35%, range ratio up to 3\.47×\\times\), where the prefill–decode balance is more delicate\. Decode\-heavy workloads \(NuminaMath\) demonstrate moderate sensitivity, with most configurations achieving CV<<8%\.

Scheduler comparison\.All three schedulers exhibit comparable sensitivity profiles within each workload category\. EB\(k^∗\\hat\{k\}^\{\*\}\) does not introduce additional tuning burden: its average CV and range ratio are similar to v0 and v1 across most configurations\. The primary differentiator remains throughput rather than robustness\.

#### F\.1\.3Optimal Configurations for Synthetic Workloads \(Figure[4](https://arxiv.org/html/2606.00516#S4.F4)\)

Tables[14](https://arxiv.org/html/2606.00516#A6.T14)and[15](https://arxiv.org/html/2606.00516#A6.T15)cover the four real\-world workloads\. The three synthetic workloads used in Figure[4](https://arxiv.org/html/2606.00516#S4.F4)\(decode\-heavy, balanced, prefill\-heavy\) have separate optima, obtained from the same grid search overB∈\{4096,…,18432\}B\\in\\\{4096,\\dots,18432\\\}andN∈\{256,…,2048\}N\\in\\\{256,\\dots,2048\\\}and reported in Table[16](https://arxiv.org/html/2606.00516#A6.T16)\.

Table 16:Best\-throughput\(B,N\)\(B,N\)per scheduler for each synthetic workload \(Qwen3\-8B\)\. These configurations underlie Figure[4](https://arxiv.org/html/2606.00516#S4.F4); selected from grid search\.

### F\.2End\-to\-End Results for Gemma\-3\-1B\-IT

Table[17](https://arxiv.org/html/2606.00516#A6.T17)reports end\-to\-end throughput for Gemma\-3\-1B\-IT on both GPUs\. The 1B\-parameter model sits at a different operating point than the Qwen models in the main text: its lightweight Attention avoids bandwidth\-induced interference \(Appendix[A\.2](https://arxiv.org/html/2606.00516#A1.SS2)\), so the marginal\-cost gapβMBe−βEBw\\beta\_\{\\mathrm\{MB\}\}^\{e\}\-\\beta\_\{\\mathrm\{EB\}\}^\{w\}on the LHS of \([7](https://arxiv.org/html/2606.00516#S3.E7)\) is small\. The RHS is also small at this scale because per\-iteration fixed overheadα\\alphascales with model size\. Both sides shrink together, leaving a narrow but consistent margin in EB’s favor\. On the RTX PRO 6000, EB\(k^∗\\hat\{k\}^\{\*\}\) achieves consistent improvements over v1 across all workloads \(average\+\+3\.9%\), primarily from better scheduling of decode\-heavy phases\. On the H200, the advantage is larger on ShareGPT \(\+\+10\.6%\) and LongBench \(\+\+4\.4%\), while WildChat shows a slight regression \(−\-1\.5%\)\.

Table 17:Throughput \(RPS\) for Gemma\-3\-1B\-IT on real\-world workloads\. % Diff\. as defined in Table[2](https://arxiv.org/html/2606.00516#S4.T2)\.
### F\.3SLO\-Constrained Goodput for EB\+

Section[4\.4](https://arxiv.org/html/2606.00516#S4.SS4)summarizes the SLO\-attainment behavior of EB\+at moderate load\. Here we report per\-request SLO attainment \(fraction of requests meeting both TTFT and TPOT targets simultaneously\) on RTX PRO 6000 with Qwen3\-8B atc=512c=512\. Under a strict TPOT target \(<50<\\\!50ms\), all schedulers fail because mean TPOT exceeds the target on this bandwidth\-constrained GPU\. Under a relaxed target \(<100<\\\!100ms\), v1 collapses to∼5%\\sim\\\!5\\%attainment because its TPOT distribution is wide; EB\+achieves the best balance, attaining80\.3%80\.3\\%at TTFT<10<\\\!10s and48\.4%48\.4\\%at TTFT<5<\\\!5s\. On H200 atc=512c=512, mean TPOT is universally low \(<53<\\\!53ms\) so TTFT dominates the SLO; EB\+selects v1 there and matches its attainment at all targets\.

Table 18:SLO attainment \(% of requests meeting both TTFT and TPOT targets\) on RTX PRO 6000 \(Qwen3\-8B,c=512c=512\)\. Under strict TPOT all schedulers fail; under relaxed TPOT, EB\+dominates at moderate\-to\-loose TTFT targets\.
### F\.4Comparison with Prefill–Decode Disaggregation

Disaggregated serving \(DistServe\(Zhonget al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib20)\), Splitwise\(Patelet al\.,[2024](https://arxiv.org/html/2606.00516#bib.bib21)\)\) is an alternative to MB that physically separates prefill and decode onto dedicated GPU pools\. We compare EB\+\(which separates phases temporally on the same GPUs under data parallelism\) against vLLM’s built\-in disaggregation scheduler\.

Two\-GPU setting \(DP=2 vs\. 1P\+1D\)\.Table[19](https://arxiv.org/html/2606.00516#A6.T19)compares v1, EB\+, and vLLM’s 1P\+1D disaggregation scheduler on2×2\\timesRTX PRO 6000 and2×2\\timesH200 \(Qwen3\-8B,μL=512\\mu\_\{L\}=512,μO=256\\mu\_\{O\}=256\)\. EB\+matches or exceeds v1 throughput at every concurrency on both GPUs\. On RTX PRO 6000, EB\+outperforms disaggregation by\+31\.8%\+31\.8\\%throughput with3\.3×3\.3\\timeslower TTFT atc=512c=512and by\+22\.8%\+22\.8\\%over v1 atc=2048c=2048\. On H200, EB\+leads disaggregation by\+17\.7%\+17\.7\\%atc=64c=64and\+14\.8%\+14\.8\\%atc=512c=512, with\+4\.3%\+4\.3\\%over v1 atc=2048c=2048\. Disaggregation OOMs atc=2048c=2048on both GPUs \(“–”\) because KV blocks remain pinned during prefill\-to\-decode transfer and the scheduler lacks admission backpressure\.

Table 19:Two\-GPU comparison: v1, EB\+\(DP=2\), and vLLM 1P\+1D disaggregation\. Throughput in tok/s\. “–” denotes OOM\.Four\-GPU setting across P:D ratios\.We further compare DP=4 against disaggregation with 1P\+3D, 2P\+2D, and 3P\+1D allocations on4×4\\timesRTX PRO 6000 and4×4\\timesH200 \(Qwen3\-8B, vLLM P2P NCCL KV transfer; “–” denotes OOM/timeout on the decode GPU\)\. Three workload mixes span the prefill–decode spectrum:*prefill\-heavy*\(μL=1024,μO=128\\mu\_\{L\}=1024,\\mu\_\{O\}=128\),*balanced*\(μL=512,μO=512\\mu\_\{L\}=512,\\mu\_\{O\}=512\), and*decode\-heavy*\(μL=128,μO=1024\\mu\_\{L\}=128,\\mu\_\{O\}=1024\); 2k requests per setting atc∈\{128,256,512\}c\\in\\\{128,256,512\\\}\.

Three observations emerge from Table[20](https://arxiv.org/html/2606.00516#A6.T20):

\(1\) The optimal P:D ratio is workload\-dependent and choosing wrong is costly\.Atc=128c=128on4×4\\timesRTX PRO 6000: prefill\-heavy favors 2P\+2D \(\+63%\+63\\%over EB\+\), balanced favors 1P\+3D \(\+14%\+14\\%\), and decode\-heavy favors EB\+itself\. No single ratio dominates: 2P\+2D is best for prefill\-heavy but worst for decode\-heavy among disagg options; 1P\+3D is best for balanced but worst for prefill\-heavy\.

\(2\) Disaggregation exhibits structural memory fragility\.Under disaggregation, a request’s entire prefill KV cache \(LLtokens\) is transferred to the decode GPU in bulk upon prefill completion\. Multiple prefill GPUs produce completed requests faster than a single decode GPU can retire them, causing concurrent KV occupancy on the decode GPU to scale asO​\(L⋅npending\)O\(L\\,\\cdot\\,n\_\{\\text\{pending\}\}\)with no backpressure mechanism\. This explains the OOM pattern: atc=256c=256, 3P\+1D OOMs on prefill\-heavy \(L=1024L=1024\) but not on decode\-heavy \(L=128L=128\)—the same number of pending requests consumes8×8\\timesmore KV memory\. Atc=512c=512, 3P\+1D OOMs on balanced \(L=512L=512\) as well; atc=2048c=2048\(Appendix[F\.4](https://arxiv.org/html/2606.00516#A6.SS4)\) all disagg configurations OOM\.

\(3\) EB\+is consistently competitive without manual tuning\.EB\+achieves the best or near\-best throughput in 7 of 9 \(workload×\\timescc\) settings on RTX PRO 6000, with 2P\+2D excelling on prefill\-heavy atc=128c=128andc=256c=256\. Crucially, EB\+maintains33–18×18\\timeslower TTFT than every disagg configuration across every setting and never OOMs\. In a deployment where workload mix is unknown or shifts over time, EB\+provides robust performance without operators predicting the optimal P:D ratio\.

Table 20:4\-GPU disaggregation comparison across concurrenciesc∈\{128,256,512\}c\\in\\\{128,256,512\\\}\(Qwen3\-8B, 2k prompts\)\. Throughput in tok/s\. “–” denotes OOM/timeout\. v1 and EB\+both use DP=4\.4×4\\timesRTX PRO 60004×4\\timesH200ccWorkloadMetricv1EB\+1P\+3D2P\+2D3P\+1Dv1EB\+1P\+3D2P\+2D3P\+1D128Prefill\-heavyThroughput25,99426,20723,24542,57731,49067,27865,29740,53582,40070,255TTFT \(ms\)1851714,3854764821331242,763662412TPOT \(ms\)43\.042\.814\.623\.132\.616\.116\.76\.48\.513\.1BalancedThroughput11,01311,06312,62911,6409,02328,64628,68631,34129,38821,963TTFT \(ms\)1211163162292727574249178233TPOT \(ms\)22\.622\.519\.221\.127\.38\.68\.67\.58\.211\.0Decode\-heavyThroughput7,7237,6907,4696,8485,45018,24519,33618,86117,22713,536TTFT \(ms\)65622231662204644233179220TPOT \(ms\)18\.118\.218\.620\.425\.67\.67\.27\.27\.910\.2256Prefill\-heavyThroughput37,62337,20823,10244,892–88,48887,26840,63877,117–TTFT \(ms\)38038110,2833,369–3043076,1032,614–TPOT \(ms\)57\.858\.514\.523\.3–23\.423\.86\.48\.3–BalancedThroughput16,91316,87019,83117,55111,40243,64843,72947,43342,01626,455TTFT \(ms\)235229844505664158152681395596TPOT \(ms\)28\.628\.722\.926\.942\.110\.910\.98\.910\.917\.6Decode\-heavyThroughput12,67812,69912,17610,5957,02130,69629,76329,49524,82016,682TTFT \(ms\)125119719404592108110715438599TPOT \(ms\)21\.421\.321\.725\.438\.98\.78\.98\.510\.616\.0512Prefill\-heavyThroughput47,94747,32222,53143,326–106,324105,43239,92578,486–TTFT \(ms\)1,1411,20521,4129,245–80584312,3575,741–TPOT \(ms\)85\.386\.114\.623\.9–36\.236\.26\.48\.2–BalancedThroughput23,91424,03725,08020,397–61,10660,98846,26148,895–TTFT \(ms\)6976974,0131,676–4644135,7531,128–TPOT \(ms\)38\.738\.629\.943\.4–14\.714\.99\.217\.3–Decode\-heavyThroughput18,41418,39617,22413,1697,57946,62246,78238,28730,56717,544TTFT \(ms\)3192841,8421,0742,2952513141,9081,2161,720TPOT \(ms\)28\.228\.228\.839\.169\.410\.910\.811\.916\.229\.4