The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

# The Inference Engineering Pareto Atlas:Which Optimizations Dominate theCost, Quality, and Latency Frontier?
Source: [https://arxiv.org/html/2609.17863](https://arxiv.org/html/2609.17863)
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly

###### Abstract

Every optimization for LLM inference arrives with its own speedup number, and the numbers do not compare\. Quantization papers report one gain, KV\-cache compression papers another, and speculative decoding, batching, and sparse attention each report their own, nearly always on different models, GPUs, prompts, and quality metrics\. A practitioner with a fixed budget, quality floor, or latency target cannot add these numbers up, and cannot tell which combination wins once the methods are stacked\. This paper builds a cost\-quality\-latency Pareto atlas: every optimization and every promising combination on common axes, with the dominant configuration named in each deployment regime\. The configuration space is too large to measure exhaustively, so we measure anchors on real hardware and calibrate a profiled simulator to fill the rest\. The anchors are 54 configurations of Qwen2\.5\-7B\-Instruct served by vLLM 0\.12 on L4, A100, and H100 instances, 18 per GPU across three campaigns; the calibrated simulator reproduces them at the anchored batch sizes, and cross\-campaign drift is below 1\.5%\. A separate quality arm scores FP16, AWQ\-4bit, FP8 weights, and an FP8 KV cache on GSM8K \(200 questions, 5\-shot\)\. Sparse attention is the one method we could only simulate\. On the calibrated grid, 18 of 36 configurations reach the frontier, and stacked combinations reach it more often than single methods \(9 of 15 against 9 of 21\)\. Measuring quality changes who wins\. AWQ\-4bit buys the most speed, down to 0\.34×\\timesper\-token latency on the L4, but loses 5\.9% of strict GSM8K accuracy and misses the 95% quality floor by a margin inside the sampling error; the loss is in answer formatting rather than arithmetic, since flexible extraction ties the FP16 baseline\. FP8 weight quantization keeps 99\.4% of baseline accuracy at 0\.61 to 0\.65×\\timeslatency on all three GPUs and is part of three of the four regime winners\. A naive FP8 KV cache posts normal throughput and answers 0 of 200 questions correctly, a configuration a speed\-only benchmark would recommend and the atlas eliminates\. Under two prompt designs, n\-gram speculative decoding measures at 0\.90 to 0\.98×\\timesand adds nothing on this stack\. The winner moves with the constraint and the GPU: the H100 takes the tight\-latency regime, and the A100 takes throughput and low cost at $0\.106 per million tokens\.

\\@IEEEabskeysecsize

Index Terms:\\@IEEEgobbleleadPARNLSPLLM inference, Pareto frontier, cost\-quality\-latency, quantization, speculative decoding, serving, benchmarking, optimization atlas\.

## IIntroduction

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig_overview.png)Fig\. 1:Study overview\. \(1\) The field reports many isolated speedups under incompatible settings, so they cannot be compared or combined\. \(2\) A practitioner under a fixed cost, quality, or latency constraint needs to know which optimization or combination dominates\. \(3\) We place every method and promising combination on common cost\-quality\-latency axes, compute the Pareto frontier, and build an atlas of which operating points dominate in each regime\.Every month brings new ways to make LLM inference cheaper or faster: low\-bit quantization\[[7](https://arxiv.org/html/2609.17863#bib.bib7),[8](https://arxiv.org/html/2609.17863#bib.bib8),[9](https://arxiv.org/html/2609.17863#bib.bib9)\], KV\-cache compression and eviction\[[10](https://arxiv.org/html/2609.17863#bib.bib10),[11](https://arxiv.org/html/2609.17863#bib.bib11)\], speculative decoding\[[6](https://arxiv.org/html/2609.17863#bib.bib6)\], continuous and SLO\-aware batching\[[3](https://arxiv.org/html/2609.17863#bib.bib3),[4](https://arxiv.org/html/2609.17863#bib.bib4)\], and sparse attention\[[12](https://arxiv.org/html/2609.17863#bib.bib12)\]\. Each is backed by a paper reporting a substantial gain\. The numbers are not comparable\. They come from different models, different GPUs, different prompts, and different quality metrics, so a “three times faster” in one paper and a “two times faster” in another cannot be added or compared, and neither can be trusted to reproduce on a given deployment \(Fig\.[2](https://arxiv.org/html/2609.17863#S1.F2)\)\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig1_frontier.png)Fig\. 2:On common axes, most published operating points are dominated\. The Pareto frontier, the set of configurations that no other configuration beats on both quality and latency, is what a practitioner should choose from\. Calibrated grid; anchors measured on RunPod instances\.This makes practical decisions hard\. An engineer who needs the lowest latency at a fixed quality, or the lowest cost at a fixed latency, cannot read the papers and pick a winner, because the methods are not on the same axes and their interactions are unknown\. And there is no neutral ground for a new method to compare against, so claims drift apart over time\. The central question is: across the full optimization stack and across hardware, which configurations sit on the cost\-quality\-latency Pareto frontier, and which one should be chosen under a given constraint?

The obstacle is the lack of a common map\. There are good benchmarks for model*quality*and good benchmarks for raw serving*throughput*\[[1](https://arxiv.org/html/2609.17863#bib.bib1),[13](https://arxiv.org/html/2609.17863#bib.bib13)\], and recent work begins to frame inference economics on a cost\-quality frontier\[[14](https://arxiv.org/html/2609.17863#bib.bib14)\]\. There is no single*atlas*that jointly places quantization, KV compression, speculation, batching, and sparse attention, alone and combined, on the three axes a deployment cares about, and that names the dominant choice in each regime\. This paper builds that atlas\.

We measure 54 anchor configurations of Qwen2\.5\-7B\-Instruct on one serving workload \(512\-token prompts, 128\-token outputs\) across an L4, an A100, and an H100, reporting quality, time\-to\-first\-token, time\-per\-output\-token, throughput, and cost per million tokens, with a GSM8K arm for quality at each precision\. Because the configuration space \(method by combination by hardware\) is far too large to measure directly, we calibrate a profiled simulator\[[2](https://arxiv.org/html/2609.17863#bib.bib2)\]against the anchors and use it to fill the space, then compute the Pareto frontier and the dominant operating points under tight\-latency, high\-throughput, and low\-cost constraints\.

Our contributions are as follows\.

1. 1\.A cost\-quality\-latency Pareto atlas \(Fig\.[5](https://arxiv.org/html/2609.17863#S3.F5)\) that places the major inference optimizations and their combinations on common axes and names the dominant configuration in each regime\.
2. 2\.A measure\-then\-simulate methodology \(Fig\.[4](https://arxiv.org/html/2609.17863#S3.F4)\) that calibrates a profiled simulator against real runs so the full configuration space can be covered from measured anchors\.
3. 3\.A set of practitioner rules: which optimizations recur on the frontier, which combinations dominate, and how the winning choice shifts with the binding constraint and the GPU\.

## IIBackground and Related Work

### II\-AThe optimization stack

The methods we place on the atlas target different costs\. Quantization\[[7](https://arxiv.org/html/2609.17863#bib.bib7),[8](https://arxiv.org/html/2609.17863#bib.bib8),[9](https://arxiv.org/html/2609.17863#bib.bib9)\]and KV\-cache compression or eviction\[[10](https://arxiv.org/html/2609.17863#bib.bib10),[11](https://arxiv.org/html/2609.17863#bib.bib11)\]shrink memory and bandwidth; speculative decoding\[[6](https://arxiv.org/html/2609.17863#bib.bib6)\]cuts decode steps; continuous batching\[[3](https://arxiv.org/html/2609.17863#bib.bib3)\]and SLO\-aware scheduling\[[4](https://arxiv.org/html/2609.17863#bib.bib4)\]raise utilization; sparse attention\[[12](https://arxiv.org/html/2609.17863#bib.bib12)\]cuts attention compute; prefill\-decode disaggregation\[[5](https://arxiv.org/html/2609.17863#bib.bib5)\]separates the two phases\. Each exposes a knob and each shifts a different axis; only a common frontier makes them comparable\.

### II\-BBenchmarks and economics

MLPerf Inference\[[1](https://arxiv.org/html/2609.17863#bib.bib1)\]standardizes serving measurement and Spec\-Bench\[[13](https://arxiv.org/html/2609.17863#bib.bib13)\]standardizes speculation, but neither places the full optimization stack on a joint cost\-quality\-latency frontier\. Recent work studies inference economics and the cost\-quality frontier for model choice\[[14](https://arxiv.org/html/2609.17863#bib.bib14)\]and broad local\-inference benchmarking\[[15](https://arxiv.org/html/2609.17863#bib.bib15)\]\. Our atlas adds the optimization and hardware axes and the explicit dominance map\.

### II\-CSimulation for large configuration spaces

The configuration space is combinatorial, so exhaustive measurement is infeasible\. Vidur\[[2](https://arxiv.org/html/2609.17863#bib.bib2)\]shows that a profiled simulator can estimate LLM\-inference latency and throughput within a few percent and search hundreds of deployment configurations cheaply\. We adopt this measure\-then\-simulate strategy: measure anchors on real hardware, calibrate the simulator, and use it to fill the frontier\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig5_dominance.png)Fig\. 3:Different optimizations sit on the frontier in different regimes, so how often each appears on the Pareto frontier across constraints is more informative than any single speedup number\. This is why the atlas reports a regime map\. Calibrated grid, measured anchors\.
### II\-DPareto dominance

A configuration is*dominated*if another configuration is at least as good on every axis and strictly better on one\. The*Pareto frontier*is the set of non\-dominated configurations; it is the only set a rational practitioner should choose from, and the choice within it is set by the binding constraint\. The atlas is, formally, the frontier plus a labeling of which configuration is optimal under each constraint\.

### II\-EOur position

We treat comparability and dominance as the deliverable: one protocol, common axes, every optimization and combination placed, the simulator calibrated against measured anchors, and the frontier and dominance map reported\. The output is a map a practitioner reads when picking an optimization\.

## IIIMethodology

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig_method.png)Fig\. 4:The measure\-then\-simulate atlas pipeline\. A fixed set of models, workloads, and GPUs is run with each optimization to produce measured anchors \(quality, TTFT, TPOT, throughput, cost\)\. A profiled simulator is calibrated against these anchors, then used to fill the large configuration space \(optimization by combination by hardware\)\. For each model and workload the Pareto frontier over cost, quality, and latency is computed, and the dominant configuration under each constraint is labeled, producing the atlas\.### III\-AAxes and metrics

Every configuration is placed on three axes:*quality*\(task accuracy, or agreement with the full\-precision model\),*latency*\(TTFT and TPOT, reported at P50 and P99\), and*cost*\(dollars per million tokens, derived from throughput and the GPU hourly price\)\. Throughput and GPU memory are recorded as supporting metrics\. Quality is always reported alongside speed so no configuration wins by silently degrading answers\.

### III\-BConfigurations and combinations

A configuration is a choice of optimization settings on a given model and GPU: a quantization bit\-width, a KV keep\-ratio or bit\-width, a speculation draft, a batching policy, a sparse\-attention budget\. Single\-method configurations are the building blocks; combinations stack compatible methods \(for example speculation over a quantized KV cache under SLO\-aware batching\)\. The combination space is large, which motivates the simulator\.

### III\-CMeasure then simulate

We measure anchor configurations directly on each GPU to capture real kernel and memory behavior, then calibrate a profiled simulator\[[2](https://arxiv.org/html/2609.17863#bib.bib2)\]so its latency and throughput predictions match the anchors within a small tolerance\. The calibrated simulator fills the rest of the space\. Algorithm[1](https://arxiv.org/html/2609.17863#alg1)states the construction\.

Algorithm 1Building the Pareto atlasInput: models, workloads, GPUs, optimization configs, anchors

measure anchor configs on real hardware; record quality, latency, cost

calibrate the profiled simulator against the anchors

simulate the full config and combination space

for each \(model, workload\) do compute the Pareto frontier over \(quality, latency, cost\)

label the dominant config under tight\-latency, high\-throughput, low\-cost

return the frontier and the dominance atlas

### III\-DReading the atlas

The atlas \(Fig\.[5](https://arxiv.org/html/2609.17863#S3.F5)\) is a grid whose rows are deployment regimes \(the binding constraint\) and whose columns are the axes or workloads; each cell names the dominant configuration\. Three expectations are under test: that a small number of combinations recur on the frontier, that the winner shifts with the binding constraint \(low cost toward aggressive quantization, tight latency toward speculation, high throughput toward batching\), and that the GPU choice moves the whole frontier\. Section[V](https://arxiv.org/html/2609.17863#S5)reports which hold\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig3_atlas.png)Fig\. 5:The atlas: the dominant configuration in each deployment regime\. The boxed cells mark where a stacked combination beats every single method\. Calibrated grid, measured anchors\.
### III\-ECost accounting

Cost per million tokens depends on the GPU hourly price and the measured throughput, both of which we state\. Simulated points carry the simulator’s calibrated error bar; measured anchors do not\. We never present a simulated latency as if it were measured, and every frontier marks which points are anchors\.

## IVExperimental Setup

### IV\-AModels, workloads, hardware, and metrics

The measured campaigns anchor one model, Qwen2\.5\-7B\-Instruct, on one serving workload \(512\-token prompt, 128\-token output\) across three GPUs, a RunPod L4, an A100 80GB PCIe, and an H100 PCIe, all served by vLLM 0\.12, with quality measured on GSM8K\. Table[I](https://arxiv.org/html/2609.17863#S4.T1)lists the protocol’s four workloads; of these, only Math \(GSM8K\) is measured here, as the quality arm\. The metric panel is quality, TTFT, TPOT \(P50 and P99\), throughput, GPU memory, and cost per million tokens, every number reported with the model, GPU, batch size, and workload stated\.

TABLE I:Protocol workloads and the cost each stresses; only Math \(GSM8K\) is measured here\.
### IV\-BOptimizations and protocol

The optimizations are quantization \(KIVI, TurboQuant, AWQ\), KV compression and eviction \(PyramidKV, H2O\), speculation \(EAGLE\-3\), continuous and SLO\-aware batching \(vLLM, Sarathi\), and sparse attention \(NSA\), each a configurable knob, plus their compatible combinations\. Of these, the campaigns anchor quantization as AWQ\-4bit and online FP8 weights, KV compression as the FP8 KV cache, and speculation as n\-gram prompt\-lookup \(draft\-model speculation is unimplemented in the serving engine used, and no EAGLE head exists for the target model\); sparse attention is simulated only\. Anchors are measured on real GPUs; the rest is simulated with a calibrated profiler\. Principal settings are in Table[II](https://arxiv.org/html/2609.17863#S4.T2)\.

TABLE II:Principal settings\.

## VResults

All latency, throughput, and cost numbers in this section derive from 54 measured anchor configurations \(July 2026 campaigns: RunPod L4 $0\.39/hr, A100 80GB PCIe $1\.39/hr, H100 PCIe $2\.89/hr; vLLM 0\.12\.0, Qwen2\.5\-7B\-Instruct, 512\-token prompts, 128\-token outputs, exact token\-count contract\), the simulator calibrated against them \(Fig\.[10](https://arxiv.org/html/2609.17863#S5.F10)\), and the measured GSM8K quality arm\. Three campaigns were run: synthetic\-prompt, realistic extractive prompt, and the final six\-arm matrix \(FP16, AWQ, FP8 weights, FP8 KV cache, and two speculation arms\) plus the GSM8K quality stage\. Anchors are cross\-campaign consistent within 1\.5%\. Total spend was $18\.

TABLE III:Dominant configuration per constraint \(calibrated grid; quality measured on GSM8K\)\.### V\-AThe quality\-latency frontier

Figure[2](https://arxiv.org/html/2609.17863#S1.F2)plots every configuration on quality versus latency, with the Pareto frontier highlighted and dominated points faded\. On the calibrated grid, 18 of 36 configurations are non\-dominated; the rest are strictly beaten\. The frontier’s zero\-quality corner is the collapsed FP8\-KV\-cache family, which is fastest on raw dominance and so survives the frontier, yet is excluded from every deployment regime by the quality guards; the raw frontier and the atlas answer different questions by design\.

### V\-BThe cost\-quality frontier

Cost per million tokens is the axis a budget owner cares about, and Fig\.[6](https://arxiv.org/html/2609.17863#S5.F6)re\-plots the frontier against it\. The FP8 \+ batching \+ sparse stack anchors the low\-cost end at $0\.106/Mtok on the A100; AWQ\-4bit is cheaper still but falls below the measured quality floor, and the KV\-cache arm collapses to zero quality outright\. The low\-latency end belongs to quantized sparse stacks on the H100; speculation measured at parity with baseline under both prompt designs\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig2_cost_quality.png)Fig\. 6:Quality versus cost per million tokens; up and to the left is better\. Calibrated grid, measured anchors\.
### V\-CThe atlas

Figure[5](https://arxiv.org/html/2609.17863#S3.F5), the paper’s central result, gives the dominant configuration in each deployment regime\. The winner shifts with the constraint: the H100 takes tight latency, and the A100 takes throughput, cost, and the quality floor\. The FP8\-weight stack appears in three of four regime winners; measured quality made it the workhorse\. No L4 configuration meets the tight\-latency budget at all, and every AWQ configuration misses the 95% quality floor by a hair \(0\.720 vs 0\.726, within the±\\pm3 pp sampling error ofn=200n\{=\}200, a boundary case the atlas puts on the record\)\.

### V\-DHow the frontier moves with hardware

Because the right optimization depends on the accelerator, Fig\.[7](https://arxiv.org/html/2609.17863#S5.F7)shows the frontier for each of the three GPUs\. The anchors confirm that the AWQ\-4bit speedup grows as the GPU gets cheaper \(0\.53×\\timesTPOT ratio on H100, 0\.49×\\timeson A100, 0\.34×\\timeson L4\), while FP8 weights hold a flat 0\.61 to 0\.65×\\timeseverywhere\. Memory\-bound budget silicon benefits most from aggressive weight compression, and FP8 is the portable default across all three GPUs\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig4_hardware.png)Fig\. 7:The quality\-latency frontier across GPU types\. Calibrated grid, measured anchors\.
### V\-EWhich optimizations recur, and which combinations win

Figure[3](https://arxiv.org/html/2609.17863#S2.F3)counts how often each optimization appears on the frontier, and Fig\.[8](https://arxiv.org/html/2609.17863#S5.F8)contrasts single methods with stacked combinations\. Quantization appears in 13 of 18 frontier members, speculation and sparsity in 6 each, KV compression in 4, batching in 3; 9 of 15 stacked configurations reach the frontier versus 9 of 21 single\-method configurations \(60% vs 43%\)\. Stacking still pays, though the measured quality arm removed the KV\-bearing stacks from every regime even where they keep frontier seats on speed alone\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig6_combo.png)Fig\. 8:Single methods versus stacked combinations on the frontier; stacks reach it more often \(9 of 15 vs 9 of 21\)\. Calibrated grid, measured anchors\.
### V\-FThe measured quality axis

The GSM8K arm \(200 questions, 5\-shot, greedy\) measures each precision: FP16 scores 0\.765 strict / 0\.800 flexible; AWQ\-4bit 0\.720 / 0\.810, where the strict drop reflects format discipline rather than arithmetic, since flexible extraction ties baseline; FP8 weights 0\.760 / 0\.805, effectively lossless; and the naive FP8 KV cache scores 0/200 on both metrics while sustaining normal measured throughput\. Throughput alone would endorse that last configuration even though it answers every question wrong; joint cost\-quality\-latency accounting removes it\.

### V\-GPicking under a constraint, and simulator fidelity

Figure[9](https://arxiv.org/html/2609.17863#S5.F9)reports the winning configuration under three fixed constraints, and Fig\.[10](https://arxiv.org/html/2609.17863#S5.F10)the simulator error against the measured anchors, the fidelity check behind every simulated point in the atlas\. Per\-arm batch curves reproduce all 54 anchors exactly at the anchored batch sizes \(the fit is interpolative by construction\); the realistic generalization estimate is therefore cross\-campaign anchor drift, below 1\.5% for every constant across three independent pod sessions on different physical hosts\.

![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig7_constraint.png)Fig\. 9:The winning configuration under tight\-latency, low\-cost, and high\-throughput constraints\. Calibrated grid, measured anchors\.![Refer to caption](https://arxiv.org/html/2609.17863v1/figures/fig8_fidelity.png)Fig\. 10:Calibrated\-simulator latency versus measured anchors; points near the diagonal validate the atlas\. Calibrated grid, measured anchors\.

## VIDiscussion and Limitations

This study uses open models, public workloads, and a calibrated profiled simulator, so the atlas is reproducible and the simulated points carry an explicit error bar against measured anchors\. The atlas is bounded by the optimizations, models, GPUs, and workloads covered, and by the price assumptions behind the cost axis, which we state so a reader can re\-cost for their own contract\. Absolute positions shift with kernel maturity and prices, so no single coordinate here will hold; the dominance structure \(which method wins in which regime\) and the protocol are what carry over\. The simulator is calibrated but not exact; every frontier marks its measured anchors, and any deployment decision near a frontier edge should be confirmed with a direct measurement\. Observed limitations from the measurement campaigns: \(1\) the quality arm is one task \(GSM8K,n=200n\{=\}200,±\\pm3 pp\) on one model; the AWQ\-vs\-floor verdict is a boundary case inside that error bar; \(2\) n\-gram speculation measured at parity \(0\.90 to 0\.98×\\times\) under two prompt designs, and draft\-model speculation could not be anchored, because vLLM 0\.12’s V1 engine raisesNotImplementedErrorfor it and no EAGLE head exists for Qwen2\.5\-7B\-Instruct; \(3\) the KV\-cache anchor is the naive FP8 path without calibrated scales; scale\-calibrated FP8\-KV or eviction methods \(H2O, PyramidKV\) may avoid the quality collapse we measured and are left for future anchors; \(4\) sparse attention has no stock\-vLLM implementation and is the one factor that is modeled rather than measured; \(5\) the calibration is interpolative at anchored batch sizes, so its generalization is bounded by the cross\-campaign drift \(1\.5%\), not the on\-anchor residual \(zero\); \(6\) prices are July\-2026 RunPod rates and shift with the market\.

## VIIConclusion

We argued that the field’s many isolated speedup claims cannot guide a real deployment, and built a cost\-quality\-latency Pareto atlas that places the full optimization stack and its combinations on common axes, calibrates a profiled simulator against measured anchors to cover the space, and names the dominant configuration in each regime\. A practitioner can read off the choice that fits a given budget, quality floor, or latency target\. On the measured grid, FP8 weight quantization is the quality\-preserving workhorse \(three of four regime winners\); AWQ buys more speed on budget silicon but falls at the measured quality floor; a naive FP8 KV cache is fast, looks normal, and is 0/200 correct; and speculation, as implementable on this stack, adds nothing\. A 50% price shock on the A100, the GPU the atlas as first built standardizes on, moves the cost and quality\-floor winners to the L4; re\-planning after a change like that is what the atlas exists for\. The durable deliverable is the protocol: anchors, calibration, frontier, and atlas, re\-run when prices and kernels change\.

## Acknowledgments

The authors thank Vizuara AI Labs for mentorship and computational resources\.

## References

- \[1\]V\. J\. Reddi et al\., “MLPerf inference benchmark,”Proc\. ISCA, 2020\. arXiv:1911\.02549\.
- \[2\]A\. Agrawal et al\., “Vidur: a large\-scale simulation framework for LLM inference,”Proc\. MLSys, 2024\. arXiv:2405\.05465\.
- \[3\]W\. Kwon et al\., “Efficient memory management for large language model serving with PagedAttention,”Proc\. SOSP, 2023\. arXiv:2309\.06180\.
- \[4\]A\. Agrawal et al\., “Taming throughput\-latency tradeoff in LLM inference with Sarathi\-Serve,”Proc\. OSDI, 2024\. arXiv:2403\.02310\.
- \[5\]Y\. Zhong et al\., “DistServe: disaggregating prefill and decoding for goodput\-optimized LLM serving,”Proc\. OSDI, 2024\. arXiv:2401\.09670\.
- \[6\]Y\. Li et al\., “EAGLE\-3: scaling up inference acceleration of LLMs via training\-time test,” arXiv:2503\.01840, 2025\.
- \[7\]Z\. Liu et al\., “KIVI: a tuning\-free asymmetric 2\-bit quantization for KV cache,”Proc\. ICML, 2024\. arXiv:2402\.02750\.
- \[8\]H\. Tang et al\., “TurboQuant: online vector quantization with near\-optimal distortion for KV cache,” arXiv:2504\.14492, 2025\.
- \[9\]J\. Lin et al\., “AWQ: activation\-aware weight quantization for LLM compression and acceleration,”Proc\. MLSys, 2024\. arXiv:2306\.00978\.
- \[10\]Z\. Cai et al\., “PyramidKV: dynamic KV cache compression based on pyramidal information funneling,” arXiv:2406\.02069, 2024\.
- \[11\]Z\. Zhang et al\., “H2O: heavy\-hitter oracle for efficient generative inference of large language models,”Proc\. NeurIPS, 2023\. arXiv:2306\.14048\.
- \[12\]J\. Yuan et al\., “Native sparse attention: hardware\-aligned and natively trainable sparse attention,” arXiv:2502\.11089, 2025\.
- \[13\]H\. Xia et al\., “Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding,”Findings of ACL, 2024\. arXiv:2401\.07851\.
- \[14\]R\. Desislavov et al\., “Trends in AI inference energy consumption: beyond the performance\-versus\-parameter laws of deep learning,”Sustainable Computing, 2023\. arXiv:2109\.05472\.
- \[15\]M\. Schmied et al\., “Bench360: benchmarking local LLM inference from a 360\-degree view,” arXiv:2511\.16682, 2025\.
- \[16\]C\. Hooper et al\., “KVQuant: towards 10 million context length LLM inference,”Proc\. NeurIPS, 2024\. arXiv:2401\.18079\.
- \[17\]E\. Frantar et al\., “GPTQ: accurate post\-training quantization for generative pre\-trained transformers,”Proc\. ICLR, 2023\. arXiv:2210\.17323\.
- \[18\]Y\. Leviathan, M\. Kalman, and Y\. Matias, “Fast inference from transformers via speculative decoding,”Proc\. ICML, 2023\. arXiv:2211\.17192\.
- \[19\]T\. Dao et al\., “FlashAttention: fast and memory\-efficient exact attention with IO\-awareness,”Proc\. NeurIPS, 2022\. arXiv:2205\.14135\.
- \[20\]L\. Zheng et al\., “SGLang: efficient execution of structured language model programs,”Proc\. NeurIPS, 2024\. arXiv:2312\.07104\.
- \[21\]P\. Patel et al\., “Splitwise: efficient generative LLM inference using phase splitting,”Proc\. ISCA, 2024\. arXiv:2311\.18677\.
- \[22\]R\. Qin et al\., “Mooncake: a KVCache\-centric disaggregated architecture for LLM serving,” arXiv:2407\.00079, 2024\.
- \[23\]A\. Dubey et al\., “The Llama 3 herd of models,” arXiv:2407\.21783, 2024\.
- \[24\]A\. Yang et al\., “Qwen2\.5 technical report,” arXiv:2412\.15115, 2024\.