衡量开放权重大语言模型在不同基准测试、推理模式和硬件上的推理能效

# Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
Source: [https://arxiv.org/html/2610.14389](https://arxiv.org/html/2610.14389)
Zeyu YangAffiliation:Department of Engineering ScienceAffiliation:University of OxfordEmail:[zeyu\.yang@eng\.ox\.ac\.uk](mailto:)Wes ArmourAffiliation:Department of Engineering ScienceAffiliation:University of OxfordEmail:[wes\.armour@oerc\.ox\.ac\.uk](mailto:)

###### Abstract

Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability\. We directly measure this trade\-off for 86 open\-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight\-GPU server\. We find large differences in energy efficiency even among models with similar capability\. Most configurations lie away from the score–energy Pareto frontier: they use a median of 2\.6–4\.8×\\timesmore energy than another configuration that scores at least as well, with differences reaching 27×\\times\. Reasoning can improve capability, but enabling thinking increases energy by about 3\.7×\\timeson average, and the benefit depends strongly on the model family and output budget\. Generated\-token count is also an imperfect proxy for energy, misordering 18–24% of model pairs\. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models\. These results show that capability alone gives an incomplete picture of efficiency\. We argue that benchmark scores should be reported together with measured energy consumption under a specified serving regime and output budget\.

## 1Introduction

Two LLMs with similar benchmark scores can require very different amounts of energy to serve\. Among the open\-weight models we evaluate that score above 80% on AIME, the GPU energy required per correct answer spans roughly33×33\\times, from 1\.5 to 50 Wh depending on the model and the hardware serving it \(Fig\.[12](https://arxiv.org/html/2610.14389#A2.F12); Appendix[B\.2](https://arxiv.org/html/2610.14389#A2.SS2)\)\. Conventional leaderboards capture the difference in capability, but not the energy required to deliver it\.

This distinction is becoming increasingly important as inference accounts for a growing share of AI computation\. Around three fifths of the energy Google reports for machine learning is attributed to inference rather than training\([Patterson et al\., 2022](https://arxiv.org/html/2610.14389#bib.bib47)\), while the output lengths of reasoning models have grown rapidly\([Emberson and others, 2025](https://arxiv.org/html/2610.14389#bib.bib42)\)\. More broadly, global data\-centre electricity consumption was approximately 415 TWh in 2024 and is projected to more than double by 2030, with AI as a major driver\([IEA, 2025](https://arxiv.org/html/2610.14389#bib.bib48)\)\. Evaluating model capability without considering the energy required to deliver it therefore gives an incomplete picture of inference efficiency\.

Measuring this trade\-off is not as simple as counting parameters, FLOPs, or generated tokens\. These quantities are often used as proxies for inference cost, but do not uniquely determine physical energy consumption\([Henderson et al\., 2020](https://arxiv.org/html/2610.14389#bib.bib32);[Dehghani et al\., 2022](https://arxiv.org/html/2610.14389#bib.bib33);[Fernandez et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib25)\)\. Direct measurements are also sensitive to serving conditions: batching, sequence length, hardware, precision, and the serving stack can all substantially change the energy required for the same workload\([Saad\-Falcon et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib10);[Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20);[Pizzini Cavagna et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib28)\)\. Meanwhile, capability leaderboards typically report benchmark scores without the corresponding measured energy consumption\. The capability–energy trade\-off across a broad set of current LLMs under a common high\-utilisation serving regime therefore remains poorly characterised\.

In this work, we measure this trade\-off directly\. We evaluate 86 open\-weight checkpoints, corresponding to 134 serving configurations, on seven scored benchmarks spanning instruction following, knowledge, science, mathematics, coding, and tool use\. Models are served with saturated continuous batching on six hardware platforms ranging from a consumer GPU to an eight\-GPU server\. Rather than assigning each model a fixed energy\-per\-token value, we meter GPU energy consumption using each benchmark’s own traffic, including the additional tokens generated by reasoning modes\. We then compare benchmark score against measured energy consumption and analyse the resulting score–energy Pareto fronts\.

#### Contributions

Our main contributions and findings are:

- •A large\-scale measurement of capability against directly measured inference energy\.In the main cross\-model comparison, only 21 of 82 configurations lie on a Pareto front; off\-front configurations use a median of2\.62\.6–4\.8×4\.8\\timesthe energy of an alternative scoring at least as well, and up to27×27\\times\(Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)\)\.
- •A controlled characterisation of the energy cost of reasoning and longer outputs\.Across 44 paired hybrid configurations, enabling thinking increases energy by about3\.7×3\.7\\times, while the capability gain depends strongly on the model lineage \(Fig\.[3](https://arxiv.org/html/2610.14389#S6.F3)\)\. Increasing the output budget can recover substantial additional capability, but at up to6\.1×6\.1\\timesthe energy; late tokens are also2\.02\.0–4\.9×4\.9\\timesmore energy\-intensive \(Fig\.[4](https://arxiv.org/html/2610.14389#S6.F4)\)\.
- •Evidence that commonly used inference\-cost proxies can give a different picture from physical energy\.Generated\-token count misorders energy for 18–24% of model pairs, and score–token Pareto fronts retain only 10 of the 25 models selected by the corresponding score–energy fronts \(Fig\.[2](https://arxiv.org/html/2610.14389#S5.F2)\)\. Parameter count is an even weaker predictor of energy\.
- •A cross\-platform analysis of deployment choices\.Vendor\-provided FP8 weights reduce GPU energy by roughly a quarter with no measurable change in aggregate score \(Fig\.[5](https://arxiv.org/html/2610.14389#S7.F5)\), while the same configurations consume0\.50\.5–2\.2×2\.2\\timesas much energy across hardware platforms but preserve almost the same energy ranking \(ρ=0\.99\\rho=0\.99; Fig\.[6](https://arxiv.org/html/2610.14389#S7.F6)\)\.

## 2Preliminaries: What Serving Energy Depends On

Serving a language model has two main phases\. Prefill processes the input sequence in parallel and is typically compute\-intensive, while decode generates tokens autoregressively and is often memory\-bandwidth\-bound at small batch sizes\([Pope et al\., 2023](https://arxiv.org/html/2610.14389#bib.bib43);[Patel et al\., 2024b](https://arxiv.org/html/2610.14389#bib.bib44)\)\. Each decode step accesses the model parameters and accumulated KV state, making memory movement a major cost\. Continuous batching improves utilisation by serving many active requests together and replacing completed requests as execution proceeds\([Yu et al\., 2022](https://arxiv.org/html/2610.14389#bib.bib45);[Kwon et al\., 2023](https://arxiv.org/html/2610.14389#bib.bib2)\)\. This allows model execution to be shared across more requests and increases throughput\. Since GPU power typically grows much less than throughput as concurrency increases\([Patel et al\., 2024a](https://arxiv.org/html/2610.14389#bib.bib35)\), higher utilisation generally reduces energy per token until KV\-cache capacity limits further batching\([Kwon et al\., 2023](https://arxiv.org/html/2610.14389#bib.bib2);[Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20)\)\.

Energy per token is therefore not a property of the model alone\. It depends on the model, hardware, serving system, and workload\. Batch size can change per\-token energy by several\-fold\([Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20)\), while input and output lengths can produce much larger variation on the same model and GPU\([Pizzini Cavagna et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib28)\)\. Long outputs are especially costly because they generate more tokens while increasing KV\-cache usage and reducing achievable concurrency\([Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20)\)\. In finite workloads, concurrency can also fall as shorter requests finish, so late tokens may be generated in smaller batches even when the GPU remains highly powered\. A single energy\-per\-token value therefore cannot reliably represent the cost of a benchmark, and simple models based only on prompt and output token counts can incur substantial error under batched serving\([Pizzini Cavagna et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib28);[Dutt et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib29)\)\.

Meaningful comparisons therefore require energy to be measured under a defined workload, serving regime, and hardware platform\. This motivates our use of direct GPU level energy measurements under high\-utilisation continuous batching, rather than assigning each model a fixed energy\-per\-token value\. Our measurement boundary excludes host and data\-centre overhead\. For context, Google’s production measurements report total serving energy at roughly1\.7×1\.7\\timesactive accelerator energy after accounting for host systems, idle capacity, and infrastructure overhead\([Elsworth et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib46)\)\. Section[4](https://arxiv.org/html/2610.14389#S4)describes the workload construction and measurement procedure\.

## 3Related Work

#### Energy benchmarking

Several benchmarks directly measure LLM inference energy\. ML\.ENERGY evaluates more than 40 models and workloads\([Chung et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib21);[Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20)\), while MLPerf Power reports system efficiency subject to workload\-specific accuracy requirements\([Tschand et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib22)\)\. The AI Energy Score measures energy per query at batch size 1\([Luccioni et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib23);[Luccioni et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib24)\)\. These efforts establish the value of direct measurement, but do not jointly characterise capability and energy across a broad model set under a common high\-utilisation serving regime, where batching, sequence shape, software, and hardware can substantially affect energy\([Fernandez et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib25);[Chung et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib20);[Saad\-Falcon et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib10)\)\.

#### Serving efficiency and reasoning length

Systems work reduces inference energy through frequency, parallelism, batching, and scheduling\([You et al\., 2023](https://arxiv.org/html/2610.14389#bib.bib34);[Patel et al\., 2024a](https://arxiv.org/html/2610.14389#bib.bib35);[Stojkovic et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib36)\), while other work controls or shortens reasoning traces\([Snell et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib37);[Muennighoff et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib38);[Aggarwal and Welleck, 2025](https://arxiv.org/html/2610.14389#bib.bib39);[Chen et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib40)\)\. Response\-length studies further show substantial variation across models and increasingly long reasoning traces\([Poddar et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib41);[Emberson and others, 2025](https://arxiv.org/html/2610.14389#bib.bib42)\)\. Our focus is complementary: we measure how these choices affect the energy required to deliver benchmark capability\.

#### Energy proxies and metrics

Several works connect capability to estimated inference cost\. Token Arena estimates joules per correct answer from hardware power assumptions\([Gao et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib16)\);[Oviedo et al\. \(2026\)](https://arxiv.org/html/2610.14389#bib.bib17)estimate per\-query energy from published throughput and power data; and Capability Frontier and OckBench use monetary cost or token counts\([Fowler et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib18);[Du et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib19)\)\. Analytical models similarly estimate energy from FLOPs or token counts\([Faiz et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib26);[Wilkins et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib27);[Pizzini Cavagna et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib28)\), although accuracy can degrade as serving conditions vary\([Pizzini Cavagna et al\., 2026](https://arxiv.org/html/2610.14389#bib.bib28);[Dutt et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib29);[Fu et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib30)\)\. Prior work therefore cautions against universal efficiency proxies\([Schwartz et al\., 2020](https://arxiv.org/html/2610.14389#bib.bib31);[Henderson et al\., 2020](https://arxiv.org/html/2610.14389#bib.bib32);[Dehghani et al\., 2022](https://arxiv.org/html/2610.14389#bib.bib33)\)\. We instead retain capability and measured energy as separate objectives and analyse their Pareto frontier\.

#### Joint energy and capability measurement

A smaller body of work directly measures energy while evaluating model quality\. Intelligence per Watt evaluates more than 20 models across eight accelerators, primarily at batch size 1\([Saad\-Falcon et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib10)\); as Appendix[C\.1](https://arxiv.org/html/2610.14389#A3.SS1)shows, this regime can have substantially higher energy per token than saturated serving\.[Jin et al\. \(2025\)](https://arxiv.org/html/2610.14389#bib.bib11)study one model family at fixed batch size, while[Stuhlmann et al\. \(2025\)](https://arxiv.org/html/2610.14389#bib.bib13);[Masthan Ali et al\. \(2026\)](https://arxiv.org/html/2610.14389#bib.bib12)consider smaller model sets under light\-load conditions\.[Caravaca et al\. \(2025\)](https://arxiv.org/html/2610.14389#bib.bib14)combine published capability scores with measured energy for a synthetic workload\. In vision,[Yang and Armour \(2025\)](https://arxiv.org/html/2610.14389#bib.bib15)jointly measured accuracy and energy for 1,200 architectures at saturating batch sizes\.

Together, these leave open the capability–energy frontier across current LLMs under high\-utilisation serving\. We address this with scored benchmark traffic, saturated continuous batching, and controlled comparisons of reasoning mode, output budget, precision, and hardware\.

## 4Methodology

#### Models

We evaluate 86 open\-weight checkpoints from eleven model families: Qwen, DeepSeek, Llama, Gemma, GLM, IBM Granite, Mistral, Phi, Nemotron, MiniMax, and gpt\-oss\. The checkpoints were released between July 2024 and August 2026, span 0\.8B–400B parameters, and include 43 dense and 43 mixture\-of\-experts models\. Within each family, we include instruction\-tuned checkpoints across the available size range, subject to compatibility with our pinned serving stack\. Vendor\-published FP8 and NVFP4 weight variants are included as separate checkpoint entries\.

Our unit of study is a*serving configuration*, rather than the model weights alone\. For hybrid models, reasoning mode is controlled through the chat template and changes both capability and generation length, so each mode forms a separate configuration\. Single\-mode checkpoints contribute one configuration, giving 134 in total\.

Sampling follows the generation configuration shipped with each checkpoint or the settings recommended in its model card; where neither is available, we use temperature 0\.6 and top\-pp0\.95\. This preserves the intended serving behaviour of each checkpoint rather than standardising decoding across models\. The full model list and sampling provenance are given in Appendix[A\.1](https://arxiv.org/html/2610.14389#A1.SS1)\.

#### Hardware

Models are grouped by weight footprint into three memory tiers: 32, 96, and 512 GB: the weight budget available to the model while retaining headroom for the KV cache\. Each tier is assigned only to platforms with sufficient memory \(Table[1](https://arxiv.org/html/2610.14389#S4.T1)\)\. We examine CPU weight offloading separately in Appendix[C\.2](https://arxiv.org/html/2610.14389#A3.SS2)\. The GB10 is capped at 96 of its 128 GB unified memory, leaving the remainder for the operating system and serving stack\.

For the main cross\-model comparison, the 32, 96, and 512 GB tiers use the RTX 5090, RTX PRO 6000, and 8×\\timesH100 as their reference platforms, respectively\. The 96 GiB model set is additionally served on three other platforms, spanning three GPU generations and three memory technologies, enabling like\-for\-like comparison of the same serving configurations across hardware\. Full system and software details are given in Appendix[A\.2](https://arxiv.org/html/2610.14389#A1.SS2)\.

Table 1:Serving platforms\. Multi\-GPU platforms are NVLink\-connected and use tensor parallelism, plus expert parallelism for MoE models on the 8\-GPU node\.
#### Benchmarks

The seven\-benchmark suite is summarised in Table[2](https://arxiv.org/html/2610.14389#S4.T2)\. All headline metrics use deterministic scoring without an LLM judge\. MMLU\-Pro and BFCL use stratified subsets sampled once with a fixed seed and then frozen as ID lists, so every configuration receives the same questions\. We use multiple seeded attempts for the higher\-variance GPQA and AIME benchmarks\. AIME 2026 and LiveCodeBench provide recent evaluation items with lower exposure to older training corpora\.

Benchmarks define the problems and scoring rules, but not a common generation budget\. We therefore use benchmark\-specific output caps held fixed across models, ranging from 4K to 32K tokens\. A generation that reaches the cap is scored normally; if truncation removes the extractable answer, it is counted as incorrect\. We do not resample truncated answers or assign partial credit\. Instead, we report truncation rates, treating completion within the common budget as part of model performance\. We separately examine the sensitivity of score and energy to larger output budgets in Fig\.[4](https://arxiv.org/html/2610.14389#S6.F4)\.

Table 2:The benchmark suite\.kk= seeded attempts\. cap = maximum new tokens\.†per turn\.
#### Evaluation harness

All benchmark harnesses communicate with the serving backend through a common OpenAI\-compatible logging proxy\. The proxy applies sampling parameters, output caps, and per\-attempt seeds, and records prompt, cached, and generated token counts\. Before evaluation, a smoke test checks the chat template, reasoning parser, answer extraction, and truncation behaviour\.

#### Deployment

We serve all models with a pinned version of vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.14389#bib.bib2)\)under saturated continuous batching\. Continuous batching, chunked prefill, and prefix caching are enabled\. We standardise the serving regime rather than a numerical concurrency: each platform retains the engine’s default admission limit, while client concurrency is set to1\.5×1\.5\\timesthat limit to maintain a continuous backlog and leave batching decisions to the server scheduler\. Prefix\-cache state is reset between benchmark runs, while reuse within a run is retained as part of the workload\.

#### Energy measurement

We use accelerator\-level energy as our primary measurement boundary\. On discrete GPUs, board power is sampled from NVIDIA’s onboard sensors and integrated over time; multi\-GPU energy is summed across devices\([Yang et al\., 2024](https://arxiv.org/html/2610.14389#bib.bib1)\)\. On the GH200 we use the GPU\+HBM reading, while on the GB10 we use the GPU\-rail energy counter\. Package\-level measurements for both superchips are recorded separately and reported in Appendix[A\.3](https://arxiv.org/html/2610.14389#A1.SS3)\. CPU and host memory on the discrete\-GPU systems, power\-supply losses, cooling, and other infrastructure overheads are outside the primary measurement boundary\.

Energy is measured from each benchmark’s own request traffic rather than from a synthetic workload\. We identify the active serving interval from server\-side token counters and exclude harness startup and CPU\-side scoring\. The first 10% and final 15% of the active interval are removed to exclude ramp\-up and drain\-out, leaving the saturated steady\-state region\.

#### Scoring and aggregation

Each benchmark is scored using its standard metric \(Table[2](https://arxiv.org/html/2610.14389#S4.T2)\) after any reasoning trace is removed by the model\-specific parser\. Scores are averaged over thekkseeded attempts\. Generated\-token counts include reasoning tokens, so the additional generation produced by reasoning modes remains part of their measured cost\.

We treat capability and energy as separate objectives rather than combining them into a score\-per\-energy ratio, which can favour inexpensive configurations with poor absolute capability\. A configuration is Pareto efficient if no other configuration in the comparison achieves at least its score while using no more energy, with a strict improvement in at least one objective\.

For the cross\-benchmark summary, capability is the arithmetic mean of the seven headline scores, while energy is the geometric mean of energy per attempt across the seven benchmark components\. The geometric mean gives each benchmark equal weight on a multiplicative scale rather than allowing the highest\-energy workloads to dominate\. Appendix[A\.4](https://arxiv.org/html/2610.14389#A1.SS4)analyses the effect of this aggregation\.

## 5The Capability–Energy Frontier

We first compare capability with energy across the three reference tiers\. Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)places each serving configuration in the score–energy plane, where higher scores and lower energy are preferable\.

Figure 1:Capability–energy frontiers across the three memory tiers\.Each panel shows one tier on its reference platform \(BF16 for 32/96 GiB; FP8 for 512 GiB\)\.xx: geometric\-mean GPU energy per attempt;yy: mean suite score\. Marker encoding follows the legend; steps show the Pareto fronts\. Fig\.[10](https://arxiv.org/html/2610.14389#A2.F10)gives the fully labelled version\.#### Most configurations are dominated

Across the three reference\-tier comparisons, only 21 of 82 configurations lie on a Pareto frontier\. For the remaining configurations, the cheapest alternative that scores at least as high uses a median of2\.62\.6–4\.8×4\.8\\timesless energy, depending on the tier; the largest difference is27×27\\times\. On the RTX PRO 6000, for example, Granite\-4\.2\-30b with thinking scores 77\.6 at 1,307 Wh, while Gemma\-4\-26B\-A4B without thinking scores 79\.5 at just 49 Wh\. Large differences in energy therefore remain even after capability is held approximately fixed\.

#### The highest scores come at a steep energy premium

Thinking configurations achieve the highest score on all three frontiers, but much of the final gain in capability is expensive\. In every tier, a non\-thinking configuration comes within five points of the highest score while using only a quarter to a sixth of its energy \(Qwen3\.5\-9B, Gemma\-4\-31B, and Qwen3\.5\-122B\-A10B\)\. Overall, non\-thinking configurations account for 15 of the 21 frontier members, compared with 6 of the 42 thinking configurations\. We examine the effect of reasoning mode and the output budget in Section[6](https://arxiv.org/html/2610.14389#S6)\.

Energy correlates more strongly with generated\-token count \(Spearmanρ=0\.70\\rho=0\.70–0\.790\.79\) than with active parameter count \(ρ=0\.28\\rho=0\.28–0\.520\.52\), but token count alone still does not explain which configurations are energy efficient\. Fig\.[2](https://arxiv.org/html/2610.14389#S5.F2)separates these two components\.

Figure 2:Energy varies through both generation length and energy per token\.Each panel shows all complete configurations on a reference platform, across weight formats\.xx: geometric\-mean generated tokens per attempt;yy: GPU energy per generated token, including prefill\. Their product gives the suite energy in Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)\. Dashed diagonals show equal energy; dotted lines show platform medians\. Marker encoding follows Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)\.
#### Generated tokens are not a reliable proxy for energy

Across the three reference platforms, generated\-token counts vary by66–14×14\\timesbetween the 90th and 10th percentiles, while energy per generated token varies independently by33–10×10\\times\. The two quantities are nearly uncorrelated \(Pearsonr=−0\.23r=\-0\.23to\+0\.02\+0\.02\), so generating more tokens does not imply that each token is more expensive\.

Token count nevertheless captures part of the variation in total energy, explaining 60–85% of its log\-scale variance\. Much of this association reflects the separation between thinking and non\-thinking configurations: within either group, variation in token count and energy per token are of comparable importance\. As a ranking proxy, however, generated tokens misorder energy for 18–24% of configuration pairs\. Two non\-thinking configurations with nearly identical token counts—Gemma\-3\-12B and Gemma\-4\-E2B on the RTX 5090—differ by23×23\\timesin energy\.

This difference is large enough to change which models appear efficient\. On the three tiers, the score–token frontier shares only 4 of 11, 3 of 6, and 3 of 8 members with the corresponding score–energy frontier\. It excludes configurations that generate many tokens cheaply, such as Gemma\-4\-26B\-A4B and Qwen3\.5\-122B\-A10B, while admitting Qwen3\-235B\-A22B\-Instruct, which generates relatively few tokens at roughly 3 J each\. Token\-based cost measures\([Du et al\., 2025](https://arxiv.org/html/2610.14389#bib.bib19)\)can therefore identify a substantially different set of efficient models from direct energy measurement\.

## 6Reasoning and Output Budget

The frontier analysis suggests that reasoning can reach higher capability, but often at substantially higher energy\. We isolate this effect using hybrid checkpoints that can be served with and without reasoning while keeping the weights and hardware fixed\.

Figure 3:Effect of enabling reasoning with the same weights on the same hardware\.Each point compares one hybrid checkpoint with thinking disabled and enabled on its tier’s reference platform \(44 pairs\), coloured by lineage; boxes show quartiles by platform and pooled\.*Left:*change in suite score, shown in points and as a ratio\.*Right:*change in GPU energy for one pass over the suite, shown in kWh and as a ratio \(log scale\)\.#### Reasoning is consistently expensive, but its capability gain is not

Across the 44 paired configurations, enabling thinking increases the suite score by a median of 4\.5 points \(mean 6\.3; 95% CI 3\.6–9\.1\), while multiplying the energy of a suite pass by3\.7×3\.7\\timeson average\. Eight of the 44 pairs lose score despite consuming more energy\.

The score gain varies much more by model lineage than by hardware\. We find no detectable platform effect on either the score or energy change \(Kruskal–Wallisp=0\.60p=0\.60and0\.880\.88, respectively\), whereas the difference between lineages is significant \(p<0\.001p<0\.001\)\. Nemotron hybrids gain a median of 20\.6 points and none loses score; GLM hybrids gain 12\.0 points in the median\. Qwen behaves very differently: its median change is−0\.7\-0\.7points, with 8 of 14 pairs scoring lower when thinking is enabled\. More generally, the gain from reasoning decreases as the corresponding non\-thinking score rises \(Spearmanρ=−0\.69\\rho=\-0\.69\)\.

These losses, however, are entangled with the output budget\. Enabling thinking raises the average share of requests truncated at the benchmark cap from 3% to 17%\. For the four Qwen configurations with the largest score losses, 25–62% of requests are truncated\. Under the optimistic clean\-completion bound of Fig\.[11](https://arxiv.org/html/2610.14389#A2.F11), these configurations would instead reach suite scores of 78–88\. The measured benefit of reasoning is therefore conditional not only on the model lineage, but also on how much generation budget it is given\.

Figure 4:Increasing the output budget can recover capability at higher energy cost\.Four thinking configurations are rerun from 8K to 128K output tokens; the main evaluation uses 32K\.*Left:*score against GPU energy per attempt, with marker area proportional to the output cap; energy is normalised to the request count\.*Right:*throughput and energy per generated token by output position\. Dashed lines: RTX 5090; solid lines: 8×\\timesH100\.
#### The output budget can change both scores and model rankings

Fig\.[4](https://arxiv.org/html/2610.14389#S6.F4)reruns four thinking configurations beyond the 32K cap used in the main evaluation\. Increasing the cap beyond 32K improves AIME 2026 by 7–25 points and LiveCodeBench by 6–21 points, but requires1\.11\.1–6\.1×6\.1\\timesas much energy\. The return on this additional energy varies sharply between models\. For the final budget doubling, Nemotron\-3\.5\-Lightning requires 1 Wh for each additional AIME point, compared with 239 Wh for GLM\-5\.3\-Flash; on LiveCodeBench the values are 5 and 1,869 Wh per point\.

More importantly, the budget can change the relative ordering of models\. DeepSeek\-V4\-Flash, for example, ranks last among the four configurations on AIME 2026 at 32K but first at the largest output cap\. The scores in the main frontier should therefore be interpreted as capability*under a common output budget*, rather than as estimates of each model’s unconstrained maximum capability\.

#### Late tokens become more expensive as the KV state grows

Within the same runs, energy per generated token rises2\.02\.0–4\.9×4\.9\\timesfrom the first 8K window of output to the last, while throughput falls to0\.190\.19–0\.46×0\.46\\times\. Longer sequences can increase cost in two ways\. Their growing inference state increases memory pressure and reduces achievable concurrency; in attention layers whose per\-token work grows with context length, later tokens can also require more state access\. In our serving logs, KV\-cache usage remains close to capacity while the number of running sequences falls to0\.190\.19–0\.37×0\.37\\times; GPU power, however, remains at0\.780\.78–0\.94×0\.94\\timesof its initial level\. Late tokens therefore require more work per sequence while being generated at lower concurrency, making them substantially more energy\-intensive than early tokens\.

## 7Deployment Choices

The previous sections compare different models and reasoning behaviours\. We next ask how much energy can be changed through deployment choices while holding the underlying model configuration fixed\. We consider two such choices: weight precision and hardware platform\.

Figure 5:Effect of replacing BF16 weights with vendor\-released FP8 weights on the same hardware\.Each point compares the same base model and reasoning mode in vendor\-released BF16 and FP8 variants on its tier’s reference platform \(28 matched pairs\), coloured by lineage; boxes show quartiles by platform and pooled\.*Left:*change in suite score, in points and as a ratio\.*Right:*change in GPU energy consumption, in kWh and as a ratio\.#### FP8 reduces energy by about a quarter with little change in score

Across the 28 matched pairs, moving from BF16 to FP8 changes the suite score by a median of−0\.15\-0\.15points \(mean−0\.16\-0\.16; 95% CI−0\.37\-0\.37to\+0\.05\+0\.05\)\. The corresponding GPU energy falls to0\.73×0\.73\\timesthat of BF16 \(geometric mean; 95% CI0\.690\.69–0\.770\.77\), with 27 of the 28 pairs using less energy\.

The saving varies by platform: FP8 uses0\.69×0\.69\\timesthe BF16 energy on the RTX PRO 6000,0\.79×0\.79\\timeson the 8×\\timesH100, and0\.84×0\.84\\timeson the RTX 5090\. The absolute saving consequently grows with model scale, from less than 0\.5 kWh per suite pass for 3–4B models on the RTX 5090 to 7\.8 kWh for Qwen3\-235B\-A22B\-Thinking on the 8×\\timesH100\. Overall, the energy reduction is much larger and more consistent than the change in capability\. This comparison also bounds the weight\-format difference in Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1), where the 32 and 96 GiB panels use BF16 and the 512 GiB panel uses FP8\.

#### Hardware rescales energy but barely changes the relative efficiency of models

The same serving configurations have almost identical energy rankings across the four platforms: relative to the RTX PRO 6000, Spearman correlation isρ=0\.99\\rho=0\.99on every platform, while benchmark scores differ by a median of only 0\.4–0\.7 points\. The score–energy Pareto frontier also contains the same members on every platform, to within one model\. Absolute energy nevertheless changes substantially\. Depending on weight format, the GH200 uses0\.610\.61–0\.77×0\.77\\timesthe accelerator energy of the RTX PRO 6000, the GB100\.530\.53–0\.62×0\.62\\times, and the two RTX A6000s1\.91\.9–2\.2×2\.2\\times\. The difference is mainly runtime for the GH200, PRO 6000 and A6000 pair; the GB10 instead trades much lower power for substantially longer runtime\.

#### Purchase cost dominates accelerator electricity at matched throughput

At the throughput of one RTX PRO 6000, the required accelerator hardware costs roughly $24–34k at our indicative secondary\-market prices, compared with $0\.5–1\.7k per year in accelerator electricity at $0\.20/kWh \(Fig\.[6](https://arxiv.org/html/2610.14389#S7.F6)\)\. This is illustrative rather than a full TCO analysis: host systems, cooling, maintenance, and infrastructure are excluded\.

Figure 6:Energy, performance, and cost across the four 96 GiB platforms\.*Top:*suite GPU energy, accelerator power, runtime, and token throughput\.*Bottom:*accelerator price, hardware cost at matched RTX PRO 6000 throughput, and corresponding annual accelerator electricity cost and power\. Secondary\-market prices use ten eBay listings; $1,000 is deducted from the GB10 price for its non\-GPU resources\. Electricity assumes $0\.20/kWh and continuous operation\. Host and infrastructure power are excluded\.

## 8Limitations and Scope

#### Output budgets

Scores are conditional on the benchmark\-specific generation caps used in our evaluation\. Verbose reasoning models can be penalised when answers are truncated: increasing the cap beyond 32K recovers substantial capability for some models, but costs up to6\.1×6\.1\\timesmore energy and can change their ranking \(Fig\.[4](https://arxiv.org/html/2610.14389#S6.F4)\)\. We also report a clean\-completion upper bound \(Fig\.[11](https://arxiv.org/html/2610.14389#A2.F11)\)\.

#### Model coverage

Our largest tier uses a single 8×\\timesH100 node and includes models up to 400B parameters\. Larger open\-weight models were beyond our compute budget, while closed API models fall outside our direct measurement setup\.

#### Measurement and serving

Our primary measurements cover accelerator energy rather than complete system or data\-centre energy; package\-level measurements for the GB10 and GH200 are reported in Appendix[A\.3](https://arxiv.org/html/2610.14389#A1.SS3)\. All main experiments use saturated continuous batching, the energy values describe a high\-utilisation serving regime and should not be assumed to hold under lighter loads\.

## 9Conclusion

We directly measured the inference energy of 136 serving configurations of open\-weight LLMs across seven benchmarks and six hardware platforms under saturated serving\. The capability–energy trade\-off varies widely: configurations off the Pareto frontier use a median of2\.62\.6–4\.8×4\.8\\timesthe energy of an alternative that scores at least as well, and up to27×27\\times\. Reasoning can improve capability, but at a substantial energy cost, and its benefit depends strongly on the model lineage and output budget; in every memory tier, a non\-thinking configuration comes within five points of the best score for only a quarter to a sixth of its energy\. Generated\-token and parameter counts do not reliably recover measured energy, while FP8 reduces energy by roughly a quarter with little change in score\. Changing hardware substantially rescales absolute energy but largely preserves the relative efficiency of models\. These results suggest that benchmark capability should be reported together with directly measured energy under a clearly stated serving regime and output budget\.

### AI use statement

We used generative AI tools to assist with literature discovery, parts of code development, and language editing during the preparation of this work\. All cited papers were checked against their original sources, AI\-assisted code was reviewed and tested by the authors, and all AI\-assisted writing was manually revised and verified\. The authors take full responsibility for the methodology, experiments, results, and final text\.

### Ethics statement

This work evaluates open\-weight language models on existing benchmarks and does not involve human subjects or the collection of personal data\. The experiments consume substantial computational resources and electricity; we therefore report the energy consumed by the evaluation campaign in Appendix[D](https://arxiv.org/html/2610.14389#A4)\. We are not aware of other significant ethical risks specific to the methodology or results presented here\.

### Reproducibility statement

We document the model roster, sampling configurations, benchmarks, hardware and software environments, serving regime, energy\-measurement procedure, and scoring methodology in the main paper and Appendix[A](https://arxiv.org/html/2610.14389#A1)\. These details are intended to support independent reproduction of the experiments\. We plan to release the evaluation codebase and result dataset publicly upon acceptance\.

## References

- P\. Aggarwal and S\. WelleckL1: controlling how long a reasoning model thinks with reinforcement learning\.InConference on Language Modeling \(COLM\),External Links:2503\.04697Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Caravacaet al\.\(2025\)F\. Caravaca, Á\. Cuevas, and R\. CuevasFrom prompts to power: measuring the energy footprint of LLM inference\.External Links:2511\.05597Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Chenet al\.\(2024\)X\. Chen, J\. Xu, T\. Liang, Z\. He, J\. Pang, D\. Yu, L\. Song, Q\. Liu, M\. Zhou, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. YuDo NOT think that much for 2\+3=? on the overthinking of o1\-like LLMs\.External Links:2412\.21187Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Chunget al\.\(2025\)J\. Chung, J\. J\. Ma, R\. Wu, J\. Liu, O\. J\. Kweon, Y\. Xia, Z\. Wu, and M\. ChowdhuryThe ML\.ENERGY benchmark: toward automated inference energy measurement and optimization\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\), Datasets and Benchmarks Track,External Links:2505\.06371Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Chunget al\.\(2026\)J\. Chung, R\. Wu, J\. J\. Ma, and M\. ChowdhuryWhere do the joules go? diagnosing inference energy consumption\.External Links:2601\.22076Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§2](https://arxiv.org/html/2610.14389#S2.p1.1),[§2](https://arxiv.org/html/2610.14389#S2.p2.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Dehghaniet al\.\(2022\)M\. Dehghani, A\. Arnab, L\. Beyer, A\. Vaswani, and Y\. TayThe efficiency misnomer\.InInternational Conference on Learning Representations \(ICLR\),External Links:2110\.12894Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Duet al\.\(2025\)Z\. Du, H\. Kang, S\. Han, T\. Krishna, and L\. ZhuOckBench: measuring the efficiency of LLM reasoning\.External Links:2511\.05722Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.14389#S5.SS0.SSS0.Px3.p3.1)\.
- Duttet al\.\(2025\)A\. Dutt, Y\. W\. Choi, A\. Sil, A\. Gandhi, A\. Balasubramanian, and N\. BalasubramanianFine\-grained energy prediction for parallelized LLM inference with PIE\-P\.Note:PreprintExternal Links:2512\.12801Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p2.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Elsworthet al\.\(2025\)C\. Elsworth, K\. Huang, D\. Patterson, I\. Schneider, R\. Sedivy, S\. Goodman, B\. Townsend, P\. Ranganathan, J\. Dean, A\. Vahdat, B\. Gomes, and J\. ManyikaMeasuring the environmental impact of delivering AI at Google scale\.External Links:2508\.15734Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p3.1)\.
- Embersonet al\.\(2025\)L\. Embersonet al\.LLM responses to benchmark questions are getting longer over time\.Note:Epoch AI data insight,[https://epoch\.ai/data\-insights/output\-length](https://epoch.ai/data-insights/output-length)Accessed 2026\-09Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p2.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Faizet al\.\(2024\)A\. Faiz, S\. Kaneda, R\. Wang, R\. Osi, P\. Sharma, F\. Chen, and L\. JiangLLMCarbon: modeling the end\-to\-end carbon footprint of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Fernandezet al\.\(2025\)J\. Fernandez, C\. Na, V\. Tiwari, Y\. Bisk, S\. Luccioni, and E\. StrubellEnergy considerations of large language model inference and efficiency optimizations\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 32556–32569\.External Links:[Link](https://aclanthology.org/2025.acl-long.1563),2504\.17674Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Fowleret al\.\(2026\)B\. Fowler, R\. Smith, D\. T\. Graviet, W\. Myers, J\. Greaves, N\. F\. Oozeer, A\. García, P\. Quirke, A\. Abdullah, F\. Barez, and S\. K\. UpadhyayThe capability frontier: benchmarks miss 82% of model performance\.In3rd Workshop on Data Problems in Foundation Models \(DATA\-FM\) at ICLR 2026,External Links:2606\.26836Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Fuet al\.\(2025\)Z\. Fu, F\. Chen, S\. Zhou, H\. Li, and L\. JiangLLMCO2: advancing accurate carbon footprint prediction for LLM inferences\.ACM SIGENERGY Energy Informatics Review5\(2\),pp\. 63–68\.External Links:[Document](https://dx.doi.org/10.1145/3757892.3757901),2410\.02950Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2026\)Y\. Gao, M\. Wang, and Y\. L\. YuToken arena: a continuous benchmark unifying energy and cognition in AI inference\.External Links:2605\.00300Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Hendersonet al\.\(2020\)P\. Henderson, J\. Hu, J\. Romoff, E\. Brunskill, D\. Jurafsky, and J\. PineauTowards the systematic reporting of the energy and carbon footprints of machine learning\.Journal of Machine Learning Research21\(248\),pp\. 1–43\.External Links:2002\.05651Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- IEA \(2025\)IEAEnergy and ai\.Technical reportInternational Energy Agency,Paris\.External Links:[Link](https://www.iea.org/reports/energy-and-ai)Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p2.1)\.
- Jainet al\.\(2025\)N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLivecodebench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.7.1)\.
- Jinet al\.\(2025\)Y\. Jin, G\. Wei, and D\. BrooksThe energy cost of reasoning: analyzing energy usage in LLMs with test\-time compute\.External Links:2505\.14733Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p1.1),[§4](https://arxiv.org/html/2610.14389#S4.SS0.SSS0.Px5.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.5.1)\.
- Luccioniet al\.\(2024\)A\. S\. Luccioni, Y\. Jernite, and E\. StrubellPower hungry processing: watts driving the cost of AI deployment?\.InProceedings of the ACM Conference on Fairness, Accountability, and Transparency \(FAccT ’24\),External Links:2311\.16863Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Luccioniet al\.\(2025\)S\. Luccioni B\. Gamazaychikovet al\.AI energy score: standardized energy efficiency ratings for AI models\.Note:[https://huggingface\.github\.io/AIEnergyScore/](https://huggingface.github.io/AIEnergyScore/)Accessed 2026\-09Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Masthan Aliet al\.\(2026\)S\. B\. Masthan Ali, S\. M\. Danish, Z\. Sattar, and A\. AsadAssessing the sustainability of LLM inference through energy–accuracy analysis\.InProceedings of the 17th ACM International Conference on Future and Sustainable Energy Systems \(e\-Energy ’26\),pp\. 627–635\.External Links:[Document](https://dx.doi.org/10.1145/3744255.3811741)Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. HashimotoS1: simple test\-time scaling\.External Links:2501\.19393Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Oviedoet al\.\(2026\)F\. Oviedo, F\. Kazhamiaka, E\. Choukse, A\. Kim, A\. Luers, M\. Nakagawa, R\. Bianchini, and J\. M\. Lavista FerresEnergy use of AI inference, efficiency pathways, and test\-time scaling\.Joule10,pp\. 102430\.External Links:[Document](https://dx.doi.org/10.1016/j.joule.2026.102430),2509\.20241Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Patelet al\.\(2024a\)P\. Patel, E\. Choukse, C\. Zhang, Í\. Goiri, B\. Warrier, N\. Mahalingam, and R\. BianchiniCharacterizing power management opportunities for LLMs in the cloud\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems \(ASPLOS\),External Links:2308\.12908Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p1.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Patelet al\.\(2024b\)P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. BianchiniSplitwise: efficient generative LLM inference using phase splitting\.InACM/IEEE 51st Annual International Symposium on Computer Architecture \(ISCA\),External Links:2311\.18677Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.8.1)\.
- Pattersonet al\.\(2022\)D\. Patterson, J\. Gonzalez, U\. Hölzle, Q\. Le, C\. Liang, L\. Munguia, D\. Rothchild, D\. So, M\. Texier, and J\. DeanThe carbon footprint of machine learning training will plateau, then shrink\.Computer55\(7\),pp\. 18–28\.External Links:2204\.05149Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p2.1)\.
- Pizzini Cavagnaet al\.\(2026\)H\. Pizzini Cavagna, A\. Proia, G\. Madella, G\. B\. Esposito, F\. Antici, D\. Cesarini, Z\. Kiziltan, and A\. BartoliniSweetSpot: an analytical model for predicting energy efficiency of LLM inference\.InProceedings of the 17th ACM/SPEC International Conference on Performance Engineering \(ICPE ’26\),pp\. 83–95\.External Links:[Document](https://dx.doi.org/10.1145/3777884.3797011)Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§2](https://arxiv.org/html/2610.14389#S2.p2.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Poddaret al\.\(2025\)S\. Poddar, P\. Koley, J\. Misra, N\. Ganguly, and S\. GhoshBrevity is the soul of sustainability: characterizing llm response lengths\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 21848–21864\.Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Popeet al\.\(2023\)R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, A\. Levskaya, J\. Heek, K\. Xiao, S\. Agrawal, and J\. DeanEfficiently scaling transformer inference\.InProceedings of Machine Learning and Systems \(MLSys\),External Links:2211\.05102Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.4.1)\.
- Saad\-Falconet al\.\(2025\)J\. Saad\-Falcon, A\. Narayan, H\. O\. Akengin, J\. W\. Griffin, H\. Shandilya, A\. G\. Lafuente, M\. Goel, R\. Joseph, S\. Natarajan, E\. K\. Guha,et al\.Intelligence per watt: measuring intelligence efficiency of local AI\.External Links:2511\.07885Cited by:[§1](https://arxiv.org/html/2610.14389#S1.p3.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Schwartzet al\.\(2020\)R\. Schwartz, J\. Dodge, N\. A\. Smith, and O\. EtzioniGreen AI\.Communications of the ACM63\(12\),pp\. 54–63\.External Links:1907\.10597Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Snellet al\.\(2024\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling model parameters\.External Links:2408\.03314Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Stojkovicet al\.\(2025\)J\. Stojkovic, C\. Zhang, Í\. Goiri, J\. Torrellas, and E\. ChoukseDynamoLLM: designing LLM inference clusters for performance and energy efficiency\.InIEEE International Symposium on High\-Performance Computer Architecture \(HPCA\),External Links:2408\.00741Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Stuhlmannet al\.\(2025\)L\. Stuhlmann, M\. Fadel Argerich, and J\. FürstBench360: benchmarking local LLM inference from 360 degrees\.External Links:2511\.16682Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Tschandet al\.\(2024\)A\. Tschand, A\. T\. R\. Rajan, S\. Idgunji, A\. Ghosh, J\. Holleman, C\. Kiraly,et al\.MLPerf power: benchmarking the energy efficiency of machine learning systems from microwatts to megawatts for sustainable AI\.Note:Presented at HPCA 2025External Links:2410\.12032Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.3.1)\.
- Wilkinset al\.\(2024\)G\. Wilkins, S\. Keshav, and R\. MortierOffline energy\-optimal LLM serving: workload\-based energy models for LLM inference on heterogeneous systems\.Note:HotCarbon 2024External Links:2407\.04014Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)Z\. Yang, K\. Adamek, and W\. ArmourAccurate and convenient energy measurements for gpus: a detailed study of nvidia gpu’s built\-in power sensor\.InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–17\.Cited by:[§4](https://arxiv.org/html/2610.14389#S4.SS0.SSS0.Px6.p1.1)\.
- Yang and Armour \(2025\)Z\. Yang and W\. ArmourThe hidden joules: evaluating the energy consumption of vision backbones for progress towards more efficient model inference\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267\.Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px4.p1.1)\.
- Youet al\.\(2023\)J\. You, J\. Chung, and M\. ChowdhuryZeus: understanding and optimizing GPU energy consumption of DNN training\.In20th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 23\),External Links:2208\.06102Cited by:[§3](https://arxiv.org/html/2610.14389#S3.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2022\)G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. ChunOrca: a distributed serving system for transformer\-based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),pp\. 521–538\.External Links:[Link](https://www.usenix.org/conference/osdi22/presentation/yu)Cited by:[§2](https://arxiv.org/html/2610.14389#S2.p1.1)\.
- Zhang and Team \(2026\)Y\. Zhang and M\. TeamAmerican invitational mathematics examination \(aime\) 2026\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime26)Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.6.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Table 2](https://arxiv.org/html/2610.14389#S4.T2.4.2.1)\.

## Appendix AExperimental Setup

### A\.1Models

Tables[3](https://arxiv.org/html/2610.14389#A1.T3)–[5](https://arxiv.org/html/2610.14389#A1.T5)list the checkpoints in each memory tier\. A checkpoint is assigned to the smallest tier in which its stored weights fit while retaining headroom for the KV cache\. Vendor\-published FP8 and NVFP4 releases are treated as separate checkpoints, so lower\-precision variants can appear in a smaller tier than their higher\-precision counterparts\.

Table 3:Roster of the 32 GiB tier \(31 checkpoints, 47 study rows\), served on a single NVIDIA GeForce RTX 5090\.Params: billions of parameters; MoE checkpoints report total / active parameters \(Gemma 4 E\-series: total / effective\)\.Weights: size of the safetensors weights loaded by the server\.Precision: stored weight format; gpt\-oss stores its MoE experts in MXFP4\.Modes: I = instruct, T = thinking, NT = non\-thinking; hybrid checkpoints listed as T, NT contribute two study rows\. L, M, H denote documented low, medium, and high reasoning\-effort settings\.Released: checkpoint release date\.Hybrid checkpoints contribute one study row per documented reasoning mode, selected through the vendor\-provided chat\-template controls \(for example,enable\_thinking\)\. gpt\-oss and GLM\-5\.3\-Flash similarly contribute one row per documented reasoning\-effort level\. A model\-specific reasoning parser in the serving engine removes the reasoning trace before answer scoring\.

Sampling follows vendor\-provided settings\. For 79 configurations, parameters are read from the checkpoint’sgeneration\_config\.jsonand applied by the proxy; for another 17, the serving engine applies that configuration directly\. The 15 Mistral configurations use model\-card settings\. For the remaining 25 configurations, for which no settings are documented, we use temperature 0\.6 and top\-pp0\.95\.

Table 4:Roster of the 96 GiB tier \(29 checkpoints, 47 study rows\), served on all four 96 GiB platforms: GB10, RTX PRO 6000 Blackwell, GH200, and 2×\\timesRTX A6000 \(TP=2\)\. Columns are defined in Table[3](https://arxiv.org/html/2610.14389#A1.T3)\.Table 5:Roster of the 512 GiB tier \(26 checkpoints, 42 study rows\), served on the 8×\\timesH100 node with TP=8 and expert parallelism enabled for MoE checkpoints\. Columns are defined in Table[3](https://arxiv.org/html/2610.14389#A1.T3)\.CheckpointParamsWeightsPrecisionModesReleased\(B\)\(GiB\)AlibabaQwen/Qwen3\-235B\-A22B\-Instruct\-2507235 / 22437\.9BF16I2025\-09\-17Qwen3\-235B\-A22B\-Instruct\-2507\-FP8235 / 22220\.2FP8I2025\-09\-17Qwen3\-235B\-A22B\-Thinking\-2507235 / 22437\.9BF16T2025\-08\-17Qwen3\-235B\-A22B\-Thinking\-2507\-FP8235 / 22220\.2FP8T2025\-07\-30Qwen3\.5\-122B\-A10B122 / 10233\.0BF16T, NT2026\-03\-09Qwen3\.5\-122B\-A10B\-FP8122 / 10118\.4FP8T, NT2026\-03\-09Qwen3\.5\-397B\-A17B\-FP8397 / 17378\.3FP8T, NT2026\-03\-09Qwen3\.8\-Flash\-Next180 / 6335\.3BF16T, NT2026\-08\-24Qwen3\.8\-Flash\-Next\-FP8180 / 6172\.8FP8T, NT2026\-08\-24DeepSeekdeepseek\-ai/DeepSeek\-R1\-Distill\-Llama\-70B70131\.4BF16T2025\-01\-20DeepSeek\-V4\-Flash284 / 13148\.7FP4/FP8T, NT2026\-04\-24Metameta\-llama/Llama\-3\.1\-70B\-Instruct70131\.4BF16I2024\-07\-23Llama\-4\-Scout\-17B\-16E\-Instruct109 / 17202\.4BF16I2025\-04\-05Llama\-4\-Maverick\-17B\-128E\-Instruct400 / 17388\.1FP8I2025\-04\-05MiniMaxMiniMaxAI/MiniMax\-M2230 / 10214\.3FP8T2025\-10\-27MiniMax\-M2\.5230 / 10214\.3FP8T2026\-02\-12MiniMax\-M2\.7230 / 10214\.3FP8T2026\-03\-18Mistral AImistralai/Mistral\-Medium\-3\.5\-128B128248\.9BF16T, NT2026\-04\-29Mistral\-Small\-4\-119B\-2603119 / 6\.5225\.2BF16T, NT2026\-03\-16NVIDIAnvidia/Nemotron\-3\-Super\-120B\-A12B\-BF16120 / 12230\.2BF16T, NT2026\-03\-11Nemotron\-3\-Super\-120B\-A12B\-FP8120 / 12119\.5FP8T, NT2026\-03\-11Z\.aizai\-org/GLM\-4\.5\-Air106 / 12205\.8BF16T, NT2025\-07\-28GLM\-4\.5\-Air\-FP8106 / 12104\.8FP8T, NT2025\-07\-28GLM\-4\.5\-FP8355 / 32336\.5FP8T, NT2025\-07\-28GLM\-4\.7\-FP8355 / 32337\.2FP8T, NT2025\-12\-22GLM\-5\.3\-Flash320 / 18305\.8FP8L, H, M2026\-08\-25
### A\.2Hardware

Table[6](https://arxiv.org/html/2610.14389#A1.T6)gives the hardware, software, measurement, and serving details of each platform\. All accelerators run at their default power limits\. Single\-accelerator platforms serve each model on one device; the A6000 pair uses tensor parallelism over NVLink, while the 8×\\timesH100 node uses tensor parallelism across eight GPUs with expert parallelism enabled for MoE models\. The GB10 serving engine is limited to 96 GiB of its 128 GB unified memory\.

The server admission limit follows the pinned serving engine’s default: 256 concurrent sequences on the RTX 5090, A6000 pair, and GB10, and up to 1,024 on the other three platforms, with the engine reducing the limit where required by the model\.

Table 6:Hardware, software, measurement, and serving details for the six platforms in Table[1](https://arxiv.org/html/2610.14389#S4.T1)\.
### A\.3Energy Measurement

#### Instrumentation

On discrete GPUs, board power is read from the driver’s onboard sensor and integrated over time; multi\-GPU energy is summed across devices\. On the GB10, we read the SoC’s cumulative energy counters and use the GPU\-rail counter as the primary measurement, while also recording the CPU and package counters\. On the GH200, the primary GPU reading covers the GPU and HBM3; whole\-module power for the Grace Hopper superchip is recorded alongside it\.

#### Integrated superchips

Figure[7](https://arxiv.org/html/2610.14389#A1.F7)compares these primary accelerator measurements with the wider package or module boundary\. On the GB10, package energy is a median1\.69×1\.69\\timesthe GPU\-rail energy \(5th–95th percentile1\.521\.52–1\.85×1\.85\\times\): 59% is attributed to the GPU rail, 10% to the CPU cores, and 31% to the remainder of the package, including the LPDDR5X memory\. On the GH200, module energy is1\.125×1\.125\\timesthe GPU\+HBM reading \(1\.111\.11–1\.19×1\.19\\times\); the remaining 11% covers the Grace CPU, LPDDR5X, and NVLink\-C2C\.

The additional package or module draw is approximately constant at 34 W on the GB10 and 77 W on the GH200, so these shares vary little across reasoning modes, weight formats, and benchmarks\. The measurement boundary matters when comparing against discrete GPUs, whose board\-power measurements include device memory\. For the same configurations, the GB10 consumes0\.61×0\.61\\timesthe RTX PRO 6000 energy at GPU scope but1\.03×1\.03\\timesat package scope; the corresponding GH200 ratios are0\.64×0\.64\\timesand0\.73×0\.73\\times\.

Figure 7:Energy outside the primary accelerator measurement on the two superchips\.*Left:*energy over all metered attempts, split by component\. For the GB10, the primary measurement is the GPU rail; for the GH200, it is the GPU\+HBM reading\.*Right:*the same shares by reasoning mode, weight format, and benchmark, with 95% bootstrap intervals on the primary\-counter share and mean power at right\. Package/module measurements still exclude power\-supply losses, fans, and storage\.

### A\.4Suite Aggregation

The suite contains seven components, with AIME 2025 and AIME 2026 combined into one component by averaging their scores and summing their energies\. Suite score is the arithmetic mean of the seven component scores, while suite energy is the geometric mean of their per\-attempt energies\.

We use the geometric mean to prevent benchmarks with more or longer generations from dominating the aggregate\. Under a simple sum, LiveCodeBench contributes a median 25–29% of the summed suite\-pass energy and is the largest component for 93 of the 134 configurations with complete reference\-platform results, whereas IFEval contributes only 3–5% \(Fig\.[8](https://arxiv.org/html/2610.14389#A1.F8)\)\. LiveCodeBench also accounts for 27–36% of the cross\-configuration variation under the sum, compared with 3–7% for IFEval\. Under the geometric mean, each component contributes 9–20%, much closer to the equal share of one seventh \(Fig\.[9](https://arxiv.org/html/2610.14389#A1.F9)\)\.

The aggregation choice has little effect on model ordering: the arithmetic\-sum and geometric\-mean rankings have Spearmanρ=0\.985\\rho=0\.985–0\.9950\.995, and dropping any one component givesρ≥0\.986\\rho\\geq 0\.986\. We use the geometric mean because changing a benchmark’s number of items or attempts then rescales all configurations equally, leaving pairwise energy ratios and rankings unchanged\.

Figure 8:Benchmark shares of summed suite\-pass energy\.For every complete configuration on each reference platform, boxes show the share of total suite\-pass energy contributed by each benchmark, separated into non\-thinking \(orange\) and thinking \(blue\) configurations\. Points show individual configurations\. The dashed line marks one seventh, the equal weight assigned to each component by the geometric mean\.Figure 9:Each benchmark’s contribution to cross\-configuration variation in aggregate energy\.For each reference platform, bars show the contribution of each benchmark under the arithmetic sum of raw energies \(light\) and the geometric mean used in the paper \(dark, on log energies\)\. Contributions sum to 100%; error bars are 95% bootstrap intervals over configurations\. The dashed line marks an equal one\-seventh contribution\.

## Appendix BAdditional Results

### B\.1The Frontier in Detail

Fig\.[10](https://arxiv.org/html/2610.14389#A2.F10)reproduces the three panels of Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)at full width with every configuration labelled\. Each panel uses the majority weight format of its tier: BF16 for the 32 and 96 GiB tiers, and FP8 for the 512 GiB tier\.

Figure 10:Score–energy fronts with every configuration labelled\.Enlarged versions of the three panels in Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1); marker encoding is unchanged\.#### Robustness of the fronts

We recompute the fronts over 2,000 resampled draws\. In each draw, benchmark scores are resampled over their items and attempts, giving a typical suite\-score standard error of about 0\.8 points, while component energies are perturbed by 7%, the median variation between repeated attempts of the same benchmark\. Eighteen of the 21 frontier members remain on their front in at least 85% of draws\.

The three less\-stable members are all near\-ties\. Qwen3\.5\-4B without thinking is within 0\.1 score points of Nemotron\-3\-Nano\-4B with thinking and remains on the front in 77% of draws\. On the 512 GiB tier, Mistral\-Medium\-3\.5 and GLM\-4\.5\-Air without thinking are within 2% of the energy of neighbouring frontier configurations and remain on the front in 65% and 49% of draws, respectively\. No off\-front configuration joins a frontier in more than 40% of draws\. When any one suite component is removed, 16 of the 21 frontier members remain\.

Figure 11:Score–energy fronts under the clean\-completion bound\.As in Fig\.[10](https://arxiv.org/html/2610.14389#A2.F10), but benchmark scores are recomputed using only requests that finish within the output cap\. Energy is unchanged and the Pareto fronts are recomputed\.
#### The clean\-completion bound

Fig\.[11](https://arxiv.org/html/2610.14389#A2.F11)replaces each benchmark score with its score over requests that complete within the output cap, equivalently dividing the observed score by one minus the truncated\-request fraction when truncated requests contribute zero\.

Energy is left unchanged\. The bound is deliberately optimistic: it assumes that truncated answers would be correct at the same rate as completed answers, while charging no additional energy for allowing them to finish\. It therefore estimates an upper bound on the capability recoverable from a larger output budget\.

The effect is largest where truncation is most severe\. Across the three tiers, thinking configurations gain a median of 3\.5, 3\.9, and 5\.9 points, compared with 2\.2, 0\.9, and 0\.6 points for non\-thinking configurations\. Overall rankings remain similar to the scored suite \(Spearmanρ=0\.79\\rho=0\.79, 0\.75, and 0\.85\), but thinking configurations move to the top of each frontier\. On the RTX 5090, Qwen3\.5 thinking configurations join the frontier at scores of 78–84; on the RTX PRO 6000, Qwen3\.6\-35B\-A3B and two Nemotron thinking configurations join at 85–88; and on the 8×\\timesH100, four thinking configurations join at 86–90\. Every non\-thinking frontier member remains\.

### B\.2Energy per Correct Answer

Fig\.[12](https://arxiv.org/html/2610.14389#A2.F12)reports benchmark energy divided by the number of correct answers for every configuration on its tier’s reference platform\. BFCL is excluded because its headline score is a category macro\-average rather than a count of correct answers\.

The cost of a correct AIME answer ranges from 0\.54 to 815 Wh\. The upper end is driven by configurations with near\-zero accuracy; excluding configurations below 5% accuracy narrows the range to 0\.54–50 Wh\. Across the other benchmarks, the median cost per correct answer ranges from 0\.12 Wh on IFEval to 5\.9 Wh on LiveCodeBench, with thinking configurations having a 1\.7–4\.5×\\timeshigher median than non\-thinking configurations\.

Cost and accuracy are only weakly correlated, with Spearmanρ=0\.12\\rho=0\.12–0\.370\.37across benchmarks\. Among the 29 configurations achieving at least 80% on AIME, a correct answer costs from 1\.5 Wh \(gpt\-oss\-120b, medium effort, RTX PRO 6000\) to 50 Wh \(GLM\-4\.7 with thinking, 8×\\timesH100\), a 33×\\timesspread\. Within every five\-point accuracy band from 40% to 90%, the most energy\-intensive configuration uses 12–36×\\timesas much energy per correct answer as the least\. These comparisons span memory tiers and therefore combine model and hardware effects\.

Figure 12:Energy per correct answer\.All configurations are shown on their tier’s reference platform; AIME 2025 and 2026 are pooled, and BFCL is excluded\.*Left:*energy per correct answer across benchmarks; boxes summarise configurations and points denote reasoning mode, with the extremes labelled\.*Right:*energy per correct AIME answer against AIME accuracy\. The dashed step gives the minimum energy per correct answer among configurations achieving at least each accuracy\.
### B\.3Efficiency over Release Date

Fig\.[13](https://arxiv.org/html/2610.14389#A2.F13)places each complete configuration at its checkpoint release date\. Energy per score point spans114×114\\times,59×59\\times, and27×27\\timesacross the three reference platforms, but shows no detectable trend with release date: the fitted multiplicative change per year is1\.18×1\.18\\times,1\.77×1\.77\\times, and1\.07×1\.07\\times, with all 95% intervals including one\.

Scores do increase with release date\. Among non\-thinking configurations, the fitted increase is 22–29% per year on all three platforms, while the energy of thinking configurations increases at least as quickly on the two single\-GPU platforms\. The best observed energy per score point improves by32×32\\times,12×12\\times, and5×5\\timesrelative to the oldest configurations, although the current records were already established between April 2025 and April 2026\.

Within lineages at matched model size, the 30 successor transitions change energy per score point by a median of0\.97×0\.97\\times\. Non\-thinking successors gain 7\.6 score points at approximately unchanged energy, whereas thinking successors lose 1\.3 points while using0\.94×0\.94\\timesthe energy\. These temporal trends should be interpreted cautiously: the roster contains few releases before mid\-2025 and is not a random sample of the model ecosystem\.

Figure 13:Efficiency, capability, and energy over release date\.Each complete configuration is placed at its checkpoint release date on its tier’s reference platform, across all weight formats\. Rows show energy per score point, suite score, and suite energy\. Solid steps show the best value observed up to each date; dashed and dotted lines show log\-scale least\-squares trends for non\-thinking and thinking configurations\. Marker encoding follows Fig\.[1](https://arxiv.org/html/2610.14389#S5.F1)\.

## Appendix CServing Outside the Saturated Regime

### C\.1Batch Size 1

On the 8×\\timesH100 node, we re\-served the AIME 2026 and LiveCodeBench traffic of seven configurations with one request in flight, preserving the prompt and generation lengths of the benchmark requests\. After warm\-up, each configuration was metered for 256 s\. The seven configurations comprise three MoE checkpoints in both reasoning modes and the dense DeepSeek\-R1\-Distill\-Llama\-70B\.

At batch size 1, the node draws 1\.7–1\.9 kW for the MoE configurations and 3\.2 kW for the dense model, while generating 71–190 tokens/s\. For a given configuration, energy per generated token varies by at most1\.08×1\.08\\timesbetween the two benchmarks and increases with active parameter count, from 9\.8 J at 12B active parameters to 33 J for the dense 70B model\.

Relative to the corresponding campaign runs, energy per generated token is6\.6×6\.6\\timeshigher at batch size 1 on average \(geometric mean;2\.92\.9–18\.5×18\.5\\timesacross configuration–benchmark pairs\)\. Throughput falls by11×11\\times, while power falls by only1\.5×1\.5\\times\. These results cover seven configurations on one platform and quantify the sensitivity to batching rather than a universal batch\-size penalty\.

Figure 14:Batch size 1 against the campaign serving regime on the 8×\\timesH100 node\.Energy per generated token \(*left*\) and throughput \(*right*\) for seven configurations on AIME 2026 and LiveCodeBench\. Open markers show one request at a time; filled markers show the corresponding campaign runs, with multiplicative gaps annotated\. Squares show geometric means by benchmark, configuration, and across all 14 pairs\. Colour denotes lineage; circles are non\-thinking and hexagons thinking configurations\.
### C\.2Weights Offloaded to CPU Memory

The models in the 512 GiB tier do not fit within the GH200’s 96 GiB of HBM3 under our main serving configuration\. We therefore serve two of them on a single GH200 using the serving engine’s CPU\-offload mechanism, which places weights that do not fit in HBM3 in Grace LPDDR5X and transfers them over NVLink\-C2C during inference\. The two configurations are Qwen3\.5\-122B\-A10B \(68 of 118 GiB offloaded; 57%\) and Qwen3\.5\-397B\-A17B \(344 of 378 GiB; 91%\), both FP8 without thinking\. We evaluate MMLU\-Pro, AIME 2026, and LiveCodeBench under otherwise unchanged campaign settings\.

For comparison, we fit a least\-squares trend on log scales against active parameter count using the seven FP8 non\-thinking 96 GiB configurations whose weights fit entirely in HBM3 on the same GH200 \(3–30B active parameters\)\. Relative to this trend, the offloaded runs take77–19×19\\timeslonger, achieve only0\.040\.04–0\.22×0\.22\\timesthe throughput, and use1\.81\.8–7\.2×7\.2\\timesmore energy per generated token\. The energy penalty is smaller than the runtime penalty because GPU\+HBM power falls to 200–250 W during the offloaded runs, compared with 290–700 W for the in\-memory configurations\. The result is robust to the choice of reference configurations: throughput remains0\.050\.05–0\.26×0\.26\\timesand energy per token1\.51\.5–7\.2×7\.2\\timesthe fitted trend\.

With 14–131 requests in flight, a uniform\-routing approximation implies that each decode step accesses 27–97% of the experts, corresponding to roughly 180–330 GiB/s of offloaded\-weight traffic over NVLink\-C2C\. Together with the large throughput loss and reduced accelerator power, this is consistent with offloaded\-weight traffic becoming the serving bottleneck\.

Offloading changes serving cost much more than capability\. Scores differ from the same models on the 8×\\timesH100 node by at most 1\.2 points on MMLU\-Pro and LiveCodeBench, and by at most 6 points on the 30\-problem AIME 2026\. At full module scope, including the Grace CPU and memory holding the offloaded weights, energy rises to1\.41\.4–1\.5×1\.5\\timesthe primary GPU\+HBM measurement, compared with1\.11\.1–1\.25×1\.25\\timeswithout offloading\.

Figure 15:Weights offloaded to Grace memory on the GH200\.Minutes per attempt \(*left*\), throughput \(*middle*\), and energy per generated token from the GPU\+HBM measurement \(*right*\) against active parameter count for MMLU\-Pro, AIME 2026, and LiveCodeBench\. Small points show FP8 non\-thinking configurations that fit entirely in HBM3; dashed lines and bands show their log\-scale least\-squares trends and±1\\pm 1residual standard deviation\. Open markers show Qwen3\.5\-122B\-A10B \(57% offloaded\) and Qwen3\.5\-397B\-A17B \(91%\), labelled by their ratio to the fitted trend\.

## Appendix DEnergy Consumed by This Study

Fig\.[16](https://arxiv.org/html/2610.14389#A4.F16)accounts for the primary accelerator energy consumed by the completed evaluation campaign\. Across all platforms and benchmarks, the experiments consumed approximately 1\.34 MWh\. This total includes all metered benchmark attempts, including side experiments and attempts whose results were later replaced or discarded\.

The 8×\\timesH100 node accounts for the largest share, at 443 kWh \(33%\), followed by the 2×\\timesRTX A6000 platform at 349 kWh \(26%\)\. The RTX PRO 6000, RTX 5090, GH200, and GB10 account for 178, 151, 128, and 89 kWh, respectively\. By benchmark, LiveCodeBench and AIME 2026 each account for about one fifth of the total, followed by AIME 2025 and GPQA Diamond at about 16% each; IFEval accounts for less than 3%\.

These values use the primary accelerator\-level measurement boundary defined in Appendix[A\.3](https://arxiv.org/html/2610.14389#A1.SS3)\. They therefore exclude power\-supply losses, cooling, host power on the discrete\-GPU systems, model\-loading energy, and idle periods between runs\. At $0\.20/kWh, the measured accelerator energy corresponds to approximately $268 of electricity\.

Figure 16:Primary accelerator energy consumed by the completed evaluation campaign\.Energy is broken down by platform \(*left*\) and benchmark \(*right*\), totalling approximately 1\.34 MWh\. The accounting includes all metered benchmark attempts, including side experiments and attempts later replaced\. Model loading, idle periods, host power, and other infrastructure overheads are excluded\.