EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization

# Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization
Source: [https://arxiv.org/html/2605.14249](https://arxiv.org/html/2605.14249)
Zhiye Song Massachusetts Institute of Technology Cambridge, MA 02139 zhiye@mit\.edu &Kyungmi Lee Massachusetts Institute of Technology Cambridge, MA 02139 kyungmi@mit\.edu &Eun Kyung Lee IBM Research Yorktown Heights, NY 10598 eunkyung\.lee@us\.ibm\.com &Xin Zhang IBM Research Yorktown Heights, NY 10598 xzhang@us\.ibm\.com &Tamar Eilam IBM Research Yorktown Heights, NY 10598 eilamt@us\.ibm\.com &Anantha P\. Chandrakasan Massachusetts Institute of Technology Cambridge, MA 02139

###### Abstract

We present EnergyLens, an end\-to\-end framework for energy\-aware[large language model](https://arxiv.org/html/2605.14249#id3.3.id3)\([LLM](https://arxiv.org/html/2605.14249#id3.3.id3)\) inference optimization\. As[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)s scale, predicting and reducing their energy footprint has become critical for sustainability and datacenter operations, yet existing approaches either require production\-level code and expensive profiling or fail to accurately capture multi\-GPU energy behavior\. As a result, practitioners lack tools for deciding which optimizations to prioritize and for selecting among existing deployment configurations when exhaustive profiling is impractical\. EnergyLens addresses this gap with an intuitive einsum\-based interface that captures[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)specifications including fusion, parallelism, and compute\-communication overlap, combined with load\-imbalance\-aware MoE modeling and an empirically driven communication energy model for multi\-GPU settings\. We validate EnergyLens on Llama3 and Qwen3\-MoE across tensor\-parallel and expert\-parallel configurations, achieving[mean absolute percentage errors](https://arxiv.org/html/2605.14249#id5.5.id5)between 9\.25% and 13\.19% for multi\-GPU prefill and decode energy, and 12\.97% across SM allocations for Megatron\-style overlap\. Our energy\-driven exploration reveals up to1\.47×1\.47\\timesand52\.9×52\.9\\timesenergy variation across configurations in prefill and decode efficiency and motivates distributed serving\. We further show that compute\-communication overlap is difficult to optimize with intuition alone, but EnergyLens correctly identifies Pareto\-optimal overlap configurations\.

## 1Introduction

The arrival of generative AI, with[LLMs](https://arxiv.org/html/2605.14249#id3.3.id3)at its core, has transformed machine learning\. As[LLMs](https://arxiv.org/html/2605.14249#id3.3.id3)scale to hundreds of billions of parameters and serve an ever\-growing user base, their compute demands translate into substantial energy costs and carbon footprints\(Faiz et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib5);Tschand et al\.,[2024](https://arxiv.org/html/2605.14249#bib.bib26);Henderson et al\.,[2020](https://arxiv.org/html/2605.14249#bib.bib8);Strubell et al\.,[2020](https://arxiv.org/html/2605.14249#bib.bib25)\)\. Understanding and optimizing inference energy has therefore become critical for both sustainability and datacenter provisioning and serving efficiency\.

Meeting this challenge requires energy models that are useful throughout the[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)inference development cycle, not only after deployment\. In practice, practitioners need energy estimates in two settings: before implementation, to decide which optimizations such as fusion, parallelism, or compute\-communication overlap to prioritize, and during deployment, to select among configurations as workloads and latency targets vary, when exhaustive profiling is too expensive\. A model for space exploration is especially important in settings where the Pareto front is difficult to infer from intuition alone\. Existing approaches each address part of this problem but fall short for energy\-aware[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)optimization\. Direct measurement is accurate, but requires GPU access and significant engineering effort to implement every configuration, making broad exploration slow and costly\. Coarse holistic methods, including[thermal design power](https://arxiv.org/html/2605.14249#id13.13.id13)\([TDP](https://arxiv.org/html/2605.14249#id13.13.id13)\)\-based estimation\(Lacoste and Dandres,[2019](https://arxiv.org/html/2605.14249#bib.bib12);Luccioni et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib19)\), are easy to apply but miss the strong power variations between[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)inference phases; in our Llama3\-70B measurements on A100 GPUs, TDP\-based method can overestimate decode energy by up to 60%\. LLMCO2\(Fu et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib6)\)predicts power but does not natively support fused implementations or[mixture of experts](https://arxiv.org/html/2605.14249#id17.17.id17)\([MoE](https://arxiv.org/html/2605.14249#id17.17.id17)\) architectures, nor does it provide the energy attribution needed to understand optimization opportunities\. Kernel\-level predictors such as EnergAIzer\(Lee et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib13)\)provide compute\-kernel attribution, but do not capture the multi\-GPU effects that increasingly dominate practical[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)serving, including parallelism and compute\-communication overlap\.

These limitations are especially important because modern serving stacks increasingly combine distributed execution with architectural diversity\. Models are commonly deployed across multiple GPUs to satisfy memory and latency constraints\(Shoeybi et al\.,[2019](https://arxiv.org/html/2605.14249#bib.bib24);Aminabadi et al\.,[2022](https://arxiv.org/html/2605.14249#bib.bib2)\), while[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)models introduce additional expert\-parallel execution and routing imbalance that change both kernel shapes and per\-GPU work distribution\. As multi\-GPU parallelism places greater demand on the network, accurate communication energy modeling becomes essential\. Otherwise, significant optimization opportunities can be missed\. Take Llama3\-70B with 8\-GPU tensor parallel as an example, communication accounts for 23% of total energy\. Compute\-communication overlap\(NVIDIA,[a](https://arxiv.org/html/2605.14249#bib.bib20);Hong et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib9)\)further complicates latency\-energy trade\-offs in ways that intuition and existing tools cannot capture\. The resulting configuration space spans fusion strategies, parallelism options, overlap settings, batch sizes, each interacting with sequence lengths and latency targets\. This combinatorial explosion makes exhaustive profiling impractical\.

EnergyLens addresses this gap by enabling rapid exploration of the energy\-latency trade\-off space without requiring existing code or GPU access\. Developers specify[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)architectures with fusion, parallelism, and overlap settings in a few lines of code using our simple interface\. The framework parses these specifications and produces detailed energy breakdowns and energy\-latency trade\-offs with empirically driven, overlap\-aware communication modeling and load\-imbalance\-aware[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)modeling\. For example, comparing two fusion settings for Llama3\-70B prefill across 7 batch sizes and 32 input lengths would require profiling on an 8\-GPU server for 12 hours, plus substantial engineering effort\. Using EnergyLens, generating the corresponding results and visualizations only takes 15 minutes\.

Our main contributions are as follows:

- •We introduce EnergyLens, a framework that lets practitioners describe dense and[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)inference in a few lines of einsum\-based interface with parallelism and overlap annotations, and predicts detailed energy and latency breakdowns without requiring execution traces, production implementations, or GPU access\.
- •We develop a distributed energy modeling stack that combines empirically driven communication energy modeling, overlap\-aware aggregation, and load\-imbalance\-aware[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)modeling, enabling support for tensor parallelism, expert parallelism, and Megatron\-style compute\-communication overlap\.
- •We validate EnergyLens on Llama3 and Qwen3\-MoE, achieving[mean absolute percentage errors](https://arxiv.org/html/2605.14249#id5.5.id5)\([MAPEs](https://arxiv.org/html/2605.14249#id5.5.id5)\) between 9\.25% and 13\.19% for multi\-GPU energy and 12\.97% for overlap evaluations\. We show that these errors are small enough to recover the best deployment configurations, including non\-intuitive overlap\-related trade\-offs\.

## 2Related Work

### 2\.1Kernel\-Level Performance Modeling

Several kernel performance models\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16);Lee et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib13)\)offer faster alternatives to cycle\-accurate GPU simulation\(Kandiah et al\.,[2021](https://arxiv.org/html/2605.14249#bib.bib11)\)\. We adopt EnergAIzer\(Lee et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib13)\)as our primary compute kernel backend because it provides both latency and power estimates; our framework’s modular design also supports alternative latency backends including NeuSight\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15)\)and Li et al\.\(Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\)\.

### 2\.2LLM Performance Modeling

Existing[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)performance modeling spans both training\(Liang et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib18);Li et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib17);Won et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib27)\)and inference\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Cho et al\.,[2024](https://arxiv.org/html/2605.14249#bib.bib3);Agrawal et al\.,[2024](https://arxiv.org/html/2605.14249#bib.bib1)\), but these tools share two key limitations: they focus on latency and throughput while neglecting energy, and most require execution traces from existing implementations, precluding early\-stage design exploration\. LLMCO2\(Fu et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib6)\)prioritizes energy, but does not produce the energy breakdowns needed to facilitate optimization opportunities\. Moreover, the graph neural network assumes the unfused version of the standard transformer \(e\.g\. Llama2\)\. Applying operation fusion alters this topology, requiring profiling of the fused implementation and retraining the model, neither of which is addressed in the paper\. LLMServingSim v2\.0\(Cho et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib4)\)adds a coarse three\-state power model to its performance\-focused model, but uses a single power number for active GPUs, missing the diverse power profiles such as between prefill and decode phases\. EnergyLens addresses all these gaps: energy\-centric modeling, breakdown visibility, and execution\-trace\-free input\.

### 2\.3Compute\-Communication Overlap

Recent work overlaps tensor\-parallel collectives with GEMMs to hide communication latency in distributed training; this is also increasingly explored in the prefill phase of distributed inference\(NVIDIA,[a](https://arxiv.org/html/2605.14249#bib.bib20);Lee et al\.,[2025a](https://arxiv.org/html/2605.14249#bib.bib14);Hong et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib9)\)\. Predictive modeling of its energy impact, however, remains unexplored\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Fu et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib6);Cho et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib4)\)\.Lee et al\.\([2025a](https://arxiv.org/html/2605.14249#bib.bib14)\)characterizes overlap empirically but does not predict performance\.Hong et al\.\([2025](https://arxiv.org/html/2605.14249#bib.bib9)\)optimizes overlap strategies using a latency model that assumes a fixed SM allocation for communication and linear scaling of GEMM partitions\. This latency model deteriorates with varying SM allocations or under smaller workloads\.

## 3EnergyLens Framework

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/framework2.png)Figure 1:EnergyLens framework enables energy\-latency optimization in the high\-dimensional exploration space at early stages of model implementation and deployment\.[Figure˜1](https://arxiv.org/html/2605.14249#S3.F1)illustrates the EnergyLens framework\. The*EnergyLens Interface*\([Section˜3\.1](https://arxiv.org/html/2605.14249#S3.SS1)\) provides an einsum\-based specification for describing LLM architectures, fusion strategies, and parallelism settings\. A*Lightweight Interpreter*\([Section˜3\.2](https://arxiv.org/html/2605.14249#S3.SS2)\) parses these specifications to extract GEMM dimensions, arithmetic intensity, and required communication kernels\. It also supports load imbalance modeling in[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)models\. The resulting kernel information feeds into an*Overlap\-Aware Communication Model*\([Section˜3\.3](https://arxiv.org/html/2605.14249#S3.SS3)\), which captures multi\-GPU communication energy, and a*Compute Kernel Model*that leverages EnergAIzer\(Lee et al\.,[2026](https://arxiv.org/html/2605.14249#bib.bib13)\)for single\-GPU kernel latency and power estimation, with support for alternative compute latency backends such as NeuSight and Li’s model\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\)\. The kernel\-level results go through an aggregation stage that accounts for compute\-communication overlap \([Section˜3\.4](https://arxiv.org/html/2605.14249#S3.SS4)\) to produce end\-to\-end energy breakdowns and energy\-latency Pareto fronts\.

### 3\.1EnergyLens Interface

EnergyLens exposes an einsum\-based interface for specifying LLM inference operators, with added lightweight annotations for parallelism and overlap\. We choose einsum notation because it is widely used in frameworks such as PyTorch for expressing tensor operations\(PyTorch Foundation,[2025](https://arxiv.org/html/2605.14249#bib.bib23)\), enabling intuitive descriptions that are easily extended to distributed implementations\.

[Figure˜2](https://arxiv.org/html/2605.14249#S3.F2)shows a complete fused[MoE](https://arxiv.org/html/2605.14249#id17.17.id17)specification in a few lines of code, with symbols defined in[Table˜2](https://arxiv.org/html/2605.14249#A2.T2)\. Fused and unfused dense model examples can be found in[Appendix˜B](https://arxiv.org/html/2605.14249#A2)\. For example, the fused QKV projection is written asbsm,miKh\-\>bsiKh, and tensor parallelism is specified by annotating the sharded dimension \(e\.g\.,parallel="K"\)\. The same specification covers both prefill and decode:ssdenotes sequence length in prefill ands=1s\{=\}1in decode\. Model dimensions are supplied at runtime, so one specification can represent models at different scales\.

llm\_MoE\.EQS=\[

op\("bsm,miKh\-\>bsiKh",parallel="K",label="QKV␣Projection"\),\|\\label\{line:qkv\}\|

op\("attention",parallel="H",attn\_eqs=attn\_eqs,label="Attention"\),\|\\label\{line:attn\}\|

op\("bsHh,Hhm\-\>bsm",parallel="H",label="Output␣Projection"\),\|\\label\{line:out\}\|

op\("bsm,Em\-\>bsE",label="Router"\),\|\\label\{line:router\}\|

op\("bsm\-\>bsmA",label="Scatter"\),\|\\label\{line:scatter\}\|

op\("ETm,EmF\-\>ETF",parallel="E",label="Gate␣&␣Up␣Projection"\),\|\\label\{line:gateup\}\|

op\("ETf,Efm\-\>ETm",parallel="E",label="Down␣Projection"\),\|\\label\{line:down\}\|

op\("bsAm,bsA\-\>bsm",parallel="A",label="Reduction"\)\|\\label\{line:reduce\}\|

\]

Figure 2:Full specification of a fused MoE implementation with[TP](https://arxiv.org/html/2605.14249#id4.4.id4)in the attention block and[EP](https://arxiv.org/html/2605.14249#id15.15.id15)in the MoE block \(symbols andattn\_eqsdefined in[Appendix˜B](https://arxiv.org/html/2605.14249#A2)\)\.Alternative fusion strategies are expressed by modifying only the symbolic operator descriptions\. For instance,[Figure˜2](https://arxiv.org/html/2605.14249#S3.F2)fuses Q, K, and V projections into one operator \(lineLABEL:line:qkv\) and combines the gate and up projections \(lineLABEL:line:gateup\)\. EnergyLens also supports distributed execution by annotating the partitioned dimension, e\.g\.,parallel="m"for tensor parallelism andCP\_dim="s"for context parallelism\. This compact interface enables rapid exploration of the energy\-latency optimization space across[input sequence length](https://arxiv.org/html/2605.14249#id8.8.id8)\([ISL](https://arxiv.org/html/2605.14249#id8.8.id8)\),[output sequence length](https://arxiv.org/html/2605.14249#id9.9.id9)\([OSL](https://arxiv.org/html/2605.14249#id9.9.id9)\), batch size, model size, and parallelism\.

### 3\.2Lightweight Interpreter and Aggregator

The lightweight interpreter maps each einsum specification to kernel\-level features used for energy and latency prediction\. For compute operators, it extracts the corresponding[general matrix multiplication](https://arxiv.org/html/2605.14249#id11.11.id11)\([GEMM](https://arxiv.org/html/2605.14249#id11.11.id11)\) dimensions in the standard formA​\[M,K\]×B​\[K,N\]=C​\[M,N\]A\[M,K\]\\times B\[K,N\]=C\[M,N\]\. The interpreter also computes arithmetic intensity to characterize whether an operator is compute\- or memory\-bound\.

For distributed configurations, the interpreter derives the required communication kernels from the einsum equation and the annotated sharded dimension\. For example, if the parallelized dimension is reduced, the operator requires a collective communication step, whose message size is determined from the output tensor shape and data type \(details in[Appendix˜C](https://arxiv.org/html/2605.14249#A3)\)\.

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/paper_prefill_T.png)\(a\)Tokens per expert,TT, in prefill \(batch size = 4,[EP](https://arxiv.org/html/2605.14249#id15.15.id15)= 4\)\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/paper_decode_A.png)\(b\)The number of activated experts per GPU,EE, in decode \([EP](https://arxiv.org/html/2605.14249#id15.15.id15)= 4\)\.

Figure 3:MoE routing statistics showing the average load and the load of the bottleneck GPU\.For MoE models, the dimension derivation is more involved, especially when expert\-parallel execution depends on how tokens are routed among experts\. We therefore introduce two effective quantities:EE, the number of activated experts per GPU andTT, the effective number of tokens per expert, after accounting for both load imbalance and grouped\-GEMM tile quantization\. When routing statistics are unavailable, EnergyLens can assume uniform routing to support early\-stage exploration\. When representative workloads are available, these effective values can be computed from a functional model execution to improve accuracy\. Importantly, this requires neither an optimized implementation nor distributed profiling: data collected from a single\-GPU execution can be transformed into the per\-GPU token distribution under expert parallelism\.

We extract a max pair\(Tmax,Emax\)\(T\_\{\\max\},E\_\{\\max\}\)from the most heavily loaded GPU for latency estimation, and an average pair\(Tavg,Eavg\)\(T\_\{\\text\{avg\}\},E\_\{\\text\{avg\}\}\)across all GPUs for energy estimation\.[Figure˜3](https://arxiv.org/html/2605.14249#S3.F3)shows example statistics\. For prefill, most experts are activated, butTm​a​xT\_\{max\}grows faster thanTa​v​gT\_\{avg\}due to load imbalance\. In decode, each request produces only one token; the grouped\-GEMM kernel enforces a minimum tile granularity of 16 tokens per expert regardless of batch size, though larger batches activate more experts\.

Given either ideal or measured routing statistics, per\-GPU latency is set by the bottleneck, while per\-GPU energy accounts for idle waiting during load imbalance:

ℒ=∑iℒi​\(Tmax,Emax\),ℰ=∑i\[ℰi​\(Tavg,Eavg\)\+\(ℒi​\(Tmax,Emax\)−ℒi​\(Tavg,Eavg\)\)⋅Pidle\],\\mathcal\{L\}=\\sum\_\{i\}\\mathcal\{L\}\_\{i\}\(T\_\{\\max\},E\_\{\\max\}\),\\qquad\\mathcal\{E\}=\\sum\_\{i\}\\Big\[\\mathcal\{E\}\_\{i\}\(T\_\{\\text\{avg\}\},E\_\{\\text\{avg\}\}\)\\;\+\\;\(\\mathcal\{L\}\_\{i\}\(T\_\{\\max\},E\_\{\\max\}\)\-\\mathcal\{L\}\_\{i\}\(T\_\{\\text\{avg\}\},E\_\{\\text\{avg\}\}\)\)\\cdot P\_\{\\text\{idle\}\}\\Big\],\(1\)whereℒi\\mathcal\{L\}\_\{i\}andℰi\\mathcal\{E\}\_\{i\}are the latency and energy of theii\-th operation, andPidleP\_\{\\text\{idle\}\}is the GPU idle power\.

### 3\.3Communication Energy Modeling

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/reduce_scatter_sm_variants_gpu8_latency.png)Figure 4:8\-GPU ReduceScatter is slower when fewer[SMs](https://arxiv.org/html/2605.14249#id12.12.id12)are dedicated to communication\.We propose an empirically\-driven and overlap\-aware communication energy model\. Unlike prior work that assumes fixed bandwidth utilization and only estimates latency\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15)\), we profile latency and energy consumption across various communication kernels and transfer sizes and use interpolation during prediction\. Overhead dominates at small workload sizes, resulting in higher latency and energy per bit \([Figure˜14](https://arxiv.org/html/2605.14249#A4.F14)in[Appendix˜D](https://arxiv.org/html/2605.14249#A4)\)\. This behavior is missed by bandwidth\-and\-queuing\-based models used in prior works\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\)\.[Figure˜4](https://arxiv.org/html/2605.14249#S3.F4)further shows that the number of SMs dedicated to communication impacts latency, creating a trade\-off between SM allocations for compute and communication during overlap\.

### 3\.4Compute\-Communication Overlap

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/down_projection_tp2.png)\(a\)Down projection with TP2\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/overlap_timeline.png)\(b\)Timeline \(RS: ReduceScatter; AG: AllGather\)\.

Figure 5:EnergyLens enables intuitive specification of parallelism and overlap settings\.Distributed inference frameworks increasingly overlap tensor\-parallel collectives with GEMMs to hide communication latency\(NVIDIA,[a](https://arxiv.org/html/2605.14249#bib.bib20)\)\. EnergyLens models such overlap with three additions to the einsum specification\.overlap\_stagesets the number of stages into which compute and communication are partitioned \([Figure˜5\(b\)](https://arxiv.org/html/2605.14249#S3.F5.sf2)\);overlap\_SMsets the number of SMs dedicated to communication; andoverlapspecifies the dimension along which this partitioning is applied\. For example, a down projection with four\-stage overlap and 16 SMs reserved for communication is written asop\("bsf,fm\-\>bsm",parallel="f",overlap\_stage=4,overlap\_SM=16,overlap="s",label="DownProjection"\)\.

EnergyLens models an example GEMM\-AllReduce overlapped layer as three phases in[Figure˜5\(b\)](https://arxiv.org/html/2605.14249#S3.F5.sf2): \(i\) the first non\-overlapped compute stage using all SMs, \(ii\) overlapped stages whose duration is the maximum of the SM\-restricted GEMM kernel and the SM\-restricted NCCL kernel, and \(iii\) a final exposed communication stage using all SMs\. Stage latencies are obtained by invoking our SM\-allocation\-aware model for NCCL and using the kernel\-level model on the partitioned GEMM\. The total latency is then calculated as

tt​o​t​a​l=tG​E​M​M\+tR​S\+max⁡\(tG​E​M​Mo​v​e​r​l​a​p,tR​So​v​e​r​l​a​p\)×\(s​t​a​g​e−1\),t\_\{total\}=t\_\{GEMM\}\+t\_\{RS\}\+\\max\\left\(t\_\{GEMM\_\{overlap\}\},t\_\{RS\_\{overlap\}\}\\right\)\\times\(stage\-1\),\(2\)wheretG​E​M​Mt\_\{GEMM\}and related terms denote per\-stage latencies\. The latency of an overlapped stage is larger than that of a non\-overlapped stage due to fewer SMs being available\. The impact of SM allocation on an example communication kernel is shown in[Figure˜4](https://arxiv.org/html/2605.14249#S3.F4)\. Prior overlap latency model assumes that a quarter\-sized partition of GEMM would take a quarter of the latency\(Hong et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib9)\), which is not accurate at smaller workloads due to decreased algorithmic intensity and kernel overhead\. In contrast, EnergyLens queries the compute kernel model for the quarter\-sized partition, which is important at small batch\-sequence products to accurately determine whether multi\-stage overlap is profitable\.

For overlapped stages we attribute power using the GEMM estimate, since the majority of SMs and memory bandwidth serve compute\. For the exposed communication stage we use the same overlapped\-stage power rather than steady\-state NCCL power; we find the latter underestimates energy, potentially because GPU DVFS cannot adjust voltage during such short kernel durations\.

### 3\.5Energy Trade\-offs

We introduce energy\-centric metrics analogous to traditional latency metrics for LLM inference:[energy to first token](https://arxiv.org/html/2605.14249#id2.2.id2)\([ETFT](https://arxiv.org/html/2605.14249#id2.2.id2)\) measures the energy consumption during prefill \(similar to[time to first token](https://arxiv.org/html/2605.14249#id1.1.id1)\([TTFT](https://arxiv.org/html/2605.14249#id1.1.id1)\)\), while[energy per output token](https://arxiv.org/html/2605.14249#id6.6.id6)\([EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)\) measures decode phase energy normalized by[OSL](https://arxiv.org/html/2605.14249#id9.9.id9)\(analogous to[time per output token](https://arxiv.org/html/2605.14249#id7.7.id7)\([TPOT](https://arxiv.org/html/2605.14249#id7.7.id7)\)\)\. Both[ETFT](https://arxiv.org/html/2605.14249#id2.2.id2)and[EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)are normalized per request\. EnergyLens’s interface enables users to directly explore these workload\-level energy metrics, along with traditional latency and throughput metrics, from a few lines of model descriptions\.

## 4Evaluation

### 4\.1Evaluation Methods

We evaluate on Llama3\-8B, Llama3\-70B, and Qwen3\-30B\-A3B as representative dense and MoE models\(Grattafiori et al,[2024](https://arxiv.org/html/2605.14249#bib.bib7);Yang et al\.,[2025](https://arxiv.org/html/2605.14249#bib.bib28)\)\. Single\-GPU evaluation is detailed in[Appendix˜E](https://arxiv.org/html/2605.14249#A5), where we validate that high\-level einsum specifications are sufficient to predict end\-to\-end prefill and decode energy before introducing multi\-GPU effects\. Ground\-truth measurements use TensorRT\-LLM\(NVIDIA,[c](https://arxiv.org/html/2605.14249#bib.bib22)\)on a server with 8 A100\-SXM4\-80GB GPUs\. To capture steady\-state behavior, we apply a 20 s warmup before measurement\. Each workload is then characterized for at least one full iteration and at least 40 s\. We measure end\-to\-end latency directly and GPU energy via[NVIDIA Management Library](https://arxiv.org/html/2605.14249#id14.14.id14)\([NVML](https://arxiv.org/html/2605.14249#id14.14.id14)\)\. We use Torch Profiler\(PyTorch Foundation,[2025](https://arxiv.org/html/2605.14249#bib.bib23)\)to validate that the communication collectives are predicted correctly\. We also extract the profiled latency breakdown and confirm the actual batch sizes used by TensorRT\-LLM, which may differ from the requested values at long contexts\. Execution traces also verified that there is no overlap between compute and communication kernels in these experiments\.

Table 1:\\AcfpMAPE on dense and MoE models are sufficient for design space optimization\.Llama3\-8BLlama3\-70BQwen3\-30B\-A3BSingle GPU\\AcTP\\AcTP \(Overlap\)Single GPU[tensor parallelism](https://arxiv.org/html/2605.14249#id4.4.id4)\([TP](https://arxiv.org/html/2605.14249#id4.4.id4)\)\+[EP](https://arxiv.org/html/2605.14249#id15.15.id15)PrefillEnergy11\.31%12\.18%12\.97%12\.13%9\.25%Latency11\.30%12\.39%11\.18%12\.17%21\.85%DecodeEnergy12\.87%12\.86%N/A†13\.46%13\.19%Latency15\.21%25\.45%N/A†16\.83%27\.16%†Multi\-stage compute\-communication overlap is a prefill technique and does not apply to decodebecause the smaller partition would further reduce the compute intensity of the GEMM operation\.For compute\-communication overlap experiments \([Section˜4\.3](https://arxiv.org/html/2605.14249#S4.SS3)\), we additionally use Megatron\-LM v0\.15\.3\(Shoeybi et al\.,[2019](https://arxiv.org/html/2605.14249#bib.bib24)\)as ground truth\. Profiler traces reveal that tensor\-parallel all\-reduce is decomposed as a reduce\-scatter overlapped with GEMM followed by an exposed all\-gather\.

### 4\.2Multi\-GPU Inference Evaluation

We report the overall prediction errors on dense and MoE models with various parallelism configurations in[Table˜1](https://arxiv.org/html/2605.14249#S4.T1)\(sweep parameters specified in[Appendix˜M](https://arxiv.org/html/2605.14249#A13)\)\. For prefill, most operators are compute\-bound at practical[ISLs](https://arxiv.org/html/2605.14249#id8.8.id8), so further increasing arithmetic intensity through batching yields less energy efficiency gain than lower tensor parallelism\.[Figure˜6\(a\)](https://arxiv.org/html/2605.14249#S4.F6.sf1)shows that compute energy remains nearly constant across[TP](https://arxiv.org/html/2605.14249#id4.4.id4)configurations, while AllReduce cost grows with TP\. At TP8, AllReduce accounts for 23% of total energy\.[Figure˜6\(b\)](https://arxiv.org/html/2605.14249#S4.F6.sf2)further shows that the predicted latency breakdown closely matches the measured execution trace, validating the kernel\-level decomposition used for energy attribution\. We observe the same trend on Qwen3\-MoE \([Figure˜19](https://arxiv.org/html/2605.14249#A8.F19)in[Appendix˜H](https://arxiv.org/html/2605.14249#A8)\): compute energy is stable across[TP](https://arxiv.org/html/2605.14249#id4.4.id4)degrees while communication overhead grows\.

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_energy_by_tp_batch2_in4096.png)\(a\)Prefill energy breakdown\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_latency_by_tp_batch2_in4096.png)\(b\)Prefill latency breakdown\.

Figure 6:Llama3\-70B prefill phase at B2, ISL=4096 across tensor parallelism configurations \(TP2, TP4, TP8\), showing increasing communication overhead as TP degree increases\. The grey bars are the measurement, and the rightmost bars in each cluster are the EnergyLens prediction\.![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/kernel_energy_stacked_bar_tep2_no32_64.png)\(a\)Decode energy breakdown\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/kernel_latency_stacked_bar_tep2_no32_64.png)\(b\)Decode latency breakdown\.

Figure 7:Qwen3\-30B\-A3B decode phase at TP2 in attention layers and EP2 in MoE layers, with ISL=512 and OSL=1024\. Larger batches substantially reduce decode energy per token\.Decode exhibits a different trade\-off, where energy varies more strongly with batch size than with[TP](https://arxiv.org/html/2605.14249#id4.4.id4), and larger batches substantially reduce decode energy\. However, batching is constrained by GPU memory, since increasing batch size reduces the maximum sequence length that can fit in DRAM\. This trade\-off arises because decode processes 1 token per step, producing skewed GEMMs with low arithmetic intensity\. In MoE models, this effect manifests differently across operation types\. The output projections behave similarly to dense models: for example, the output projectionop\("bsHh,Hhm\-\>bsm", parallel="H"\)maps to a matmul withM=bM\{=\}bandN=mN\{=\}m\. At small batch sizes, these kernels are memory\-bound and underutilize the GPU, while scaling up batch sizes increases arithmetic intensity and reduces energy per output token\. The expert layers depend on the effective tokens per expert,TT, and activated experts per GPU,EE\. The gate and up projectionop\("ETm,EmF\-\>ETF", parallel="E"\)maps to a grouped GEMM withM=TM\{=\}TandN=FN\{=\}F, where minimum tile size forcesTTto be at least 16 regardless of batch size during decode \([Figure˜3](https://arxiv.org/html/2605.14249#S3.F3)\)\. This means that at small batch sizes, these GEMMs are also padded and severely underutilized\. While larger batches activate more experts, it also increases the number of useful tokens per expert, thereby improving utilization\.

Decode latency is harder to predict accurately, because compute backends behave worse with low\-arithmetic\-intensity kernels and highly skewed matrix shapes present in the decode phase of both standard and MoE models\. We observe similar limitations across alternative compute backends\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\), indicating that this issue is not backend\-specific \(details in[Appendix˜J](https://arxiv.org/html/2605.14249#A10)\)\. Nevertheless, EnergyLens still captures the key trend relevant for design exploration: decode latency changes only modestly with batch size, while energy improves substantially\.

### 4\.3Compute\-Communication Overlap

We validate overlap modeling on Llama3\-70B with overlap configurations including no overlap and 4\-stage overlap with 1, 4, and 16 SMs dedicated to communication \(full sweep parameters specified in[Appendix˜M](https://arxiv.org/html/2605.14249#A13)\)\. Across this sweep, EnergyLens predicts energy with a MAPE of 12\.97% and latency with a MAPE of 11\.18%\. More importantly, EnergyLens helps to predict the relative trade\-offs across SM allocations: dedicating too many SMs to communication wastes compute resources, while allocating too few makes communication the bottleneck of the overlapped stages\. The ability to include overlap as a dimension in energy\-latency trade\-off will be exploited in[Section˜5\.2](https://arxiv.org/html/2605.14249#S5.SS2)\.

## 5Energy\-Centric Exploration

The results in[Section˜4](https://arxiv.org/html/2605.14249#S4)show that EnergyLens’s energy prediction error is small relative to the variations across practical deployment choices\. What matters most for optimization is preserving the ordering between candidate configurations to identify favorable regions of the design space rather than exact pointwise estimation\. In this section, we further show that EnergyLens’ accuracy enables us to identify optimal configurations, not merely perform post\-hoc energy analysis\. We leverage this property below to derive energy\-driven insights on batching, tensor parallelism, disaggregated serving, and overlap tuning\.

### 5\.1Disaggregated Serving

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/energy_vs_latency_input4096.png)\(a\)Prefill \(ISL=4096\): TP2 dominates the Pareto frontier due to reduced communication overhead\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/decode_energy_vs_latency_input512_output1024_log.png)\(b\)Decode \(ISL=512, OSL=1024\): larger batch sizes achieve order\-of\-magnitude[EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)improvement\.

Figure 8:Energy\-latency trade\-offs for Llama3\-70B\. Configurations exceeding GPU memory are omitted\.Key Insight 1:At equal[TTFT](https://arxiv.org/html/2605.14249#id1.1.id1), smaller batch size and lower tensor parallelism is more energy efficient\.Two approaches can reduce[TTFT](https://arxiv.org/html/2605.14249#id1.1.id1): smaller batch sizes decrease compute per batch, while higher[TP](https://arxiv.org/html/2605.14249#id4.4.id4)distributes the same compute across more GPUs\.[Figure˜8\(a\)](https://arxiv.org/html/2605.14249#S5.F8.sf1)reveals that most Pareto\-optimal configurations use TP2, due to reduced communication overhead \([Figure˜6\(a\)](https://arxiv.org/html/2605.14249#S4.F6.sf1)\); the lower bound on TP degree comes from GPU memory requirements\. At[ISL](https://arxiv.org/html/2605.14249#id8.8.id8)4096, TP2 with batch size 4 is both faster and consumes 24% less energy than TP8 with batch size 16\. This is confirmed by measurements, which show 30% energy savings\. Across all ISLs for 70B prefill, B4 TP2 over B16 TP8 yields an average 20\.87% energy reduction with lower latency\.

Key Insight 2: Increasing batch size yields significant energy savings during decode, at minimal latency cost\.[Figure˜8\(b\)](https://arxiv.org/html/2605.14249#S5.F8.sf2)plots[EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)against[TPOT](https://arxiv.org/html/2605.14249#id7.7.id7): larger batch sizes dramatically reduce[EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)with only modest latency increase\. At batch size 1, skewed matrix shapes \(M=b=1M=b=1\) severely underutilize GPUs; increasing batch size shifts[GEMM](https://arxiv.org/html/2605.14249#id11.11.id11)kernels toward compute\-bound regimes, improving efficiency\. Across[OSL](https://arxiv.org/html/2605.14249#id9.9.id9)up to 7680 and TP8, batch size 16 achieves 92\.63%[EPOT](https://arxiv.org/html/2605.14249#id6.6.id6)reduction versus batch size 1\. The memory requirement of larger batch size can be mitigated with higher degrees of parallelism\. These contrasting objectives, lower batch sizes for prefill latency versus larger batch sizes for decode efficiency, strongly motivate disaggregated serving\(Zhong et al\.,[2024](https://arxiv.org/html/2605.14249#bib.bib29)\), where prefill and decode are independently optimized\.

### 5\.2Megatron\-Style Overlap

Key Insight 3:Overlap tuning is hard to reason with intuition alone, and EnergyLens identifies Pareto\-optimal configurations much more reliably than a naive maximum\-overlap approach\.

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/energy_vs_latency_pareto_input4096_compare.png)Figure 9:Llama3\-70B \(ISL=4096\): the predicted Pareto\-optimal points fall on the Pareto Front of 55 measured configurations with varying overlap settings, batch size, and tensor parallelism\.Overlap introduces another optimization dimension beyond tensor parallelism and batch size\. Instead of relying on intuition to prioritize maximum overlap, EnergyLens predicts the full configuration space and identifies the Pareto\-optimal configurations\.[Figure˜9](https://arxiv.org/html/2605.14249#S5.F9)shows this at ISL 4096 across TP degrees, batch sizes, and communication SM allocations\. In this example, the Pareto frontier is dominated by non\-overlapped TP2 settings, with overlap becoming beneficial only at the most latency\-aggressive point\. This illustrates why overlap is difficult to tune heuristically: some Pareto points avoid overlap altogether and reduce communication through lower TP\.

We measure the exact Pareto frontiers in 32 scenarios with different[ISLs](https://arxiv.org/html/2605.14249#id8.8.id8), allowing both overlapped and non\-overlapped settings\. A common intuition is to use maximum overlap \(16 SMs for communication\) and sweep only batch size and tensor parallelism\. This maximum\-overlap heuristic recovers only 20% of the exact measured Pareto frontier, because it cannot discover configurations where disabling overlap or dedicating a moderate number of SMs to communication is preferable\. In contrast, EnergyLens predicts energy and latency for all 55 configurations, selects the predicted Pareto\-optimal subset, which recovers 69% of the exact frontier\. This is achieved without any measurement\.[Figure˜9](https://arxiv.org/html/2605.14249#S5.F9)confirms that the configurations selected by EnergyLens lie on the measured Pareto frontier\. This shows that the value of EnergyLens is not merely in shrinking the search space, but in directly identifying the Pareto configurations without measurement\.

## 6Conclusion

We present EnergyLens, a framework for predicting LLM inference energy from a high\-level specification without requiring implementations, traces, or GPU access\. EnergyLens models dense and MoE models, multi\-GPU communication, and Megatron\-style overlap, achieving multi\-GPU energy[mean absolute percentage errors](https://arxiv.org/html/2605.14249#id5.5.id5)\([MAPEs](https://arxiv.org/html/2605.14249#id5.5.id5)\) between 9\.25% and 13\.19%, sufficient for identifying optimal deployment regimes\.

Despite the limitation in lower decode latency accuracy for small workloads, EnergyLens prediction is effective for energy\-aware design\-space exploration: EnergyLens reveals large energy variation across configurations, motivates disaggregated serving, and identifies Pareto overlap settings significantly better than the naive maximum\-overlap approach\.

## Acknowledgments and Disclosure of Funding

This work was supported by the MIT\-IBM Watson AI Lab and by the Natural Sciences and Engineering Research Council of Canada \(NSERC\) Postgraduate Scholarship program\.

## References

- Agrawal et al\. \(2024\)Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S Gulavani, Ramachandran Ramjee, and Alexey Tumanov\.Vidur: A Large\-Scale Simulation Framework for LLM Inference\.In*MLSys*, 2024\.
- Aminabadi et al\. \(2022\)Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He\.DeepSpeed\- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale\.*International Conference for High Performance Computing, Networking, Storage and Analysis, SC*, 2022\.ISSN 21674337\.doi:10\.1109/SC41404\.2022\.00051\.
- Cho et al\. \(2024\)Jaehong Cho, Minsu Kim, Hyunmin Choi, Guseul Heo, and Jongse Park\.LLMServingSim: A HW/SW Co\-Simulation Infrastructure for LLM Inference Serving at Scale\.In*2024 IEEE International Symposium on Workload Characterization \(IISWC\)*, pages 15–29, September 2024\.doi:10\.1109/IISWC63097\.2024\.00012\.URL[http://arxiv\.org/abs/2408\.05499](http://arxiv.org/abs/2408.05499)\.arXiv:2408\.05499 \[cs\]\.
- Cho et al\. \(2026\)Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park\.LLMServingSim 2\.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure, March 2026\.URL[http://arxiv\.org/abs/2602\.23036](http://arxiv.org/abs/2602.23036)\.arXiv:2602\.23036 \[cs\]\.
- Faiz et al\. \(2023\)Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang\.LLMCarbon: Modeling the end\-to\-end Carbon Footprint of Large Language Models\.pages 1–15, 2023\.URL[http://arxiv\.org/abs/2309\.14393](http://arxiv.org/abs/2309.14393)\.arXiv: 2309\.14393\.
- Fu et al\. \(2025\)Zhenxiao Fu, Fan Chen, Shan Zhou, Haitong Li, and Lei Jiang\.LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences\.*ACM SIGENERGY Energy Informatics Review*, 5\(2\), 2025\.
- Grattafiori et al \(2024\)Grattafiori et al\.The Llama 3 Herd of Models, 2024\.
- Henderson et al\. \(2020\)Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau\.Towards the systematic reporting of the energy and carbon footprints of machine learning\.*Journal of Machine Learning Research*, 21, 2020\.ISSN 15337928\.arXiv: 2002\.05651\.
- Hong et al\. \(2025\)Ke Hong, Xiuhong Li, Minxu Liu, Qiuli Mao, Tianqi Wu, Zixiao Huang, Lufang Chen, Zhong Wang, Yichong Zhang, Zhenhua Zhu, Guohao Dai, and Yu Wang\.Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering, October 2025\.URL[http://arxiv\.org/abs/2504\.19519](http://arxiv.org/abs/2504.19519)\.arXiv:2504\.19519 \[cs\]\.
- \(10\)Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He\.DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models\.
- Kandiah et al\. \(2021\)Vijay Kandiah, Scott Peverelle, Mahmoud Khairy, Junrui Pan, Amogh Manjunath, Timothy G\. Rogers, Tor M\. Aamodt, and Nikos Hardavellas\.AccelWattch: A power modeling framework for modern GPUs\.*Proceedings of the Annual International Symposium on Microarchitecture, MICRO*, pages 738–753, 2021\.ISSN 10724451\.doi:10\.1145/3466752\.3480063\.ISBN: 9781450385572\.
- Lacoste and Dandres \(2019\)Alexandre Lacoste and Thomas Dandres\.Quantifying the Carbon Emissions of Machine Learning\.2019\.arXiv: 1910\.09700v2\.
- Lee et al\. \(2026\)Kyungmi Lee, Zhiye Song, Eun Kyung Lee, Xin Zhang, Tamar Eilam, and Anantha P\. Chandrakasan\.EnergAIzer: Fast and Accurate GPU Power Estimation Framework for AI Workloads\.*IEEE International Symposium on Performance Analysis of Systems and Software*, 2026\.
- Lee et al\. \(2025a\)Seonho Lee, Jihwan Oh, Junkyum Kim, Seokjin Go, Jongse Park, and Divya Mahajan\.Characterizing Compute\-Communication Overlap in GPU\-Accelerated Distributed Deep Learning: Performance and Power Implications, July 2025a\.URL[http://arxiv\.org/abs/2507\.03114](http://arxiv.org/abs/2507.03114)\.arXiv:2507\.03114 \[cs\]\.
- Lee et al\. \(2025b\)Seonho Lee, Amar Phanishayee, and Divya Mahajan\.Forecasting GPU Performance for Deep Learning Training and Inference\.In*Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1*, pages 493–508, March 2025b\.doi:10\.1145/3669940\.3707265\.URL[http://arxiv\.org/abs/2407\.13853](http://arxiv.org/abs/2407.13853)\.arXiv:2407\.13853 \[cs\]\.
- Li et al\. \(2023\)Ying Li, Yifan Sun, and Adwait Jog\.Path Forward Beyond Simulators: Fast and Accurate GPU Execution Time Prediction for DNN Workloads\.In*56th Annual IEEE/ACM International Symposium on Microarchitecture*, pages 380–394, Toronto ON Canada, October 2023\. ACM\.ISBN 979\-8\-4007\-0329\-4\.doi:10\.1145/3613424\.3614277\.URL[https://dl\.acm\.org/doi/10\.1145/3613424\.3614277](https://dl.acm.org/doi/10.1145/3613424.3614277)\.
- Li et al\. \(2025\)Ying Li, Yuhui Bao, Gongyu Wang, Xinxin Mei, Pranav Vaid, Anandaroop Ghosh, Adwait Jog, Darius Bunandar, Ajay Joshi, and Yifan Sun\.TrioSim: A Lightweight Simulator for Large\-Scale DNN Workloads on Multi\-GPU Systems\.In*Proceedings of the 52nd Annual International Symposium on Computer Architecture*, pages 1524–1538, Tokyo Japan, June 2025\. ACM\.ISBN 979\-8\-4007\-1261\-6\.doi:10\.1145/3695053\.3731082\.URL[https://dl\.acm\.org/doi/10\.1145/3695053\.3731082](https://dl.acm.org/doi/10.1145/3695053.3731082)\.
- Liang et al\. \(2025\)Mingyu Liang, Hiwot Tadese Kassa, Wenyin Fu, Brian Coutinho, Louis Feng, and Christina Delimitrou\.Lumos: Efficient Performance Modeling and Estimation for Large\-scale LLM Training\.In*MLSys Conference*, 2025\.
- Luccioni et al\. \(2023\)Alexandra Sasha Luccioni, Sylvain Viguier, and Anne\-Laure Ligozat\.Estimating the carbon footprint of BLOOM, a 176B parameter language model\.*Journal of Machine Learning Research*, 24\(1\), January 2023\.ISSN 1532\-4435\.
- NVIDIA \(a\)NVIDIA\.NVIDIA NeMo Framework User Guide: Communication Overlap, a\.URL[https://docs\.nvidia\.com/nemo\-framework/user\-guide/24\.09/nemotoolkit/features/optimizations/communication\_overlap\.html](https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/features/optimizations/communication_overlap.html)\.
- NVIDIA \(b\)NVIDIA\.NCCL Tests, b\.URL[https://github\.com/NVIDIA/nccl\-tests](https://github.com/NVIDIA/nccl-tests)\.
- NVIDIA \(c\)NVIDIA\.TensorRT\-LLM, c\.URL[https://github\.com/NVIDIA/TensorRT\-LLM](https://github.com/NVIDIA/TensorRT-LLM)\.
- PyTorch Foundation \(2025\)PyTorch Foundation\.Torch Profiler, 2025\.URL[https://docs\.pytorch\.org/docs/stable/profiler\.html](https://docs.pytorch.org/docs/stable/profiler.html)\.
- Shoeybi et al\. \(2019\)Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro\.Megatron\-LM: Training Multi\-Billion Parameter Language Models Using Model Parallelism\.2019\.URL[http://arxiv\.org/abs/1909\.08053](http://arxiv.org/abs/1909.08053)\.arXiv: 1909\.08053\.
- Strubell et al\. \(2020\)Emma Strubell, Ananya Ganesh, Andrew McCallum, Ananya Ganesh, and Andrew McCallum\.Energy and Policy Considerations for Modern Deep Learning Research\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 34\(09\):13693–13696, 2020\.doi:10\.1609/aaai\.v34i09\.7123\.URL[https://ojs\.aaai\.org/index\.php/AAAI/article/view/7123](https://ojs.aaai.org/index.php/AAAI/article/view/7123)\.arXiv: 1906\.02243v1\.
- Tschand et al\. \(2024\)Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, Pawan Ambalkar, Ritika Borkar, Ramesh Chukka, Trevor Cockrell, Oliver Curtis, Grigori Fursin, Miro Hodak, Hiwot Kassa, Anton Lokhmotov, Dejan Miskovic, Yuechao Pan, Manu Prasad Manmathan, Liz Raymond, Tom St\. John, Arjun Suresh, Rowan Taubitz, Sean Zhan, Scott Wasson, David Kanter, and Vijay Janapa Reddi\.MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI\.pages 1–14, 2024\.URL[http://arxiv\.org/abs/2410\.12032](http://arxiv.org/abs/2410.12032)\.arXiv: 2410\.12032\.
- Won et al\. \(2023\)William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna\.ASTRA\-sim2\.0: Modeling Hierarchical Networks and Disaggregated Systems for Large\-model Training at Scale\.*Proceedings \- 2023 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2023*, pages 283–294, 2023\.doi:10\.1109/ISPASS57527\.2023\.00035\.arXiv: 2303\.14006 ISBN: 9798350397390\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu\.Qwen3 Technical Report, May 2025\.URL[http://arxiv\.org/abs/2505\.09388](http://arxiv.org/abs/2505.09388)\.arXiv:2505\.09388 \[cs\]\.
- Zhong et al\. \(2024\)Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang\.DistServe: Disaggregating Prefill and Decoding for Goodput\-optimized Large Language Model Serving\.In*18th USENIX Symposium on Operating Systems Design and Implementation*\. arXiv, June 2024\.doi:10\.48550/arXiv\.2401\.09670\.URL[http://arxiv\.org/abs/2401\.09670](http://arxiv.org/abs/2401.09670)\.arXiv:2401\.09670 \[cs\]\.

## Appendix AMotivation \- Power Measurements

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/power_aggregate_decode_avg.png)Figure 10:Observed GPU power consumption varies by up to 60% in the decode phase, not captured by[TDP](https://arxiv.org/html/2605.14249#id13.13.id13)\. Llama3\-70B inference on different numbers of A100\-80GB GPUs \(TP2, TP4, and TP8\) and batch sizes \(B2, B8 and B32\)\.Latency multiplied by[TDP](https://arxiv.org/html/2605.14249#id13.13.id13)does not accurately capture the energy consumption, as power varies greatly especially during the decode phase\.

## Appendix BModel Specification

[Figure˜12](https://arxiv.org/html/2605.14249#A2.F12)is used in both Llama3\-8B and Llama3\-70B evaluations in[Appendix˜E](https://arxiv.org/html/2605.14249#A5)and[Section˜4\.2](https://arxiv.org/html/2605.14249#S4.SS2)\. It corresponds to the fusion strategy used in TensorRT\-LLM verified via Torch traces\.

llm\_model\.EQS=\[

op\("bsm,mHh\-\>bsHh",parallel="H",label="Q␣Projection"\),\|\\label\{line:unfus\_q\}\|

op\("bsm,mKh\-\>bsKh",parallel="K",label="K␣Projection"\),\|\\label\{line:unfus\_k\}\|

op\("bsm,mKh\-\>bsKh",parallel="K",label="V␣Projection"\),\|\\label\{line:unfus\_v\}\|

op\("attention",parallel="H",label="Attention"\),

op\("bsHh,Hhm\-\>bsm",parallel="H",label="Output␣Projection"\),

op\("bsm,mf\-\>bsf",parallel="f",label="Gate␣Projection"\),\|\\label\{line:unfus\_gate\}\|

op\("bsm,mf\-\>bsf",parallel="f",label="Up␣Projection"\),\|\\label\{line:unfus\_up\}\|

op\("bsf,fm\-\>bsm",parallel="f",label="Down␣Projection"\)

\]

Figure 11:Full specification of an unfused Llama3 implementation\.llm\_model\.EQS=\[

op\("bsm,miKh\-\>bsiKh",parallel="K",label="QKV␣Projection"\),\|\\label\{line:fus1\}\|

op\("attention",parallel="K",attn\_eqs=attn\_eqs,label="Attention"\),

op\("bsHh,Hhm\-\>bsm",parallel="H",label="Output␣Projection"\),

op\("bsm,mF\-\>bsF",parallel="F",label="Gate␣&␣Up␣Projection"\),\|\\label\{line:fus2\}\|

op\("bsf,fm\-\>bsm",parallel="f",label="Down␣Projection"\)\|\\label\{line:down\}\|

\]

Figure 12:Full specification of a fused Llama3 implementation\.attn\_eqs=\[

op\("bKrsh,bKzh\-\>bKrsz",parallel="K",label="QK"\),\|\\label\{line:qk\}\|

op\("bKrsz,bKzh\-\>bKrsh",parallel="K",label="AV"\)\|\\label\{line:av\}\|

\]

Figure 13:Grouped query attention sub\-equations with attention\-head\-parallel used in[Figures˜2](https://arxiv.org/html/2605.14249#S3.F2),[12](https://arxiv.org/html/2605.14249#A2.F12)and[11](https://arxiv.org/html/2605.14249#A2.F11)\.Table 2:LLM symbols used in[Figures˜12](https://arxiv.org/html/2605.14249#A2.F12),[2](https://arxiv.org/html/2605.14249#S3.F2)and[11](https://arxiv.org/html/2605.14249#A2.F11)
## Appendix CCommunication Kernel Detection

Algorithm 1AllReduce Collective Kernel Detection1:functionNeedAllReduce\(op\)

2:

summation\_indices←ParseEinsum1​\(op\.eq\)\\textit\{summation\\\_indices\}\\leftarrow\\textsc\{ParseEinsum1\}\(\\text\{op\.eq\}\)
3:if

op\.parallel∈summation\_indices\\text\{op\.parallel\}\\in\\textit\{summation\\\_indices\}then

4:returntrue

5:else

6:returnfalse

7:endif

8:endfunction

9:

10:ifNeedAllReduce\(op\)then

11:

output←ParseEinsum2​\(op\.eq\)\\textit\{output\}\\leftarrow\\textsc\{ParseEinsum2\}\(\\text\{op\.eq\}\)
12:

size←GetMatrixSize​\(output,self\.params\)×t\\textit\{size\}\\leftarrow\\textsc\{GetMatrixSize\}\(\\textit\{output\},\\text\{self\.params\}\)\\times t
13:

all\_reduce\.Append​\(size\)\\textit\{all\\\_reduce\}\.\\textsc\{Append\}\(\\textit\{size\}\)
14:else

15:

all\_reduce\.Append​\(0\)\\textit\{all\\\_reduce\}\.\\textsc\{Append\}\(0\)
16:endif

Algorithm 2All\-to\-All Collective Kernel Detection1:functionNeedAll2All\(op, prev\_op\)

2:if

op\.CP\_dim≠prev\_op\.CP\_dim\\text\{op\.CP\\\_dim\}\\neq\\text\{prev\\\_op\.CP\\\_dim\}then

3:returntrue

4:else

5:returnfalse

6:endif

7:endfunction

8:

9:ifNeedAll2All\(

op,prev\_op\\textit\{op\},\\textit\{prev\\\_op\}\)then

10:

output←ParseEinsum2​\(prev\_op\.eq\)\\textit\{output\}\\leftarrow\\textsc\{ParseEinsum2\}\(\\text\{prev\\\_op\.eq\}\)
11:

size←GetMatrixSizePerRank​\(output,self\.params\)\\textit\{size\}\\leftarrow\\textsc\{GetMatrixSizePerRank\}\(\\textit\{output\},\\text\{self\.params\}\)
12:

kernels\.Append​\(transpose,size\)\\textit\{kernels\}\.\\textsc\{Append\}\(\\textit\{transpose\},\\textit\{size\}\)⊳\\trianglerightContinuous tensor layout required for NCCL

13:

kernels\.Append​\(all2all,size\)\\textit\{kernels\}\.\\textsc\{Append\}\(\\textit\{all2all\},\\textit\{size\}\)
14:

kernels\.Append​\(transpose,size\)\\textit\{kernels\}\.\\textsc\{Append\}\(\\textit\{transpose\},\\textit\{size\}\)
15:endif

[Algorithms˜2](https://arxiv.org/html/2605.14249#alg2)and[1](https://arxiv.org/html/2605.14249#alg1)show the non\-overlapped kernel detection mechanism\. Compute\-communication overlap modeling is detailed in[Section˜3\.4](https://arxiv.org/html/2605.14249#S3.SS4)\.

## Appendix DAdditional Plots for Communication

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/total_energy_overlay_all_reduce.png)\(a\)Energy
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/latency_overlay_all_reduce.png)\(b\)Latency

Figure 14:Total energy and latency ofAllReducefor 2, 4, and 8 GPUs\.Overhead dominates at small workload sizes, resulting in higher latency and energy per bit\. This behavior is missed by bandwidth\-and\-queuing\-based models used in prior works\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\)\.

## Appendix ESingle\-GPU Evaluation

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/single_gpu_etft_per_batch_1_4.png)Figure 15:The energy to first token \(ETFT\) prediction is validated on single\-GPU Llama3\-8B inferences with a MAPE of 11\.31%\. The B4 measurement curve ends at 2048 input tokens where TensorRT\-LLM switches to smaller batch sizes\.![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/single_gpu_decode_energy_per_batch_in512.png)Figure 16:The energy per request prediction is validated on single\-GPU Llama3\-8B inferences with a MAPE of 12\.87%\.We first validate EnergyLens in the single\-GPU setting with Llama3\-8B\. This evaluation tests whether our high\-level einsum\-based specification is sufficient to predict end\-to\-end inference energy from model structure alone, without requiring implementation\-level kernel details from the user\. It also provides a clean evaluation before introducing multi\-GPU communication effects\.

For prefill, all input tokens are processed simultaneously by the LLM\. We sweep[ISL](https://arxiv.org/html/2605.14249#id8.8.id8)linearly up to 8192 tokens and batch size logarithmically up to 64\. Because TensorRT\-LLM may change the realized batch size at long contexts, we use Torch traces to verify the actual executed batch size: for example, at 7936[ISL](https://arxiv.org/html/2605.14249#id8.8.id8), the engine always uses batch size 1\. We include all observed[ISL](https://arxiv.org/html/2605.14249#id8.8.id8)\-batch\-size pairs in the[MAPE](https://arxiv.org/html/2605.14249#id5.5.id5)calculation\.[Figure˜15](https://arxiv.org/html/2605.14249#A5.F15)plots the predicted[ETFT](https://arxiv.org/html/2605.14249#id2.2.id2), normalized per request\. Since these operations already have high arithmetic intensity,[ETFT](https://arxiv.org/html/2605.14249#id2.2.id2)is largely insensitive to batch size\. EnergyLens closely matches measurements, achieving a MAPE of 11\.31%\.

Decode behaves very differently from prefill\. Since only the most recent token is processed at each iteration \(s=1s=1\), decode kernels have lower arithmetic intensity and lower GPU utilization\. While prior work validates prefill latency or energy\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15),[2026](https://arxiv.org/html/2605.14249#bib.bib13)\), decode is often overlooked\. We therefore validate decode predictions in[Figure˜16](https://arxiv.org/html/2605.14249#A5.F16)\. Comparing the y\-axes of[Figures˜15](https://arxiv.org/html/2605.14249#A5.F15)and[16](https://arxiv.org/html/2605.14249#A5.F16), decoding 4000 output tokens consumes orders of magnitude more energy than prefilling 4000 input tokens\. Most importantly, EnergyLens correctly predicts the key trend relevant for optimization: at fixed[OSL](https://arxiv.org/html/2605.14249#id9.9.id9), energy per request decreases substantially with larger batch sizes, in contrast to prefill behavior\.

## Appendix FEvaluation for Llama3\-8B with[context parallelism](https://arxiv.org/html/2605.14249#id16.16.id16)\([CP](https://arxiv.org/html/2605.14249#id16.16.id16)\)

\\Acf

CP is increasingly used for long context prefill\([Jacobs et al\.,](https://arxiv.org/html/2605.14249#bib.bib10)\)\. EnergyLens inserts the appropriate communication kernels as described in[Algorithm˜2](https://arxiv.org/html/2605.14249#alg2)\. An example specification of the fused dense transformer with[CP](https://arxiv.org/html/2605.14249#id16.16.id16)is provided below\.

We validated EnergyLens’s support for context parallelism \(the variety proposed by DeepSpeed\-Ulysses\) on Llama3\-8B\. This is tested on CP2 with the same sweep settings described in[Appendix˜M](https://arxiv.org/html/2605.14249#A13), achieving MAPEs of 14\.69% and 12\.58% for energy and latency, respectively\. The communication and memory transpose kernel patterns are validated with Torch profiler\.

attn\_eqs=\[

op\("bKrsh,bKzh\-\>bKrsz",CP\_dim="K",label="QK"\),\|\\label\{line:qk\}\|

op\("bKrsz,bKzh\-\>bKrsh",CP\_dim="K",label="AV"\)\|\\label\{line:av\}\|

\]

llm\_model\.EQS=\[

op\("bsm,mi\-\>bsi",CP\_dim="s",label="QKV␣Projection"\),\|\\label\{line:fus1\}\|

op\("attention",CP\_dim="H",label="Attention"\),

op\("bsHh,Hhm\-\>bsm",CP\_dim="s",label="Output␣Projection"\),

op\("bsm,mF\-\>bsF",CP\_dim="s",label="Gate␣&␣Up␣Projection"\),\|\\label\{line:fus2\}\|

op\("bsf,fm\-\>bsm",CP\_dim="s",label="Down␣Projection"\)\|\\label\{line:down\}\|

\]

Figure 17:Full specification of a fused Llama3 implementation with context parallelism\.
## Appendix GAdditional Plots for Llama3\-70B with TP

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_energy_stacked_bar_tp4_no32_64.png)\(a\)Decode energy breakdown with the same legend as[Figure˜6\(a\)](https://arxiv.org/html/2605.14249#S4.F6.sf1)\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_latency_stacked_bar_tp4_no32_64.png)\(b\)Decode latency breakdown with the same legend as[Figure˜6\(a\)](https://arxiv.org/html/2605.14249#S4.F6.sf1)\.
\(c\)Memory constraint\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/CI_vs_batch_size_TP4.png)\(d\)Down projection\.
In all breakdown figures, the profiled latency breakdown is from the Torch traces of the first 10 decoding steps, instead of the full[OSL](https://arxiv.org/html/2605.14249#id9.9.id9)=1024\.

Figure 18:Energy and latency breakdown for Llama3\-70B decode phase at TP4\. \(a\) Energy breakdown shows the benefit of batching\. \(b\) Latency breakdown shows limitations of kernel\-level latency prediction for small decode operations\.In the decode phase, energy consumption varies more significantly with batch size than with tensor parallelism \([Figure˜8\(b\)](https://arxiv.org/html/2605.14249#S5.F8.sf2)\)\. However, energy savings trade off with memory requirements\. Increase in batch size means only shorter sequences can fit inside the GPU DRAM \([Figure˜18\(c\)](https://arxiv.org/html/2605.14249#A7.F18.sf3)\)\. During decode, operations exhibit skewed tensor shapes that result in low arithmetic intensity\. For example, EnergyLens parses the down projectionop\("bsf,fm\-\>bsm", parallel="f"\)into a standard matmul operation with output dimensionsM=bM=bandN=mN=m\(sinces=1s=1in decode\)\. WithMMequivalent to the batch size, the GPU’s[SMs](https://arxiv.org/html/2605.14249#id12.12.id12), the compute units, remain underutilized due to being memory\-bound \([Figure˜18\(d\)](https://arxiv.org/html/2605.14249#A7.F18.sf4)\)\. Scaling up batch size improves arithmetic intensity, making it more efficient\.

Across parallelism configurations \(TP2, TP4, and TP8\), batch sizes \(1 to 16\), and[OSL](https://arxiv.org/html/2605.14249#id9.9.id9)up to 7680 tokens, EnergyLens achieves a[MAPE](https://arxiv.org/html/2605.14249#id5.5.id5)of 12\.86% for decode energy\. Decode latency exhibits larger errors \(MAPE=25\.45%\)\. The latency breakdown in[Figure˜18\(b\)](https://arxiv.org/html/2605.14249#A7.F18.sf2)reveals limitations of kernel\-level latency prediction for low\-arithmetic\-intensity operations and skewed matrix shapes inherent to small decode operations\. We verified this limitation persists across alternative compute kernel backends\(Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15);Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16)\), confirming it is not backend\-specific\. Despite higher latency errors, EnergyLens correctly captures that decode latency remains relatively independent of batch size\.

## Appendix HAdditional Plots for Qwen3\-30B\-A3B with TP\+EP

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/kernel_energy_by_tp_batch2_in4096.png)\(a\)Prefill energy breakdown\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/sftp/kernel_latency_by_tp_batch2_in4096.png)\(b\)Prefill latency breakdown\.

Figure 19:Qwen3\-30B\-A3B prefill phase at B2, ISL=4096 across[TP](https://arxiv.org/html/2605.14249#id4.4.id4)in attention and[EP](https://arxiv.org/html/2605.14249#id15.15.id15)in MoE, demonstrating increasing communication overhead as TP degree increases\.Qwen3\-30B\-A3B is evaluated with TP \+ EP up to 4 GPUs, because there are only 4 key\-value heads in the attention layer\. The measured results are carefully calibrated to contain warmup iterations and measured at the steady state\. However, the profiled breakdown results are averaged across all the layers of 1 iteration for prefill and 10 iterations for decode\. Similarly to the dense model, we observe that increased TP degree leads to increased communication cost\.

## Appendix IEvaluation of Kernel Fusion

A key advantage of EnergyLens is that it enables rapid exploration of fusion strategies across the full range of serving scenarios without requiring implementation or profiling\. Instead, the developers can specify any range of[ISL](https://arxiv.org/html/2605.14249#id8.8.id8),[OSL](https://arxiv.org/html/2605.14249#id9.9.id9), batch sizes, and tensor parallelism configurations for evaluation\. The fused and unfused Llama3 specifications are in[Figure˜12](https://arxiv.org/html/2605.14249#A2.F12)and[Figure˜11](https://arxiv.org/html/2605.14249#A2.F11), with the fused QKV projections shown in[Figure˜20](https://arxiv.org/html/2605.14249#A9.F20)\.

We demonstrate this capability by evaluating the energy impact of layer fusion across example deployment settings in[Figure˜21](https://arxiv.org/html/2605.14249#A9.F21)\. The percentage energy saving in the prefill phase, averaged across batch sizes \(1 to 64\), ISL \(256 to 8192\), and TP \(2,4,8\) is 4\.37%\. The energy breakdown reveals that fusing Q, K, V projections yields substantial savings by reducing memory traffic\. This benefit is particularly pronounced in the decode phase where operations exhibit lower arithmetic intensity and are more sensitive to kernel overhead\. AtI​S​L=512ISL=512, we observe an average energy saving of 8\.18% for decode phase workloads\.

![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/qkv_fusion_precise.png)Figure 20:EnergyLens allows developers to rapidly assess unfused and fused implementations\. Example shown here for the query \(Q\), key \(K\), and value \(V\) projections\.![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_energy_comparison_fused_vs_unfused_by_tp_batch2_in4096.png)\(a\)Prefill phase \(B2, ISL=4096\)\.
![Refer to caption](https://arxiv.org/html/2605.14249v1/figs/kernel_energy_comparison_fused_vs_unfused_tp2.png)\(b\)Decode phase \(TP2, ISL=512, OSL=1024\), same legend as \(a\)\.

Figure 21:EnergyLens enables fast evaluation of kernel fusion\.
## Appendix JLimitations

Table 3:Decode latency MAPE across different kernel\-level modeling backends for Llama3\-70B\.In the decode phase of LLM inference,[GEMM](https://arxiv.org/html/2605.14249#id11.11.id11)kernels exhibit low arithmetic intensity and skewed matrix shapes that challenge existing kernel latency estimation tools\(Li et al\.,[2023](https://arxiv.org/html/2605.14249#bib.bib16);Lee et al\.,[2025b](https://arxiv.org/html/2605.14249#bib.bib15),[2026](https://arxiv.org/html/2605.14249#bib.bib13)\)\. To assess whether this limitation stems from our default backend, we leverage EnergyLens’s modular design to evaluate alternative backends\. As shown in Table[3](https://arxiv.org/html/2605.14249#A10.T3), NeuSight yields comparable decode latency errors, confirming that the challenge is inherent to small, low\-intensity operations rather than backend\-specific\. Custom kernel libraries \(e\.g\., TensorRT\-LLM\) may achieve superior performance compared to cuBLAS\-dispatched kernels, contributing to discrepancies observed in[Figure˜18\(b\)](https://arxiv.org/html/2605.14249#A7.F18.sf2)\. For communication kernels, we observe high consistency with measurements at large transfer sizes, though small discrepancies appear at smaller sizes\.

## Appendix KAccuracy

EnergyLens focuses on evaluating the energy efficiency of[LLM](https://arxiv.org/html/2605.14249#id3.3.id3)inference, achieving energy[MAPE](https://arxiv.org/html/2605.14249#id5.5.id5)s of 12\.18% and 12\.86% for prefill and decode phases respectively\. These prediction errors are substantially smaller than the energy variations across different deployment configurations observed in[Section˜5\.1](https://arxiv.org/html/2605.14249#S5.SS1), validating EnergyLens’s utility for energy\-driven optimization\.

Beyond energy prediction, EnergyLens supports latency estimation to enable energy\-latency trade\-off analysis\. While achieving high accuracy during prefill \(MAPE=12\.39%\), latency prediction exhibits larger errors during decode \(MAPE=25\.45%\)\. This limitation causes configurations B4 and B8 in[Figure˜8\(b\)](https://arxiv.org/html/2605.14249#S5.F8.sf2)to appear slightly slower than measured\. Despite this discrepancy, EnergyLens correctly captures the behavior that decode latency remains relatively independent of batch size\.

## Appendix LReproducibility

We reran the Llama3\-70B[TP](https://arxiv.org/html/2605.14249#id4.4.id4)experiments\. This occurred because our initial experiment data was accidentally wiped\. We report the old and new[MAPEs](https://arxiv.org/html/2605.14249#id5.5.id5)below\. All[MAPEs](https://arxiv.org/html/2605.14249#id5.5.id5)closely align with the original, showing that our experimental setup, including the profiling process, is reproducible\.

Table 4:Comparison of original and reproduced MAPE values for Llama3\-70B[TP](https://arxiv.org/html/2605.14249#id4.4.id4)experiments\.
## Appendix MExperimental Setup and Compute Resources

Here we provide details in addition to[Section˜4\.1](https://arxiv.org/html/2605.14249#S4.SS1)\. All models are evaluated in bf16 precision\. For Llama3 models, we use TensorRT\-LLM v0\.14\.[Figure˜12](https://arxiv.org/html/2605.14249#A2.F12)represents it in the EnergyLens interface\. Qwen3\-30B\-A3B experiments were done with TensorRT\-LLM v1\.0 and[Figure˜2](https://arxiv.org/html/2605.14249#S3.F2)describes the fusion and parallel strategy\. This combination of[TP](https://arxiv.org/html/2605.14249#id4.4.id4)and[EP](https://arxiv.org/html/2605.14249#id15.15.id15)is termed astepin TensorRT\-LLM documentation\. The workload used in profiling is from Hugging Face and can be obtained withload\_dataset\("wikipedia", "20220301\.en", split="train"\)\.

For all prefill MAPE results, the evaluation swept input sequence length linearly spaced with interval of 256 up to 8192 and batch size exponentially spaced up to 64 with a factor of 2\. For all decode MAPE results, the evaluation used ISL 512, swept output sequence length linear spaced up to OSL 7680 or until out\-of\-memory if earlier, and swept batch size up to 16\. Actual runtime batch sizes used by TensorRT\-LLM at long contexts were verified with Torch Profiler, and all observed batch\-size/sequence\-length pairs were included in the MAPE calculation\.

The Llama3\-70B overlap MAPE results were obtained with Megatron\-style compute\-communication overlap in the prefill phase\. Overlap configurations including no overlap and 4\-stage overlap with 1, 4, or 16 SMs dedicated to communication\.

All Llama3 TP experiments include TP2, TP4, TP8\. All Qwen experiments include TP\+EP2 and TP\+EP4 \(limited by the number of key\-value heads\)\.

The main cost of the experiments comes from validating the accuracy by actually running the workloads\. We provide a table of the GPU\-hour cost below\. We also estimate 200 GPU\-hour of auxiliary experiments, mainly collecting execution traces to validate our model\.

Table 5:Aggregate GPU\-hours by experiment group\.### M\.1Licenses

We use the following existing assets in our experiments: TensorRT\-LLM v0\.14 and v1\.0 \(https://github\.com/NVIDIA/TensorRT\-LLM, license: Apache 2\.0\); Megatron\-LM v0\.15\.3 \(https://github\.com/NVIDIA/Megatron\-LM, license: BSD 3\-Clause\); PyTorch / Torch Profiler \(https://github\.com/pytorch/pytorch, license: BSD\-style\); NCCL microbenchmarks \(https://github\.com/NVIDIA/nccl\-tests, license: BSD 3\-Clause "New" or "Revised" License\); Llama 3 models \(https://github\.com/meta\-llama/llama3, license/terms: Meta Llama 3 Community License Agreement\); Qwen3\-30B\-A3B \(https://huggingface\.co/Qwen/Qwen3\-30B\-A3B, license/terms: Apache 2\.0\); EnergAIzer \(https://zenodo\.org/records/18916559, MIT license\)\. We cite the corresponding papers and repositories and use these assets in accordance with their stated terms\.

## Appendix NAbbreviations

TTFTtime to first tokenETFTenergy to first tokenLLMlarge language modelTPtensor parallelismMAPEmean absolute percentage errorEPOTenergy per output tokenTPOTtime per output tokenISLinput sequence lengthOSLoutput sequence lengthFFNfeed\-forward networkGEMMgeneral matrix multiplicationSMstreaming multiprocessorTDPthermal design powerNVMLNVIDIA Management LibraryEPexpert parallelismCPcontext parallelismMoEmixture of experts