[2511.07885] Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

# Intelligence per Watt:Measuring Intelligence Efficiency of Local AI
Source: [https://arxiv.org/html/2511.07885](https://arxiv.org/html/2511.07885)
Avanika NarayanHakki Orhun AkenginJ\. Wes GriffinHerumb ShandilyaAdrian Gamarra LafuenteMedhya GoelRebecca JosephShlok NatarajanEtash Kumar GuhaShang ZhuBen AthiwaratkunJohn HennessyAzalia MirhoseiniChristopher Ré

## 1Introduction

Large language model \(LLM\) queries are predominantly processed by frontier models deployed in centralized cloud infrastructure\[[55](https://arxiv.org/html/2511.07885#bib.bib55),[2](https://arxiv.org/html/2511.07885#bib.bib2)\]\. This centralized approach faces mounting resource constraints as inference workloads scale from billions to trillions of queries daily\[[2](https://arxiv.org/html/2511.07885#bib.bib2)\]\. History suggests an alternative path forward\. From 1946\-2009, computing efficiency \(performance\-per\-watt\) doubled every 1\.5 years\[[37](https://arxiv.org/html/2511.07885#bib.bib37)\], enabling a redistribution of computing workloads from data center mainframes to personal computers\. This transition occurred when efficiency improvements enabled computing to meet user needs within personal device power constraints, not when PCs surpassed mainframes in raw performance\.

Three converging trends suggest a similar inflection point may be emerging for LLM inference\. First, recent advances have producedlocal LMs: small models \(≤20​B\\leq 20Bactive parameters\) such asQwen3\[[62](https://arxiv.org/html/2511.07885#bib.bib62)\],Llama3\.1\[[26](https://arxiv.org/html/2511.07885#bib.bib26)\], andgpt\-oss\[[1](https://arxiv.org/html/2511.07885#bib.bib1)\]that achieve competitive performance on many benchmarks while requiring less energy and compute than larger, frontier models\[[1](https://arxiv.org/html/2511.07885#bib.bib1)\]\. Second, local accelerators \(e\.g\., Apple M4 Max, AMD Ryzen AI\) now have sufficient memory capacity and compute throughput to host these models with interactive latencies\[[8](https://arxiv.org/html/2511.07885#bib.bib8)\]\. Third, a wave of open\-source personal\-AI agent stacks designed for on\-device execution \(e\.g\.,OpenClaw\[[74](https://arxiv.org/html/2511.07885#bib.bib74)\],Hermes Agent\[[44](https://arxiv.org/html/2511.07885#bib.bib44)\],OpenJarvis\[[63](https://arxiv.org/html/2511.07885#bib.bib63)\]\),PicoClaw\[[71](https://arxiv.org/html/2511.07885#bib.bib71)\], andZeroClaw\[[88](https://arxiv.org/html/2511.07885#bib.bib88)\]has emerged, reflecting growing interest in local\-first system design\. This raises the question:Can local inference viably redistribute demand from centralized infrastructure?

![Refer to caption](https://arxiv.org/html/2511.07885v5/figures/main_figure.png)Figure 1:Intelligence per Watt: A Study of Local Intelligence Efficiency\.We present thefirst systematic study of local AI inference efficiencyacross models, hardware, and real\-world workloads\.\(Left\)Intelligence efficiency is defined as task accuracy per unit of power, capturing both capabilities delivered and energy consumed\.\(Left\-Middle\)We conductcomprehensive performance profilingacross 20\+ state\-of\-the\-art local LMs \(≤20​B\\leq 20Bactive parameters\), diverse hardware accelerators \(Apple,NVIDIA,AMD\), multiple performance metrics, and 1M\+ real\-world queries spanning chat and reasoning tasks\.\(Right\-Middle\)Local LM capabilities are improving rapidly: win/tie rate versus frontier models increases from23\.2%23\.2\\%\(2023\) to71\.3%71\.3\\%\(2025\), a3\.1×3\.1\\timesimprovement in accuracy, demonstrating that local models can accurately handle significant portions of single\-turn chat and reasoning queries\.\(Right\)Intelligence per watt improves5\.3×5\.3\\timesfrom 2023–2025, driven by advances in both model architectures and hardware accelerators, with local accelerators showing1\.5×1\.5\\timesefficiency headroom compared to enterprise\-grade systems\.Answering this requires measuring two factors: thecapabilityof local LMs to accurately respond to a subset of real\-world queries, and theefficiencywith which local accelerators convert power into useful computation\. To assess this, we need a unified metric that captures both the intelligence delivered \(model capability\) and the energy required \(accelerator efficiency\)\. We introduceintelligence per watt\(IPW\): task accuracy per unit of power consumption\.IPWdirectly measures the fundamental tradeoff facing local inference: achieving sufficient task performance within constrained power budgets\. This metric enables systematic comparison across model\-accelerator configurations and quantifies efficiency gains from model architecture innovations\[[62](https://arxiv.org/html/2511.07885#bib.bib62),[1](https://arxiv.org/html/2511.07885#bib.bib1),[24](https://arxiv.org/html/2511.07885#bib.bib24),[35](https://arxiv.org/html/2511.07885#bib.bib35)\], post\-training techniques\[[30](https://arxiv.org/html/2511.07885#bib.bib30),[57](https://arxiv.org/html/2511.07885#bib.bib57),[68](https://arxiv.org/html/2511.07885#bib.bib68),[22](https://arxiv.org/html/2511.07885#bib.bib22)\], and accelerator improvements\[[39](https://arxiv.org/html/2511.07885#bib.bib39),[51](https://arxiv.org/html/2511.07885#bib.bib51),[4](https://arxiv.org/html/2511.07885#bib.bib4)\]\.

To evaluate the viability of local inference and measure progress inIPW, we conduct a large\-scale empirical study addressing three questions:

- •Q1:What fraction of current inference queries can be solved by local LMs on local accelerators, and how has this changed over time?
- •Q2:How has intelligence per watt improved across successive generations of local models and accelerators, and what are the relative contributions of model versus accelerator advances?
- •Q3:What resource savings \(e\.g\. compute, energy, dollar cost\) are possible by distributing workloads across local and cloud infrastructure?

Our study evaluates 20\+ local LMs across 8 hardware accelerators on 1M queries spanning naturalistic user conversations\[[20](https://arxiv.org/html/2511.07885#bib.bib20)\], general reasoning tasks\[[87](https://arxiv.org/html/2511.07885#bib.bib87)\], and standardized benchmarks measuring knowledge breadth \(MMLU Pro\[[80](https://arxiv.org/html/2511.07885#bib.bib80)\]\) and expert\-level reasoning \(SuperGPQA\[[59](https://arxiv.org/html/2511.07885#bib.bib59)\]\)\. We focus on single\-turn interactions because they constitute a substantial portion of LLM usage\[[21](https://arxiv.org/html/2511.07885#bib.bib21),[79](https://arxiv.org/html/2511.07885#bib.bib79),[70](https://arxiv.org/html/2511.07885#bib.bib70)\]\.111We report a multi\-turn extension on GAIA and TerminalBenchV2 in App\.[E\.12](https://arxiv.org/html/2511.07885#A5.SS12)confirming the qualitative patterns generalize\.We compare state\-of\-the\-art local LMs from October 2025 \(Qwen3,gpt\-oss,Gemma3,IBM Granite4\) alongside 2023\-2024 models \(Mixtral\-8x7B,Llama\-3\.1\-8B\) onNVIDIA,AMD, andAppleaccelerators, measuring accuracy, latency, energy, compute, cost, and memory per query \(Section[3](https://arxiv.org/html/2511.07885#S3)\)\. We release our hardware\-agnostic profiling harness to support reproducible efficiency benchmarking\.

Our work makes three primary contributions\.\(1\)We introduce intelligence per watt as a unified metric for evaluating local inference viability, and conduct the first large\-scale empirical study measuring its evolution across 1M\+ queries, 20\+ models, and 8 hardware accelerators spanning 2023\-2025\.\(2\)\(Q1, Q2\)We demonstrate that88\.7%88\.7\\%of single\-turn chat and reasoning queries can be successfully handled by small local models \(with coverage varying by domain\), and that IPW has improved5\.3×5\.3\\timesover two years through compounding model \(3\.1×3\.1\\times\) and hardware \(1\.7×1\.7\\times\) advances\.222The corresponding per\-joule decomposition \(Figure[3](https://arxiv.org/html/2511.07885#S4.F3)\) yields18\.0×18\.0\\timesoverall,3\.1×3\.1\\timesmodel and5\.9×5\.9\\timeshardware, since hardware progress disproportionately reduces latency\. BothMixtral\-8x7B\(47B total,∼12\.9\{\\sim\}12\.9B active per token\) andgpt\-oss\-120b\(120B total,≤20\\leq 20B active per token\) are mixture\-of\-experts models; our≤20\\leq 20B threshold refers toactiveparameters per forward pass, which determine per\-query power and latency\. Total parameter count governs storage: at FP4,Mixtral\-8x7Bfits within 24 GB GDDR6 \(Quadro RTX 6000\), andgpt\-oss\-120bfits within 128 GB unified memory \(Apple M4 Max\)\.\(3\)\(Q3\)We show that hybrid local\-cloud routing yields6060–80%80\\%reductions in energy, compute, and cost compared to a batched cloud baseline; even an80%80\\%\-accurate router \(a realistic target\) captures∼\\sim80%80\\%of oracle gains while maintaining answer quality\. Together, these findings establish local inference as a practical complement to centralized infrastructure whose viability continues expanding\.

## 2Preliminaries

We formalize local and cloud inference infrastructure and introduce metrics for measuring intelligence efficiency\.

Inference Infrastructure: Queries, Models and Accelerators\.We consider an inference infrastructure serving a stream of user queries𝒬=\{q1,q2,…,qn\}\\mathcal\{Q\}=\\\{q\_\{1\},q\_\{2\},\\ldots,q\_\{n\}\\\}, where each queryqiq\_\{i\}represents a user\-generated request \(e\.g\., chat messages, reasoning tasks\)\. Letℳlocal=\{m1,…,mk\}\\mathcal\{M\}\_\{\\text\{local\}\}=\\\{m\_\{1\},\\ldots,m\_\{k\}\\\}denote a set oflocal LMswith≤20​B\\leq 20Bactive parameters each, andℳcloud=\{M1,…,Mℓ\}\\mathcal\{M\}\_\{\\text\{cloud\}\}=\\\{M\_\{1\},\\ldots,M\_\{\\ell\}\\\}denotefrontier LMswith≥100​B\\geq 100Bparameters\. Similarly, letℋlocal\\mathcal\{H\}\_\{\\text\{local\}\}representlocal accelerators\(e\.g\.,Apple M4,AMD Ryzen\) andℋcloud\\mathcal\{H\}\_\{\\text\{cloud\}\}representcloud accelerators\(e\.g\.,NVIDIA H200,AMD MI300X\)\. We defineℳlocal\\mathcal\{M\}\_\{\\text\{local\}\}by active parameters rather than total parameters because per\-query inference efficiency depends on parameters touched per forward pass; total parameters instead govern storage, a separate constraint we verify per accelerator\.

Inference Serving: Local and Cloud\.We distinguish between two inference paradigms:local inference, where queries are processed by modelsm∈ℳlocalm\\in\\mathcal\{M\}\_\{\\text\{local\}\}on acceleratorh∈ℋlocalh\\in\\mathcal\{H\}\_\{\\text\{local\}\}, andcloud inference, where queries are processed by modelsM∈ℳcloudM\\in\\mathcal\{M\}\_\{\\text\{cloud\}\}on acceleratorH∈ℋcloudH\\in\\mathcal\{H\}\_\{\\text\{cloud\}\}\. Arouting functionr:𝒬→ℳlocal∪ℳcloudr:\\mathcal\{Q\}\\rightarrow\\mathcal\{M\}\_\{\\text\{local\}\}\\cup\\mathcal\{M\}\_\{\\text\{cloud\}\}assigns each query to either a local model \(up to 20B active parameters\) or a cloud model \(at least 100B parameters\)\.

Intelligence Efficiency Metrics\.We introduce a family of metrics to quantify how efficiently inference systems convert energy into useful computation\. For a model\-accelerator pair\(m,h\)\(m,h\), letacc​\(m,q\)\\text\{acc\}\(m,q\)denote the accuracy of modelmmon queryqq,ppl​\(m,q\)\\text\{ppl\}\(m,q\)denote the perplexity,P⁡\(m,h,q\)P\(m,h,q\)denote the average power consumption \(in watts\) during inference for queryqq, andτ⁡\(m,h,q\)\\tau\(m,h,q\)denote the total latency \(in seconds\) for generating the response, including both prefill and decoding phases\.

We define four complementary efficiency metrics:

Power\-based metricsmeasure efficiency relative to instantaneous power draw:

- •Accuracy per watt:APW​\(m,h\)=𝔼q∼𝒬​\[acc​\(m,q\)\]𝔼q∼𝒬​\[P​\(m,h,q\)\]\\text\{APW\}\(m,h\)=\\frac\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[\\text\{acc\}\(m,q\)\]\}\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[P\(m,h,q\)\]\}
- •Perplexity per watt: PPW​\(m,h\)=1𝔼q∼𝒬​\[ppl​\(m,q\)\]⋅𝔼q∼𝒬​\[P⁡\(m,h,q\)\]\\text\{PPW\}\(m,h\)=\\frac\{1\}\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[\\text\{ppl\}\(m,q\)\]\\cdot\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[P\(m,h,q\)\]\}

Energy\-based metricsmeasure efficiency relative to total energy consumed per query:

- •Accuracy per joule: APJ​\(m,h\)=𝔼q∼𝒬​\[acc​\(m,q\)\]𝔼q∼𝒬​\[P⁡\(m,h,q\)⋅τ⁡\(m,h,q\)\]\\text\{APJ\}\(m,h\)=\\frac\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[\\text\{acc\}\(m,q\)\]\}\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[P\(m,h,q\)\\cdot\\tau\(m,h,q\)\]\}
- •Perplexity per joule: PPJ​\(m,h\)=1𝔼q∼𝒬​\[ppl​\(m,q\)\]⋅𝔼q∼𝒬​\[P⁡\(m,h,q\)⋅τ⁡\(m,h,q\)\]\\text\{PPJ\}\(m,h\)=\\frac\{1\}\{\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[\\text\{ppl\}\(m,q\)\]\\cdot\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\[P\(m,h,q\)\\cdot\\tau\(m,h,q\)\]\}

whereP⁡\(m,h,q\)⋅τ⁡\(m,h,q\)P\(m,h,q\)\\cdot\\tau\(m,h,q\)represents the energy consumption \(in joules\) for processing queryqq\.

Power\-based metrics \(APW,PPW\) capture the instantaneous efficiency of the inference system, reflecting the hardware’s ability to deliver performance at a given power draw\. Energy\-based metrics \(APJ,PPJ\) capture the total efficiency per query, accounting for both power consumption and generation latency\. Together, these metrics provide a comprehensive view of inference efficiency:intelligence per wattquantifies the steady\-state efficiency of model\-accelerator pairs, whileintelligence per joulequantifies the end\-to\-end efficiency from a user’s perspective, including the time cost of generation\. We report results across all four metrics throughout the paper \([Table 2](https://arxiv.org/html/2511.07885#S4.T2), Figures[3](https://arxiv.org/html/2511.07885#S4.F3)and[8](https://arxiv.org/html/2511.07885#A5.F8), Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14)\) so that conclusions are robust to the choice of formulation; relative rankings and qualitative trends are preserved across IPW, IPJ, PPW, and PPJ, while absolute rates of change differ in informative ways \(e\.g\., per\-joule gains exceed per\-watt gains because hardware progress reduces both power draw and generation latency\)\. Throughout the main text we useaccuracyas a binary indicatoracc​\(m,q\)∈\{0,1\}\\text\{acc\}\(m,q\)\\in\\\{0,1\\\}: for benchmarks with ground\-truth answers \(MMLU Pro,SuperGPQA,NaturalReasoning\) we use exact\-match correctness, while for open\-ended chat queries \(Wildchat\) we follow established practice in chat evaluation\[[14](https://arxiv.org/html/2511.07885#bib.bib14)\]and defineacc​\(m,q\)=1\\text\{acc\}\(m,q\)=1whenever the LLM\-judge verdict is\[\[A\>B\]\],\[\[A\>\>B\]\], or\[\[A=B\]\]\(i\.e\., the local model wins or ties against the frontier reference\)\.

## 3Dataset and Profiling Harness

In this section, we provide details on the dataset selection and profiling harness\.

### 3\.1Dataset Selection

Query CurationWe curate over1​M1Mqueries across four complementary benchmarks designed to measure both naturalistic deployment scenarios and controlled capability assessment\. To ensure our findings about local inference efficiency generalize across task distributions, we combine naturalistic queries that reflect real\-world LLM usage patterns with standardized benchmarks that enable systematic evaluation of knowledge breadth and reasoning capabilities across diverse domains\.

Fornaturalistic chat tasks, we source queries fromWildchat\[[20](https://arxiv.org/html/2511.07885#bib.bib20)\]: a dataset of1​M1Mreal ChatGPT prompts, spanning 1 month of user traffic\. Forgeneral reasoning tasks, we source queries fromNaturalReasoning\[[87](https://arxiv.org/html/2511.07885#bib.bib87)\], which provides approximately 1\.2 million reasoning\-focused queries spanning diverse domains including mathematics, physics, and chemistry\. Forstandardized knowledge evaluation, we useMMLU Pro\[[80](https://arxiv.org/html/2511.07885#bib.bib80)\]: an enhanced version of MMLU with increased difficulty \(10 vs\. 4 answer choices\) and improved robustness to prompt variations, measuring multi\-domain knowledge understanding\. Forexpert\-level reasoning across specialized disciplines, we evaluate onSuperGPQA\[[59](https://arxiv.org/html/2511.07885#bib.bib59)\]: a comprehensive benchmark spanning 285 graduate\-level disciplines with emphasis on technical domains and specialized fields underrepresented in typical evaluations \(e\.g\., light industry, agriculture, service sciences\)\.

We perform robust data cleaning and filtering \(see App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px5)\) on each dataset before sampling queries:500​K500KfromWildchat,500​K500KfromNaturalReasoning,12​K12KfromMMLU Pro, and26\.5​K26\.5KfromSuperGPQA\(see Table[1](https://arxiv.org/html/2511.07885#S3.T1)\)\. Furthermore, we usegpt\-4o\-minito annotate each query with a category from the Anthropic Economic Index\[[27](https://arxiv.org/html/2511.07885#bib.bib27)\], which maps AI queries to occupations in the U\.S\. Department of Labor’s O\*NET\. We consider 22 categories, spanning “Architecture and Engineering” to “Healthcare Support” \(full list and category breakdown in App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px5), Table[3](https://arxiv.org/html/2511.07885#A2.T3)\)\.

Dataset OriginCategory\|N\|\|N\|WildchatChat500KNaturalReasoningReasoning500KMMLU ProKnowledge12KSuperGPQAGrad\. Reasoning26\.5K
CategoryItemsModel FamiliesQwen3,gpt\-oss,Gemma,IBM Granite 4\.0AcceleratorsNVIDIA A100,H200,GH200,B200,Quadro RTX 6000,RTX 6000 Ada,AMD MI300X,Apple M4 Max,Sambanova SN40L

Table 1:Dataset Overview\.\(Left\)Query composition with sizes\.\(Right\)Models and Accelerators\.Hardware AcceleratorsWe profile diverse accelerators spanning local, workstation, and datacenter tiers: theNVIDIA A100 40 GB SXM4 \(Ampere\)\[[45](https://arxiv.org/html/2511.07885#bib.bib45)\],NVIDIA H200 SXM \(Hopper\)\[[46](https://arxiv.org/html/2511.07885#bib.bib46)\],NVIDIA GH200 Grace Hopper Superchip\[[49](https://arxiv.org/html/2511.07885#bib.bib49)\],NVIDIA B200 \(Blackwell\)\[[50](https://arxiv.org/html/2511.07885#bib.bib50)\],NVIDIA Quadro RTX 6000\[[47](https://arxiv.org/html/2511.07885#bib.bib47)\], NVIDIA RTX 6000 Ada\[[48](https://arxiv.org/html/2511.07885#bib.bib48)\],AMD Instinct MI300X \(CDNA 3, OAM\)\[[3](https://arxiv.org/html/2511.07885#bib.bib3)\], SambaNova SN40L\[[64](https://arxiv.org/html/2511.07885#bib.bib64)\]andApple Mac Studio \(M4 Max\)\[[8](https://arxiv.org/html/2511.07885#bib.bib8)\]\. We additionally evaluate a smartphone\-class accelerator \(Apple A18 Proon iPhone 16 Pro\) in App\.[E\.11](https://arxiv.org/html/2511.07885#A5.SS11)\. These systems were chosen because of their different memory capacities \(ranging from 40 GB to 768 GB\), memory bandwidth \(from 546 GB/s to 8 TB/s\), and power consumption \(145W to 1000W\) \(see Table[7](https://arxiv.org/html/2511.07885#A2.T7)for more details\)\.

ModelsWe collect model generations over theQwen3\[[62](https://arxiv.org/html/2511.07885#bib.bib62)\],gpt\-oss\[[1](https://arxiv.org/html/2511.07885#bib.bib1)\],Gemma3\[[24](https://arxiv.org/html/2511.07885#bib.bib24)\], andIBM Granite 4\.0\[[35](https://arxiv.org/html/2511.07885#bib.bib35)\]families\. ForQwen3, we useQwen3\-4B,Qwen3\-8B,Qwen3\-14B,Qwen3\-32B, andQwen3\-235B\. ForGPT\-OSS, we consider thegpt\-oss\-20bandgpt\-oss\-120bmodels\. For theGemma3family, we useGemma3 1B Instruct,Gemma3 4B Instruct, andGemma3 12B Instructmodels\. ForIBM Granite 4\.0, we usegranite\-4\.0\-h\-micro,granite\-4\.0\-h\-tiny, andgranite\-4\.0\-h\-smallmodels\. We evaluate state\-of\-the\-art cloud models as of October 2025, includingClaude Sonnet 4\.5\[[6](https://arxiv.org/html/2511.07885#bib.bib6)\],Gemini 2\.5 Pro\[[17](https://arxiv.org/html/2511.07885#bib.bib17)\], andGPT\-5\(2025\-08\-07\)\[[54](https://arxiv.org/html/2511.07885#bib.bib54)\]\. For our longitudinal analysis, we evaluateMixtral\-8x7B\[[36](https://arxiv.org/html/2511.07885#bib.bib36)\]andLlama3\.1\-8B\[[36](https://arxiv.org/html/2511.07885#bib.bib36)\]\. For each model, we generate responses across all dataset queries on each of the hardware backends\. Full details of inference hyperparameters can be found in App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px5)\.

MetricsFor each\(query, model, hardware\)triple we collect accuracy plus efficiency metrics: latency, throughput, time\-to\-first\-token \(TTFT\), and more \(see Table[6](https://arxiv.org/html/2511.07885#A2.T6)\) We use LLM\-as\-a\-judge \(prompts in App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px3)\) to score generated responses against reference answers\. ForWildchat, reference answers are responses fromQwen3\-235B, the SOTA open\-source model on LMArena \(as of August 2025\)\[[14](https://arxiv.org/html/2511.07885#bib.bib14)\]\. ForNaturalReasoning,MMLU Pro, andSuperGPQA, we use the provided ground truth answers from each benchmark\.

### 3\.2Profiling Harness

We develop an end\-to\-end, cross\-platform profiling harness for inference workloads that ensures reproducible results and easily accommodates new models, tasks, and hardware backends\. It comprises three components \(distributed multi\-GPU inference, response evaluation, and system\-level telemetry collection\) and currently supportsNVIDIA,macOS\(Apple Silicon\), andAMDsystems\. Given a dataset, model, and backend, the harness orchestrates inference over all input queries, evaluates outputs \(via exact match or LLM\-as\-a\-judge\), and records detailed telemetry: latency, throughput, time\-to\-first\-token \(TTFT\), energy consumption, and more \(Table[6](https://arxiv.org/html/2511.07885#A2.T6)\)\. Telemetry is collected via vendor APIs, synchronized at nanosecond resolution, and normalized to common units \(watts, joules, megabytes\)\. For energy measurements, we follow standard practices\[[65](https://arxiv.org/html/2511.07885#bib.bib65),[23](https://arxiv.org/html/2511.07885#bib.bib23),[82](https://arxiv.org/html/2511.07885#bib.bib82)\]\. OnNVIDIAsystems we query NVML for per\-device power, energy, memory usage, and temperature \(accelerator\-only scope\); onAMDsystems we query ROCm SMI for power, temperature, and VRAM usage \(accelerator\-only scope\); onmacOSsystems we extract GPU power frompowermetrics\(processor\_power\.actualon Apple Silicon, isolating the GPU subsystem rather than full SoC package power\) so that all per\-query power measurements correspond to the AI\-accelerator subsystem on each platform\. In all cases, we compute energy via numerical integration over time and sample at 50 ms intervals, providing higher temporal resolution than prior work \(100 ms\[[65](https://arxiv.org/html/2511.07885#bib.bib65)\]or 15 s\[[23](https://arxiv.org/html/2511.07885#bib.bib23)\]\)\. For multi\-GPU configurations, we aggregate energy from each GPU individually rather than extrapolating from a single device\[[65](https://arxiv.org/html/2511.07885#bib.bib65)\]\. We use a custom harness rather thanCodeCarbon\[[16](https://arxiv.org/html/2511.07885#bib.bib16)\], which similarly relies on NVML and RAPL, primarily for two reasons: \(i\)CodeCarbon’s default sampling cadence \(15 s\) is too coarse for fine\-grained per\-query attribution on short generations, and \(ii\) our cross\-platform requirements include AMD \(ROCm SMI\) and Apple Silicon \(powermetrics\) telemetry thatCodeCarbondoes not natively support\. Software\-based power measurements can introduce inaccuracies of 10–15%, with variations distributed across different hardware components due to architectural differences in workloads between CPUs, GPUs, and NPUs\[[85](https://arxiv.org/html/2511.07885#bib.bib85)\]\. Even hardware wattage meters may fall short for milliwatt\-level precision, though our approach aligns with established practices and provides consistent relative comparisons across configurations\. Full implementation details are provided in App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px6)\.

## 4Intelligence Efficiency Study

Figure 2:Local Models Rival Cloud Models Across Diverse Benchmarks:Individual model performance scales with size, ranging from 31\.5–69\.4% forIBM Granite4\-H\-Small, 30\.0–83\.6% forGemma3\-12B, 51\.5–80\.4% forGPT\-OSS\-120B, and 66\.5–89\.5% forGemini 2\.5 Pro\. Local routing \(best local LM per query\) achieves 97\.8%, 88\.3%, 77\.0%, and 92\.4% onWildchat,NaturalReasoning,SuperGPQA, andMMLU Prorespectively, surpassing cloud routing \(100%, 82\.9%, 66\.5%, 87\.4%\) on three of four benchmarks\.We investigate whether recent advances in local LMs and local accelerators enable local inference to viably complement centralized cloud infrastructure by handling a substantial fraction of inference queries\. Using our curated dataset, we examine three interconnected questions:\(1\)the extent to which current workloads can be handled locally \(Section[4\.1](https://arxiv.org/html/2511.07885#S4.SS1)\),\(2\)how intelligence efficiency has evolved from 2023–2025 \(Section[4\.2](https://arxiv.org/html/2511.07885#S4.SS2)\), and\(3\)what gains query routing across local and cloud models can deliver in practice \(Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3)\)\. We use single\-query inference \(bs=1\\text\{bs\}=1\) to isolate intrinsic model\-accelerator efficiency from system\-level scheduling and follow standard local\-inference benchmarking practice\[[28](https://arxiv.org/html/2511.07885#bib.bib28)\]; the §4\.3 routing simulation is the exception, running the cloud baseline at bs=16 to reflect production serving\.

### 4\.1Can Local Models and Accelerators Handle Current Inference Workloads?\(Q1\)

We measurequery coverage\(the percentage of dataset queries answered correctly\) across three configurations \(Figure[2](https://arxiv.org/html/2511.07885#S4.F2)\): individual local LMs, the best\-of\-local ensemble \(routing to the best local LM\), and the best\-of\-cloud baseline \(routing to the best frontier model\)\. Our findings are as follows:

Local LM coverage increases with scale and time\.AcrossWildchat,NaturalReasoning,SuperGPQA, andMMLU Pro, individual model coverage ranges from49\.6%49\.6\\%forQwen3\-4B, on average, to71\.4%71\.4\\%forgpt\-oss\-120b, with consistent improvements at each scale point:Qwen3\-8Bachieves57\.5%57\.5\\%andQwen3\-14Breaches60\.0%60\.0\\%\. Coverage has improved substantially from 2023 to 2025 \(Figure[12](https://arxiv.org/html/2511.07885#A5.F12)\): the best local LMs achieved a32\.2%32\.2\\%relative improvement on chat queries and a50\.1%50\.1\\%relative improvement on reasoning queries over this period\. While improvements from 2023\-2025 are relatively uniform across difficulty levels for chat tasks, reasoning tasks show markedly slower progress on the hardest problems \(see App[E](https://arxiv.org/html/2511.07885#A5)\)\. These results demonstrate that larger local LMs can handle progressively more queries without requiring cloud infrastructure, with the best individual local LM \(gpt\-oss\-120b\) successfully answering almost three\-fourths of the single\-turn chat and reasoning queries studied\.

Model diversity substantially improves coverage\.Routing queries to the most appropriate local LM rather than using a single model achieves88\.7%88\.7\\%overall coverage, a28\.828\.8percentage point improvement overQwen3\-14Band16\.316\.3percentage points over individualgpt\-oss\-120bperformance, on average\. This gap between individual models and best\-of\-local demonstrates that architectural, pretraining, and post\-training diversity captures complementary capabilities: different models excel on different query types, and intelligent routing can exploit these complementary strengths\. On reasoning benchmarks, best\-of\-local even surpasses best\-of\-cloud \([Figure 2](https://arxiv.org/html/2511.07885#S4.F2)\); this is a best\-of\-NNselection effect, since best\-of\-local selects from20\+20\+diverse local models while best\-of\-cloud selects from three frontier models, and a sufficiently diverse local ensemble can therefore exceed any single frontier model on subsets of queries where local strengths are complementary\.

Chat queries are more amenable to local processing than reasoning queries\.The best local LM achieves88\.9%88\.9\\%coverage onWildchatversus64\.9%64\.9\\%onNaturalReasoning, a24\.024\.0pp gap consistent with findings that77%77\\%of real\-world ChatGPT queries involve practical guidance, information seeking, or writing\[[10](https://arxiv.org/html/2511.07885#bib.bib10)\]\. These tasks are well\-suited to local models, while reasoning\-intensive queries more often require frontier capabilities for technical domains \(Architecture, Engineering, Life & Physical Science; see Figure[5](https://arxiv.org/html/2511.07885#A2.F5)in App\.[B](https://arxiv.org/html/2511.07885#A2)\)\. Even onNaturalReasoning/SuperGPQA/MMLU Pro, local LMs handle over four\-fifths of reasoning queries studied, suggesting significant opportunities for local inference even in technically demanding domains\.

Evaluation on standardized benchmarks confirms local LM viability across task distributions\.OnMMLU Pro\(multi\-domain knowledge\) andSuperGPQA\(graduate\-level reasoning\), best\-of\-local achieves93\.4%93\.4\\%and83\.6%83\.6\\%coverage respectively \(vs\.80\.4%80\.4\\%and51\.5%51\.5\\%for the best individual local model\), with coverage exceeding93%93\\%for creative/humanities fields but dropping to60%60\\%for technical disciplines like Architecture & Engineering \(Figure[5](https://arxiv.org/html/2511.07885#A2.F5)\), confirming that local LMs handle most conversational and knowledge\-recall tasks while complex specialized reasoning still benefits from frontier capabilities\.

Local accelerator memory capacity is expanding rapidly\.From 2012 to 2025, local accelerator memory grew∼\\sim126×126\\times\(Figure[13](https://arxiv.org/html/2511.07885#A5.F13)in App\.[E\.8](https://arxiv.org/html/2511.07885#A5.SS8)\); the jump from sub\-20 GB to 200\+ GB through unified\-memory architectures like Apple Silicon removes the key constraint that previously forced workloads to cloud infrastructure, enabling the88–2020B\-active\-parameter models that handle the majority of queries today to run efficiently on local hardware\.

### 4\.2How Intelligence Efficient is Local Inference?\(Q2\)

202320242025SOTALocal Model[Mixtral\-8x7B\-v0\.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)[Llama\-3\.1\-8B\-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)[GPT\-OSS\-120B](https://huggingface.co/openai/gpt-oss-120b)SOTAAcceleratorNVIDIA QuadroRTX 6000NVIDIA RTX6000 AdaAppleM4 MaxSuccess Rate23\.2±1\.9%23\.2\\pm 1\.9\\%48\.7±2\.7%48\.7\\pm 2\.7\\%71\.3±2\.2%71\.3\\pm 2\.2\\%Intelligenceper Watt\(7\.92±0\.32\)\(7\.92\\pm 0\.32\)×10−4\\times 10^\{\-4\}\(1\.80±0\.21\)\(1\.80\\pm 0\.21\)×10−3\\times 10^\{\-3\}\(4\.18±0\.53\)\(4\.18\\pm 0\.53\)×10−3\\times 10^\{\-3\}YoY EfficiencyGain—2\.27×2\.32×Table 2:Increase in Intelligence per Watt for Local LMs: Accuracy per watt has improved over5×5\\timesin two years, driven by advances in both model architectures \(fromMixtral\-8x7BtoGPT\-OSS\-120B\) and accelerator hardware \(from NVIDIA Quadro RTX 6000 to Apple M4 Max\)\. Values are mean±\\pm1\-σ\\sigmastandard deviation across measurement runs\.Intelligence efficiency is improving over time\. Table[2](https://arxiv.org/html/2511.07885#S4.T2)tracks the evolution of local LM capabilities from 2023 to 2025, measuring the best available local LM \(≤20​B\\leq 20Bactive parameters\) paired with state\-of\-the\-art accelerators each year\. On our curated dataset of chat and reasoning queries, accuracy per watt has improved5\.3×5\.3\\timesover this two\-year period: in 2023,Mixtral\-8x7B\-v0\.1onNVIDIA Quadro RTX 6000achieved7\.92×10−47\.92\\times 10^\{\-4\}accuracy per watt; by 2024,Llama\-3\.1\-8B\-InstructonNVIDIA RTX 6000 Adareached1\.80×10−31\.80\\times 10^\{\-3\}\(a2\.27×2\.27\\timesyear\-over\-year gain\); and in 2025,gpt\-oss\-120bonApple M4 Maxachieved4\.18×10−34\.18\\times 10^\{\-3\}\(a2\.32×2\.32\\timesgain\)\. Notably, local LM coverage on single\-turn chat and reasoning queries has increased in lockstep with efficiency gains: from23\.2%23\.2\\%in 2023 to48\.7%48\.7\\%in 2024 to71\.3%71\.3\\%in 2025\. This progression reflects compounding improvements in both model architectures, whichachieve higher accuracythrough advances in pretraining\[[15](https://arxiv.org/html/2511.07885#bib.bib15),[32](https://arxiv.org/html/2511.07885#bib.bib32),[53](https://arxiv.org/html/2511.07885#bib.bib53),[18](https://arxiv.org/html/2511.07885#bib.bib18)\], post\-training\[[9](https://arxiv.org/html/2511.07885#bib.bib9),[68](https://arxiv.org/html/2511.07885#bib.bib68),[19](https://arxiv.org/html/2511.07885#bib.bib19)\], and parameter utilization via mixture\-of\-experts \(MoE\) architectures\[[69](https://arxiv.org/html/2511.07885#bib.bib69),[18](https://arxiv.org/html/2511.07885#bib.bib18)\], and hardware accelerators, which delivermore compute \(FLOPs\) and memory per watt\[[45](https://arxiv.org/html/2511.07885#bib.bib45),[46](https://arxiv.org/html/2511.07885#bib.bib46)\]\.

The decomposition depends on the metric: under accuracy\-per\-watt, model progress contributes 3\.1× and hardware 1\.7× \(Table[2](https://arxiv.org/html/2511.07885#S4.T2)\), while under accuracy\-per\-joule it yields 3\.1× and 5\.9× respectively \(Figure[3](https://arxiv.org/html/2511.07885#S4.F3)\)\. Model progress dominates per\-watt efficiency; hardware progress dominates per\-joule efficiency, because newer accelerators \(HBM3e bandwidth, dedicated tensor units\) reduce latency as well as power\. Both matter — per\-watt for thermally\-constrained deployment, per\-joule for end\-to\-end energy budgets — and Figure[3](https://arxiv.org/html/2511.07885#S4.F3)shows the per\-joule trend is consistent across nine model families \(Llama,Phi,Gemma,Mistral,Falcon,DeepSeek,Qwen,gpt\-oss\), with perplexity\-based confirmation in Figure[8](https://arxiv.org/html/2511.07885#A5.F8)\(App\.[E\.3](https://arxiv.org/html/2511.07885#A5.SS3)\)\. Sustained progress in MoE architectures, quantization, and unified\-memory capacity is the precondition for these trends to continue\. We explore quantization tradeoffs \(App\.[E\.5](https://arxiv.org/html/2511.07885#A5.SS5)\) and serving\-stack sensitivity — batching \(App\.[E\.9](https://arxiv.org/html/2511.07885#A5.SS9)\) and framework choice \(App\.[E\.10](https://arxiv.org/html/2511.07885#A5.SS10)\), finding that FP4 saves 3–3\.5×\\timesenergy per precision step, cloud bs=6464gives 11–20×\\timeshigher IPJ than bs=11, and IPW rankings across vLLM, SGLang, and llama\.cpp are preserved\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/intelligence_efficiency_trend_bar_plot_image.png)Figure 3:Increase in Intelligence per Joule for Local LMs and Accelerators: Efficiency improved18\.0×18\.0\\timesover 16 months, decomposed into3\.1×3\.1\\timesfrom local LMs and5\.9×5\.9\\timesfrom local accelerators\.Local accelerator efficiency has room for improvement: While local accelerators enable deployment of capable models outside data centers, cloud\-grade hardware maintains a substantial efficiency advantage on the same workloads\. AcrossQwen3andGPT\-OSSvariants, theNVIDIA B200achieves1\.40×1\.40\\timeshigher IPW and1\.61\.6–2\.3×2\.3\\timeshigher IPJ thanApple M4 Max, andSambaNova SN40Lachieves up to1\.78×1\.78\\timeshigher IPW and6\.56\.5–7\.4×7\.4\\timeshigher IPJ \(Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14)in App\.[E\.8](https://arxiv.org/html/2511.07885#A5.SS8)\)\. Similar trends are observed on multi\-turn agentic workloads \(see App\.[E\.12](https://arxiv.org/html/2511.07885#A5.SS12)\)\. Per\-joule gaps widen relative to per\-watt gaps because cloud accelerators not only consume less power per unit of accuracy but also complete queries faster\. These gaps stem from purpose\-built components in enterprise accelerators \(HBM3e, dedicated tensor units, optimized memory hierarchies\), whereas local accelerators use unified\-memory architectures that balance diverse workloads under thermal and power constraints\. For these comparisons we use bs =11for both local and cloud, so the gap is intrinsic rather than a batching artifact, revealing substantial headroom for on\-device AI components\.

### 4\.3What Efficiency Gains Can Effective Query Routing Deliver?\(Q3\)

We simulate a hybrid local\-cloud system serving80\.280\.2M queries over 24 hours, representative of realistic daily inference workloads\[[81](https://arxiv.org/html/2511.07885#bib.bib81)\]\. Queries are routed between four small local LMs \(Qwen3\-4B/8B/14B,gpt\-oss\-20b\) onApple M4 Max\(bs =11\) and a frontier model \(Qwen3\-235B\) onNVIDIA H200\(bs =1616, reflecting production serving conditions\)\. Figure[4](https://arxiv.org/html/2511.07885#S4.F4)compares five strategies: routing all queries to the largest model \(baseline\), oracle, and realistic routers at60%60\\%/80%80\\%routing accuracy; misrouted queries fall back to the cloud model\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/figures/burst_gpt_plot_v3.1.png)Figure 4:Energy, Compute, and Capital Gains from Model Routing\. Cumulative resource consumption over 24 hours and 80\.2M LLM queries\[[81](https://arxiv.org/html/2511.07885#bib.bib81)\], routing between44small LMs on Apple M4 Max \(bs =11\) andQwen3\-235Bon an H200 \(bs =1616, batched cloud baseline\)\. The 80%\-accurate router achieves 64\.3% energy, 61\.8% compute, and 59\.0% cost savings versus routing all queries toQwen3\-235B, capturing the majority of gains achievable by the Oracle\.Oracle routing establishes theoretical upper bounds\.Assuming perfect query\-to\-model assignment, oracle routing reduces energy by80\.4%80\.4\\%, compute by77\.3%77\.3\\%, and cost by73\.8%73\.8\\%versus cloud\-only deployment to the largest model \(Figure[4](https://arxiv.org/html/2511.07885#S4.F4)\)\. The80\.7%80\.7\\%of queries that local models can handle correctly are assigned to substantially smaller local models, while frontier compute is reserved for the remaining19\.3%19\.3\\%\. The dominant effect is the reduction in model size, not a hardware efficiency advantage\. The cloud\-only baseline already operates atbs=16\\text\{bs\}=16, so the reported savings are computed against a batched cloud baseline rather thanbs=1\\text\{bs\}=1\(see App\.[E\.9](https://arxiv.org/html/2511.07885#A5.SS9)\)\.

Practical routers achieve substantial gains without perfect accuracy\.A router with80%80\\%accuracy \(a realistic target, with prior work such as RouteLLM\[[52](https://arxiv.org/html/2511.07885#bib.bib52)\]reporting7070–85%85\\%on real workloads\) captures∼\\sim80%80\\%of oracle gains:64\.3%64\.3\\%energy,61\.8%61\.8\\%compute, and59\.0%59\.0\\%cost reduction\. Even a60%60\\%\-accurate router delivers48\.4%/46\.7%/44\.5%48\.4\\%/46\.7\\%/44\.5\\%savings, with misrouted queries falling back to the frontier model so end\-to\-end answer quality is maintained\. The oracle uses ground\-truth correctness as a theoretical upper bound; deployable routers instead usepre\-routingconfidence estimation, and the60%/80%60\\%/80\\%scenarios above explicitly model this imperfection\. At billions of queries daily, these savings scale linearly to annual energy savings in terawatt\-hours\. Although local accelerators are less efficient per query \(§[4\.2](https://arxiv.org/html/2511.07885#S4.SS2)\), routing inverts this at the system level: efficient AI infrastructure comes not from local hardware matching cloud, but from routing across both\.

## 5Conclusion and Key Takeaways

We ask whether local inference can viably redistribute demand from centralized cloud infrastructure, and introduceintelligence per watt\(IPW\)—task accuracy per unit of power—as a unified metric for jointly evaluating local LM capability and local accelerator efficiency\. We take a first step towards answering this question through a longitudinal study of20\+20\+models,88accelerators, and11M queries spanning 2023–2025\. We find that88\.7%88\.7\\%of single\-turn chat and reasoning queries can be handled locally, IPW has improved5\.3×5\.3\\timesover two years through compounding model \(3\.1×3\.1\\times\) and hardware \(1\.7×1\.7\\times\) advances, and although cloud accelerators retain a1\.41\.4–7\.4×7\.4\\timesper\-query efficiency edge \(App\.[E\.8](https://arxiv.org/html/2511.07885#A5.SS8)\), hybrid local\-cloud routing yields6060–80%80\\%aggregate energy, compute, and cost reductions at realistic routing accuracy\. Our study provides three practical takeaways \(App\.[E\.1](https://arxiv.org/html/2511.07885#A5.SS1)\): MoE architectures deliver the best IPW on memory\-rich local devices, aggressive FP4 quantization beats smaller\-but\-higher\-precision models, and router accuracy past∼\\sim80% matters less than expanding the local\-model ensemble\. We release our profiling harness to support periodic re\-evaluation as the local\-inference ecosystem evolves, and discuss the scope and limitations of our study in App\.[C](https://arxiv.org/html/2511.07885#A3)\. In future work, we hope to extend IPW characterization to broader workload regimes \(i\.e\., multi\-modal inference\) and hybrid local\-cloud execution patterns, and to push the local AI frontier through model\-hardware co\-design \(App\.[D](https://arxiv.org/html/2511.07885#A4)\)\.

## References

- Agarwal et al\. \[2025\]S\. Agarwal et al\.gpt\-oss\-120b and gpt\-oss\-20b model card\.*arXiv preprint arXiv:2508\.10925*, 2025\.URL[https://arxiv\.org/abs/2508\.10925](https://arxiv.org/abs/2508.10925)\.
- Alvarez & Marsal \[2025\]Alvarez & Marsal\.Rethinking ai demand part 1: Ai data centers are experiencing a surge of training demand \- what happens when the surge is over?, 2025\.Accessed: 2025\-10\-06\.
- AMD \[2023\]AMD\.Amd instinct mi300x accelerators — product specifications\.AMD official website, 2023\.URL[https://www\.amd\.com/en/products/accelerators/instinct/mi300/mi300x\.html](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)\.192GB HBM3,∼\\sim5\.3TB/s bandwidth, 750W TDP\.
- AMD \[2025\]AMD\.Accelerator specifications\.[https://www\.amd\.com/en/products/specifications/accelerators\.html](https://www.amd.com/en/products/specifications/accelerators.html), 2025\.Accessed: 2025\-10\-30\.
- Anthony et al\. \[2020\]Lasse F\. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan\.Carbontracker: Tracking and predicting the carbon footprint of training deep learning models\.In*ICML Workshop on Challenges in Deploying and Monitoring Machine Learning Systems*, 2020\.URL[https://arxiv\.org/abs/2007\.03051](https://arxiv.org/abs/2007.03051)\.arXiv:2007\.03051\.
- Anthropic \[2025\]Anthropic\.System card: Claude sonnet 4\.5\.System card, Anthropic, September 2025\.URL[https://assets\.anthropic\.com/m/12f214efcc2f457a/original/Claude\-Sonnet\-4\-5\-System\-Card\.pdf](https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf)\.
- Appel et al\. \[2025\]Ruth Appel, Peter McCrory, Alex Tamkin, Michael Stern, Miles McCain, and Tyler Neylon\.Anthropic economic index report: Uneven geographic and enterprise ai adoption\.Technical report, Anthropic, September 2025\.
- Apple \[2024\]Apple\.Apple m4 max — tech specs\.Apple Support / Press Releases, 2024\.URL[https://support\.apple\.com/en\-us/121553](https://support.apple.com/en-us/121553)\.Unified memory bandwidth up to 546 GB/s\.
- Bai et al\. \[2022\]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al\.Constitutional ai: Harmlessness from ai feedback\.*arXiv preprint arXiv:2212\.08073*, 2022\.
- Chatterji et al\. \[2025\]Aaron Chatterji, Tom Cunningham, David Deming, Zoë Hitzig, Christopher Ong, Carl Shan, and Kevin Wadman\.How people use chatgpt\.Technical report, OpenAI, September 2025\.Available at:[https://cdn\.openai\.com/pdf/a253471f\-8260\-40c6\-a2cc\-aa93fe9f142e/economic\-research\-chatgpt\-usage\-paper\.pdf](https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf)\.
- Chen et al\. \[2020\]Lingjiao Chen, Matei Zaharia, and James Y\. Zou\.Frugalml: How to use ML prediction apis more accurately and cheaply\.In*Advances in Neural Information Processing Systems 33 \(NeurIPS 2020\)*, 2020\.URL[https://proceedings\.neurips\.cc/paper/2020/hash/789ba2ae4d335e8a2ad283a3f7effced\-Abstract\.html](https://proceedings.neurips.cc/paper/2020/hash/789ba2ae4d335e8a2ad283a3f7effced-Abstract.html)\.
- Chen et al\. \[2023\]Lingjiao Chen, Matei Zaharia, and James Zou\.Frugalgpt: How to use large language models while reducing cost and improving performance\.*arXiv preprint arXiv:2305\.05176*, 2023\.URL[https://arxiv\.org/abs/2305\.05176](https://arxiv.org/abs/2305.05176)\.
- Chen et al\. \[2024\]Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang\.Routerdc: Query\-based router by dual contrastive learning for assembling large language models\.*Advances in Neural Information Processing Systems*, 37:66305–66328, 2024\.
- Chiang et al\. \[2024\]Wei\-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al\.Chatbot arena: An open platform for evaluating llms by human preference\.In*Forty\-first International Conference on Machine Learning*, 2024\.
- Chowdhery et al\. \[2023\]Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al\.Palm: Scaling language modeling with pathways\.*Journal of Machine Learning Research*, 24\(240\):1–113, 2023\.
- CodeCarbon Contributors \[2024\]CodeCarbon Contributors\.mlco2/CodeCarbon \(v2\.8\.0\)\.Zenodo, 2024\.URL[https://zenodo\.org/doi/10\.5281/zenodo\.14212766](https://zenodo.org/doi/10.5281/zenodo.14212766)\.
- Comanici et al\. \[2025\]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al\.Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025\.URL[https://arxiv\.org/abs/2507\.06261](https://arxiv.org/abs/2507.06261)\.
- DeepSeek\-AI \[2024\]DeepSeek\-AI\.Deepseek\-v3 technical report\.*arXiv preprint arXiv:2412\.19437*, 2024\.
- DeepSeek\-AI \[2025\]DeepSeek\-AI\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Deng et al\. \[2024\]Yuntian Deng, Wenting Zhao, Jack Hessel, Xiang Ren, Claire Cardie, and Yejin Choi\.Wildvis: Open source visualizer for million\-scale chat logs in the wild, 2024\.URL[https://arxiv\.org/abs/2409\.03753](https://arxiv.org/abs/2409.03753)\.
- Deng et al\. \[2023\]Yuyang Deng, Ni Zhao, and Xin Huang\.Early chatgpt user portrait through the lens of data\.*arXiv preprint arXiv:2312\.10078*, 2023\.
- Dettmers et al\. \[2022\]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer\.Llm\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.*Advances in Neural Information Processing Systems*, 35:30318–30332, 2022\.
- Fernandez et al\. \[2025\]Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell\.Energy considerations of large language model inference and efficiency optimizations\.*arXiv preprint arXiv:2504\.17674*, 2025\.
- Gemma Team et al\. \[2025\]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al\.Gemma 3 technical report, 2025\.URL[https://arxiv\.org/abs/2503\.19786](https://arxiv.org/abs/2503.19786)\.
- Gonzaga \[2026\]Waren Gonzaga\.Tinyclaw: The original tiny claw as your personal autonomous ai companion\.[https://github\.com/warengonzaga/tinyclaw](https://github.com/warengonzaga/tinyclaw), 2026\.
- Grattafiori et al\. \[2024\]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The llama 3 herd of models, 2024\.URL[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)\.
- Handa et al\. \[2025\]Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, Kevin K\. Troy, Dario Amodei, Jared Kaplan, Jack Clark, and Deep Ganguli\.Which economic tasks are performed with ai? evidence from millions of claude conversations, 2025\.URL[https://arxiv\.org/abs/2503\.04761](https://arxiv.org/abs/2503.04761)\.
- Hao et al\. \[2023\]Jianwei Hao, Piyush Subedi, Lakshmish Ramaswamy, and In Kee Kim\.Reaching for the sky: Maximizing deep learning inference throughput on edge devices with ai multi\-tenancy\.*ACM Transactions on Internet Technology*, 23\(1\):1–33, 2023\.
- Henderson et al\. \[2020\]Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau\.Towards the systematic reporting of the energy and carbon footprints of machine learning\.*Journal of Machine Learning Research*, 21\(248\):1–43, 2020\.URL[http://jmlr\.org/papers/v21/20\-312\.html](http://jmlr.org/papers/v21/20-312.html)\.
- Hinton et al\. \[2015\]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- HKUDS \[2026\]HKUDS\.Nanobot: The ultra\-lightweight personal ai agent\.[https://github\.com/HKUDS/nanobot](https://github.com/HKUDS/nanobot), 2026\.
- Hoffmann et al\. \[2022\]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al\.Training compute\-optimal large language models\.In*Advances in Neural Information Processing Systems*, volume 35, pages 30016–30030, 2022\.
- Hu et al\. \[2024\]Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay\.Routerbench: A benchmark for multi\-llm routing system\.*arXiv preprint arXiv:2403\.12031*, 2024\.URL[https://arxiv\.org/abs/2403\.12031](https://arxiv.org/abs/2403.12031)\.
- Huang et al\. \[2025\]Zhongzhan Huang, Guoming Ling, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin\.Routereval: A comprehensive benchmark for routing llms to explore model\-level scaling up in llms\.*arXiv preprint arXiv:2503\.10657*, 2025\.doi:10\.48550/arXiv\.2503\.10657\.URL[https://arxiv\.org/abs/2503\.10657](https://arxiv.org/abs/2503.10657)\.
- IBM Research \[2025\]IBM Research\.Granite 4\.0 language models\.[https://github\.com/ibm\-granite/granite\-4\.0\-language\-models](https://github.com/ibm-granite/granite-4.0-language-models), 2025\.Accessed: 2025\-10\-01\.
- Jiang et al\. \[2024\]Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al\.Mixtral of experts\.*arXiv preprint arXiv:2401\.04088*, 2024\.
- Koomey et al\. \[2010\]Jonathan Koomey, Stephen Berard, Marla Sanchez, and Henry Wong\.Implications of historical trends in the electrical efficiency of computing\.*IEEE Annals of the History of Computing*, 33\(3\):46–54, 2010\.
- Li et al\. \[2025\]Xiangchen Li, Dimitrios Spatharakis, Saeid Ghafouri, Jiakun Fan, and Dimitrios Nikolopoulos\.Sled: A speculative llm decoding framework for efficient edge serving\.*arXiv preprint arXiv:2506\.09397*, 2025\.URL[https://arxiv\.org/abs/2506\.09397](https://arxiv.org/abs/2506.09397)\.
- Median\-Group \[2019\]Median\-Group\.numbers\.[https://github\.com/Median\-Group/numbers](https://github.com/Median-Group/numbers), 2019\.GitHub repository\. Accessed: 2025\-10\-30\.
- Memovai \[2026\]Memovai\.Mimiclaw: Openclaw on a $5 chip\.[https://github\.com/memovai/mimiclaw](https://github.com/memovai/mimiclaw), 2026\.
- Miao et al\. \[2023\]Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia\.Specinfer: Accelerating generative large language model serving with tree\-based speculative inference and verification\.In*arXiv preprint arXiv:2305\.09781*, 2023\.URL[https://arxiv\.org/abs/2305\.09781](https://arxiv.org/abs/2305.09781)\.
- Narayan et al\. \[2025\]Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Ré\.Minions: Cost\-efficient collaboration between on\-device and cloud language models\.*arXiv preprint arXiv:2502\.15964*, 2025\.URL[https://arxiv\.org/abs/2502\.15964](https://arxiv.org/abs/2502.15964)\.
- NEAR AI \[2026\]NEAR AI\.Ironclaw: An agent os focused on privacy, security, and extensibility\.[https://github\.com/nearai/ironclaw](https://github.com/nearai/ironclaw), 2026\.
- Nous Research \[2026\]Nous Research\.Hermes agent: The agent that grows with you\.[https://github\.com/NousResearch/hermes\-agent](https://github.com/NousResearch/hermes-agent), 2026\.Self\-improving agent with a built\-in learning loop: autonomous skill creation, cross\-session conversation recall, and persistent user memory\.
- NVIDIA \[2021\]NVIDIA\.Nvidia a100 tensor core gpu — data sheet\.NVIDIA official documentation, 2021\.URL[https://www\.nvidia\.com/content/dam/en\-zz/Solutions/Data\-Center/a100/pdf/nvidia\-a100\-datasheet\-us\-nvidia\-1758950\-r4\-web\.pdf](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf)\.SXM4 version,∼\\sim2\.0TB/s memory bandwidth, 400W\.
- NVIDIA \[2024\]NVIDIA\.Nvidia h200 tensor core gpu — data sheet\.NVIDIA official documentation, 2024\.URL[https://www\.nvidia\.com/en\-us/data\-center/h200/](https://www.nvidia.com/en-us/data-center/h200/)\.141GB HBM3e memory, 4\.8TB/s bandwidth, up to 700W \(SXM variant\)\.
- NVIDIA Corporation \[2019\]NVIDIA Corporation\.*NVIDIA Quadro RTX 6000 Datasheet*\.NVIDIA Corporation, March 2019\.URL[https://www\.nvidia\.com/content/dam/en\-zz/Solutions/design\-visualization/quadro\-product\-literature/quadro\-rtx\-6000\-us\-nvidia\-704093\-r4\-web\.pdf](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/quadro-rtx-6000-us-nvidia-704093-r4-web.pdf)\.Document 704093\-r4\.
- NVIDIA Corporation \[2023\]NVIDIA Corporation\.*NVIDIA RTX 6000 Ada Generation Datasheet*\.NVIDIA Corporation, February 2023\.URL[https://www\.nvidia\.com/content/dam/en\-zz/Solutions/design\-visualization/rtx\-6000/proviz\-print\-rtx6000\-datasheet\-web\-2504660\.pdf](https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/rtx-6000/proviz-print-rtx6000-datasheet-web-2504660.pdf)\.Document 2647623\.
- NVIDIA Corporation \[2024\]NVIDIA Corporation\.NVIDIA Grace Hopper Superchip Architecture, 2024\.URL[https://resources\.nvidia\.com/en\-us\-data\-center\-overview\-mc/en\-us\-data\-center\-overview/grace\-hopper\-superchip\-datasheet\-partner](https://resources.nvidia.com/en-us-data-center-overview-mc/en-us-data-center-overview/grace-hopper-superchip-datasheet-partner)\.Accessed: 2025\-01\-15\.
- NVIDIA Corporation \[2025a\]NVIDIA Corporation\.NVIDIA DGX B200 System Architecture, 2025a\.URL[https://resources\.nvidia\.com/en\-us\-dgx\-systems/dgx\-b200\-datasheet](https://resources.nvidia.com/en-us-dgx-systems/dgx-b200-datasheet)\.Accessed: 2025\-01\-15\.
- NVIDIA Corporation \[2025b\]NVIDIA Corporation\.Nvidia data center gpu resource center\.[https://resources\.nvidia\.com/l/en\-us\-gpu](https://resources.nvidia.com/l/en-us-gpu), 2025b\.Accessed: 2025\-10\-30\.
- Ong et al\. \[2025\]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei\-Lin Chiang, Tianhao Wu, Joseph E\. Gonzalez, M\. Waleed Kadous, and Ion Stoica\.Routellm: Learning to route llms with preference data\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2025\.
- OpenAI \[2023\]OpenAI\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- OpenAI \[2025a\]OpenAI\.Introducing GPT\-5\.OpenAI Blog, 2025a\.URL[https://openai\.com/index/introducing\-gpt\-5/](https://openai.com/index/introducing-gpt-5/)\.Accessed 2025\-09\-13\.
- OpenAI \[2025b\]OpenAI\.Announcing the stargate project\.[https://openai\.com/index/announcing\-the\-stargate\-project/](https://openai.com/index/announcing-the-stargate-project/), January 2025b\.Accessed: 2025\-10\-06\.
- OpenRouter \[2025\]OpenRouter\.OpenRouter\.[https://openrouter\.ai](https://openrouter.ai/), 2025\.Accessed: 23 September 2025\.
- Ouyang et al\. \[2022\]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al\.Training language models to follow instructions with human feedback\.*Advances in Neural Information Processing Systems*, 35:27730–27744, 2022\.
- Oviedo et al\. \[2025\]Felipe Oviedo, Fiodar Kazhamiaka, Esha Choukse, Allen Kim, Amy Luers, Melanie Nakagawa, Ricardo Bianchini, and Juan M\. Lavista Ferres\.Energy use of ai inference: Efficiency pathways and test\-time compute, 2025\.URL[https://arxiv\.org/abs/2509\.20241](https://arxiv.org/abs/2509.20241)\.
- P Team et al\. \[2025\]P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, et al\.Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025\.URL[https://arxiv\.org/abs/2502\.14739](https://arxiv.org/abs/2502.14739)\.
- Patterson et al\. \[2021\]David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis\-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean\.Carbon emissions and large neural network training\.*arXiv preprint arXiv:2104\.10350*, 2021\.URL[https://arxiv\.org/abs/2104\.10350](https://arxiv.org/abs/2104.10350)\.
- Qiu et al\. \[2024\]Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T\. Kalbarczyk, Tamer Başar, and Ravishankar K\. Iyer\.Power\-aware deep learning model serving withμ\\mu\-serve\.In*2024 USENIX Annual Technical Conference \(USENIX ATC 24\)*, 2024\.URL[https://www\.usenix\.org/conference/atc24/presentation/qiu](https://www.usenix.org/conference/atc24/presentation/qiu)\.
- Qwen Team \[2025\]Qwen Team\.Qwen3 technical report, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Saad\-Falcon et al\. \[2026\]Jon Saad\-Falcon, Avanika Narayan, Herumb Shandilya, Hakki Orhun Akengin, Robby Manihani, Gabriel Bo, John Hennessy, Christopher Ré, and Azalia Mirhoseini\.Openjarvis: Personal ai, on personal devices\.[https://scalingintelligence\.stanford\.edu/blogs/openjarvis/](https://scalingintelligence.stanford.edu/blogs/openjarvis/), 2026\.
- SambaNova Systems \[2023\]SambaNova Systems\.Sambanova unveils new ai chip, the SN40L, powering its full stack ai platform\.Announcement, SambaNova Systems, Inc\., Palo Alto, California, September 2023\.URL[https://sambanova\.ai/blog/sn40l\-chip\-best\-inference\-solution](https://sambanova.ai/blog/sn40l-chip-best-inference-solution)\.Fourth\-generation Reconfigurable Dataflow Unit \(RDU\); 5nm TSMC, three\-tier memory architecture\. Accessed: November 10, 2025\.
- Samsi et al\. \[2023\]Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally\.From words to watts: Benchmarking the energy costs of large language model inference\.In*2023 IEEE High Performance Extreme Computing Conference \(HPEC\)*, pages 1–9\. IEEE, 2023\.
- Schuster et al\. \[2022\]Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q\. Tran, Yi Tay, and Donald Metzler\.Confident adaptive language modeling\.In*Advances in Neural Information Processing Systems 35 \(NeurIPS 2022\)*, 2022\.
- Schwartz et al\. \[2020\]Roy Schwartz, Jesse Dodge, Noah A\. Smith, and Oren Etzioni\.Green AI\.*Communications of the ACM*, 63\(12\):54–63, 2020\.doi:10\.1145/3381831\.URL[https://dl\.acm\.org/doi/10\.1145/3381831](https://dl.acm.org/doi/10.1145/3381831)\.
- Shao et al\. \[2024\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shazeer et al\. \[2017\]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\.Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer\.In*International Conference on Learning Representations*, 2017\.
- Shirey \[2025\]Trevin Shirey\.How people use chatgpt: Stats from 13,252 conversations\.[https://www\.webfx\.com/blog/ai/chatgpt\-usage\-statistics/](https://www.webfx.com/blog/ai/chatgpt-usage-statistics/), September 2025\.Accessed: November 2025\.
- Sipeed \[2026\]Sipeed\.Picoclaw: Tiny, fast, and deployable anywhere\.[https://github\.com/sipeed/picoclaw](https://github.com/sipeed/picoclaw), 2026\.
- Somerstep et al\. \[2025\]Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity\.Carrot: A cost aware rate optimal router, 2025\.URL[https://arxiv\.org/abs/2502\.03261](https://arxiv.org/abs/2502.03261)\.
- Song et al\. \[2024\]Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen\.Powerinfer: Fast large language model serving with a consumer\-grade GPU\.In*Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles \(SOSP ’24\)*\. ACM, 2024\.doi:10\.1145/3694715\.3695964\.URL[https://dl\.acm\.org/doi/10\.1145/3694715\.3695964](https://dl.acm.org/doi/10.1145/3694715.3695964)\.
- Steinberger \[2026\]Peter Steinberger\.OpenClaw: Open\-source personal AI assistant\.[https://github\.com/openclaw/openclaw](https://github.com/openclaw/openclaw), 2026\.131K\+ GitHub stars as of March 2026\. Originally released as Clawdbot \(Nov\. 2025\), briefly renamed Moltbot before becoming OpenClaw in January 2026\.
- Strubell et al\. \[2019\]Emma Strubell, Ananya Ganesh, and Andrew McCallum\.Energy and policy considerations for deep learning in NLP\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 3645–3650, Florence, Italy, 2019\. Association for Computational Linguistics\.doi:10\.18653/v1/P19\-1355\.URL[https://aclanthology\.org/P19\-1355](https://aclanthology.org/P19-1355)\.
- Tambe et al\. \[2021\]Thierry Tambe, Coleman Hooper, Lillian Pentecost, Tianyu Jia, En\-Yu Yang, Marco Donato, Victor Sanh, Paul N\. Whatmough, Alexander M\. Rush, David Brooks, and Gu\-Yeon Wei\.Edgebert: Sentence\-level energy optimizations for latency\-aware multi\-task NLP inference\.In*MICRO\-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture*\. ACM, 2021\.doi:10\.1145/3466752\.3480095\.URL[https://dl\.acm\.org/doi/10\.1145/3466752\.3480095](https://dl.acm.org/doi/10.1145/3466752.3480095)\.
- Tschand et al\. \[2025\]Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, Pawan Ambalkar, Ritika Borkar, Ramesh Chukka, Trevor Cockrell, Oliver Curtis, Grigori Fursin, Miro Hodak, Hiwot Kassa, Anton Lokhmotov, Dejan Miskovic, Yuechao Pan, Manu Prasad Manmathan, Liz Raymond, Tom St\. John, Arjun Suresh, Rowan Taubitz, Sean Zhan, Scott Wasson, David Kanter, and Vijay Janapa Reddi\.MLPerf power: Benchmarking the energy efficiency of machine learning systems fromμ\\muwatts to mwatts for sustainable AI\.In*2025 IEEE International Symposium on High\-Performance Computer Architecture \(HPCA\)*, pages 1201–1216, Las Vegas, NV, USA, 2025\. IEEE\.doi:10\.1109/HPCA61900\.2025\.00092\.
- U\.S\. Bureau of Economic Analysis \[2024\]U\.S\. Bureau of Economic Analysis\.GDP by industry, 2024\.URL[https://www\.bea\.gov/data/gdp/gdp\-industry](https://www.bea.gov/data/gdp/gdp-industry)\.
- Wang et al\. \[2024a\]Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji\.MINT: Evaluating LLMs in multi\-turn interaction with tools and language feedback\.In*The Twelfth International Conference on Learning Representations*, 2024a\.
- Wang et al\. \[2024b\]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\.Mmlu\-pro: A more robust and challenging multi\-task language understanding benchmark, 2024b\.URL[https://arxiv\.org/abs/2406\.01574](https://arxiv.org/abs/2406.01574)\.
- Wang et al\. \[2025\]Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu\.BurstGPT: A real\-world workload dataset to optimize llm serving systems\.In*Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’25\)*, Toronto, ON, Canada, 2025\. ACM\.doi:10\.1145/3711896\.3737413\.URL[https://doi\.org/10\.1145/3711896\.3737413](https://doi.org/10.1145/3711896.3737413)\.
- Wilkins et al\. \[2024\]Grant Wilkins, Srinivasan Keshav, and Richard Mortier\.Hybrid heterogeneous clusters can lower the energy consumption of llm inference workloads\.In*Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems*, pages 506–513, 2024\.
- Xie et al\. \[2025\]Zuan Xie, Yang Xu, Hongli Xu, Yunming Liao, and Zhiwei Yao\.A novel hat\-shaped device\-cloud collaborative inference framework for large language models\.In*arXiv preprint arXiv:2503\.18989*, 2025\.URL[https://arxiv\.org/abs/2503\.18989](https://arxiv.org/abs/2503.18989)\.
- Xu et al\. \[2025\]Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai\.Specee: Accelerating large language model inference with speculative early exiting\.*arXiv preprint arXiv:2504\.08850*, 2025\.URL[https://arxiv\.org/abs/2504\.08850](https://arxiv.org/abs/2504.08850)\.
- Yang et al\. \[2023\]Zeyu Yang, Karel Adamek, and Wesley Armour\.Part\-time power measurements: nvidia\-smi’s lack of attention\.*arXiv preprint arXiv:2312\.02741*, 2023\.
- You et al\. \[2023\]Jie You, Jae\-Won Chung, and Mosharaf Chowdhury\.Zeus: Understanding and optimizing gpu energy consumption of dnn training\.In*20th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 23\)*, 2023\.URL[https://www\.usenix\.org/conference/nsdi23/presentation/you](https://www.usenix.org/conference/nsdi23/presentation/you)\.
- Yuan et al\. \[2025\]Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li\.Naturalreasoning: Reasoning in the wild with 2\.8m challenging questions, 2025\.URL[https://arxiv\.org/abs/2502\.13124](https://arxiv.org/abs/2502.13124)\.
- ZeroClaw Labs \[2026\]ZeroClaw Labs\.Zeroclaw: Autonomous ai personal assistant infrastructure\.[https://github\.com/zeroclaw\-labs/zeroclaw](https://github.com/zeroclaw-labs/zeroclaw), 2026\.
- Zhang et al\. \[2025a\]Yiqun Zhang et al\.The avengers: A simple recipe for uniting smaller language models to challenge proprietary giants, 2025a\.
- Zhang et al\. \[2025b\]Yiqun Zhang et al\.Beyond gpt\-5: Making llms cheaper and better via performance\-efficiency optimized routing, 2025b\.

## Appendix ARelated Works

Below, we provide an extended treatment of related works\.

#### LLM Routing

A central challenge in local\-cloud routing systems is determining which model should handle a given query so as to maximize efficiency\. Prior work spans a broad design space, but much of it can be organized around two families of approaches: embedding\-based routers\[[89](https://arxiv.org/html/2511.07885#bib.bib89),[72](https://arxiv.org/html/2511.07885#bib.bib72),[13](https://arxiv.org/html/2511.07885#bib.bib13)\]and generative/decoder\-based routers\[[52](https://arxiv.org/html/2511.07885#bib.bib52)\]\. Embedding\-based methods rely on encoding queries \(and sometimes models\) into a vector space and then applying similarity search or lightweight classification\. Early work largely adopted binary routing, where queries are directed between just two models\. For example, RouteLLM\[[52](https://arxiv.org/html/2511.07885#bib.bib52)\]demonstrated that simple supervised classification can yield up to 85% cost reduction while maintaining GPT\-4\-level performance, but this setting was restricted to two\-model scenarios\. More recent systems generalize routing to multi\-model settings: ensemble\-style methods such as FrugalGPT\[[12](https://arxiv.org/html/2511.07885#bib.bib12)\], RouterDC\[[13](https://arxiv.org/html/2511.07885#bib.bib13)\], and Avengers Pro\[[89](https://arxiv.org/html/2511.07885#bib.bib89),[90](https://arxiv.org/html/2511.07885#bib.bib90)\]show that intelligently combining smaller models can approximate or even surpass larger frontier LMs\. Decoder\-based methods leverage a small language model to directly generate the routing decision\. Causal LLM Routing, suggests that incorporating richer query\-model interaction signals via generative modeling or cross\-attention can yield more robust routing than static embeddings\[[13](https://arxiv.org/html/2511.07885#bib.bib13)\]\. In this work, we are inspired by these novel approaches to routing, and evaluate their performance in the local\-cloud routing setup\.

#### LLM Routing Benchmarks

Recent work has explored benchmarks for LLM query routing, primarily targeting cost–quality tradeoffs across multiple models\. RouterBench\[[33](https://arxiv.org/html/2511.07885#bib.bib33)\]provides a comprehensive suite of curated academic tasks \( 405K samples\) to evaluate routing policies along cost–quality Pareto frontiers\. RouteLLM\[[52](https://arxiv.org/html/2511.07885#bib.bib52)\]introduces a preference\-trained routing framework evaluated on academic benchmarks like MMLU and MT\-Bench, with a focus on achieving quality under token cost constraints, though it remains limited to token\-level metrics\. RouterEval\[[34](https://arxiv.org/html/2511.07885#bib.bib34)\]emphasizes model selection accuracy at scale, compiling over 200M performance records across 8\.5K models and 12 benchmarks to study generalization, yet lacks coverage of real\-world queries\. In contrast, our curated dataset targets routing under naturalistic conditions, leveraging 1M real user queries fromWildchatandNaturalReasoning\. It uniquely supports the exploration of local\-cloud routing tradeoffs beyond just cost and quality, to metrics such as latency, energy, memory, throughput, and more, generated on local accelerators and enterprise\-grade accelerators\. Moreover, in contrast to existing benchmarks, which provide stale performance records limited to models released prior to July 2024, our curated dataset evaluates several state\-of\-the\-art models, including Qwen3\[[62](https://arxiv.org/html/2511.07885#bib.bib62)\]and GPT\-OSS\[[1](https://arxiv.org/html/2511.07885#bib.bib1)\], all released after May 2025\. To support ongoing benchmarking, we release our efficiency profiling harness, a hardware\-agnostic toolkit for generating fresh telemetry and evaluation records as new models become available\.

#### Local–Cloud Inference Systems

Beyond model selection, recent work explores collaborative inference protocols that split generation between local and cloud models\. Minions\[[42](https://arxiv.org/html/2511.07885#bib.bib42)\]proposes a two\-stage protocol where a small on\-device LM handles lightweight processing and a frontier LM performs high\-level reasoning, with an extended version introducing task decomposition and aggregation for improved quality\. Such collaborative schemes offer large energy and cost savings but require careful protocol design to avoid performance loss\. A parallel line of work centers on speculative decoding, where a small draft model generates candidate continuations that are verified or refined by a larger target LM\[[41](https://arxiv.org/html/2511.07885#bib.bib41),[84](https://arxiv.org/html/2511.07885#bib.bib84)\]\. These approaches primarily target latency and throughput, particularly in constrained hardware settings, and typically assume that generation will ultimately invoke a large LM\. Other hybrid protocols like SLED\[[38](https://arxiv.org/html/2511.07885#bib.bib38)\]and HAT\[[83](https://arxiv.org/html/2511.07885#bib.bib83)\]introduce edge\-cloud model partitioning with intermediate state exchange to balance device limitations with quality needs\. While these systems explore fine\-grained collaboration at the token or layer level, our work investigates the limitations of a coarser\-grained alternative: query\-level routing across multiple small and large LMs, where we measure not only accuracy and cost, but also latency, memory, and energy across diverse hardware accelerators\.

#### Efficient AI

We are inspired by work on “Green AI” which proposes treating energy as a first\-class metric alongside accuracy and cost, with calls for standardized reporting and tooling for reproducible accounting of power use and emissions during training and inference\[[67](https://arxiv.org/html/2511.07885#bib.bib67),[75](https://arxiv.org/html/2511.07885#bib.bib75),[60](https://arxiv.org/html/2511.07885#bib.bib60),[29](https://arxiv.org/html/2511.07885#bib.bib29),[5](https://arxiv.org/html/2511.07885#bib.bib5),[16](https://arxiv.org/html/2511.07885#bib.bib16),[58](https://arxiv.org/html/2511.07885#bib.bib58)\]\. Most directly related,[Fernandez et al\. 2025](https://arxiv.org/html/2511.07885#bib.bib23)jointly benchmark accuracy and energy on cloud GPUs \(A100/H100\) for fixed\-format NLP tasks \(sentiment classification, extractive QA, NLI\)\. Our work is complementary but disjoint in scope: we studylocalaccelerators \(M4 Max, RTX 6000, MI300X, smartphone NPUs\) and naturalistic real\-world query distributions, decompose efficiency longitudinally into model versus hardware contributions across successive generations, and analyze hybrid local\-cloud routing, none of which are addressed by their cloud\-only, fixed\-task setup\. Complementary to our focus on local\-cloud routing, cost and efficiency\-driven model selection strategies such as FrugalML and FrugalGPT for API and model cascades, and CALM for token\-wise early exit, dynamically allocate workloads to cheaper or smaller models while preserving quality\[[11](https://arxiv.org/html/2511.07885#bib.bib11),[12](https://arxiv.org/html/2511.07885#bib.bib12),[66](https://arxiv.org/html/2511.07885#bib.bib66)\]\. On\-device and edge studies demonstrate algorithm\-hardware co\-design for lower latency and energy consumption, exemplified by EdgeBERT’s optimizations and PowerInfer’s efficient LLM serving on commodity GPUs\[[76](https://arxiv.org/html/2511.07885#bib.bib76),[73](https://arxiv.org/html/2511.07885#bib.bib73)\]\. Finally, hardware\-aware benchmarking efforts such as “From Words to Watts” and MLPerf Power quantify inference energy across accelerators and standardize power measurement protocols\[[65](https://arxiv.org/html/2511.07885#bib.bib65),[77](https://arxiv.org/html/2511.07885#bib.bib77)\]\.

## Appendix BDataset and Profiling Harness

In this section, we provide additional details on our dataset curation for our study of hybrid local\-cloud LM systems\.

### B\.1Dataset Curation

Here, we provide additional details on the Anthropic Economic Index\[[27](https://arxiv.org/html/2511.07885#bib.bib27)\]categories used as labels \(see Table[3](https://arxiv.org/html/2511.07885#A2.T3)\), the hardware platforms profiled, and the metrics recorded in our curated dataset\.

Life, physical, and social scienceComputer and mathematicalArchitecture and engineeringEducation instruction and libraryInstallation, maintenance, and repairBusiness and financial operationsLegal servicesTransportation and material movingArts, design, sports, entertainment, and mediaProduction servicesFarming, fishing, and forestryHealthcare supportFood preparation and serving relatedHealthcare practitioners and technicalCommunity and social serviceSales and relatedOffice and administrative supportGeneral managementProtective serviceBuilding grounds cleaning and maintenanceConstruction and extractionPersonal care and serviceTable 3:Anthropic Economic Index Categories\[[27](https://arxiv.org/html/2511.07885#bib.bib27)\]\. This taxonomy categorizes occupations into 22 standardized economic domains, adapted from U\.S\. Bureau of Labor Statistics frameworks\. It is designed to support AI impact analysis by aligning labor categories with distinct task structures\.#### Query Curation

When sourcing queries from theWildchatandNaturalReasoningdatasets, we apply robust data cleaning and filtering to ensure the quality and consistency of the sampled queries\. ForNaturalReasoning, we filter out all queries that don’t contain ground truth answers\. ForWildchat, we eliminate non\-English entries to maintain linguistic uniformity across the dataset\. Queries that are malformed, nonsensical, or otherwise unintelligible \(as determined by an LLM judge, i\.e\.,GPT\-4o\-mini\) are discarded to prevent noise\. Additionally, duplicate queries are removed to reduce redundancy and avoid overrepresentation of specific prompts\. Finally, we filter out excessively long queries that exceed a 32,000\-character limit\.

#### Dataset Statistics

[Table 4](https://arxiv.org/html/2511.07885#A2.T4)reveals significant differences in how the two datasets are distributed across domains\.Wildchatis dominated by “Arts, design, sports, entertainment, and media” queries \(47\.1%\), followed by “Computer and mathematical” \(18\.1%\), whileNaturalReasoningis primarily composed of “Life, physical, and social science” \(36\.0%\) and “Computer and mathematical” \(34\.8%\) queries\. We usegpt\-4o\-minito bucket each query into its economic categorization using the prompt below\.

1Youareaquerycategorizer\.Yourtaskistocategorizethefollowinguserqueryintooneofthepredefinedcategoriesbasedonthejob/occupationdomainitrelatestomostclosely\.

2

3Query:"\{query\}"

4

5AvailableCategories:

6\-Officeandadministrativesupport

7\-Transportationandmaterialmoving

8\-Salesandrelated

9\-Foodpreparationandservingrelated

10\-Generalmanagement

11\-Businessandfinancialoperations

12\-Healthcarepractitionersandtechnical

13\-Productionservices

14\-Educationinstructionandlibrary

15\-Healthcaresupport

16\-Constructionandextraction

17\-Installation,maintenance,andrepair

18\-Computerandmathematical

19\-Buildinggroundscleaningandmaintenance

20\-Protectiveservice

21\-Personalcareandservice

22\-Architectureandengineering

23\-Communityandsocialservice

24\-Arts,design,sports,entertainment,andmedia

25\-Life,physical,andsocialscience

26\-Legalservices

27\-Farming,fishing,andforestry

28\-None

29

30Instructions:

311\.Readthequerycarefully

322\.Determinewhichjob/occupationcategorythequeryrelatestomostclosely

333\.Ifthequerydoesn’tclearlyrelatetoanyspecificoccupationcategory,use"None"

344\.RespondwithONLYthecategoryname,exactlyaslistedabove

35

36Category:

Solvability rates vary dramatically by domain and dataset type, where a query’s solvability is defined as its ability to be answered correctly by any of the available local LMs \(e\.g\. Qwen models or GPT OSS\)\.Wildchatqueries show consistently high solvability across most domains \(generally\>94%\>94\\%\), with particularly strong performance in creative and social domains\. In contrast,NaturalReasoningexhibits more variable solvability, with technical domains like “Architecture and engineering” showing only 41\.5% solvability compared to 99\.4% for the same domain inWildchat\. This disparity reflects the complexity difference between open\-ended chat queries and analytical reasoning tasks, supporting our findings that chat queries are more amenable to local model routing than reasoning\-intensive queries\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/figures/domain_routing_stacked_bar.png)Figure 5:Local Win/Tie\-Rate vs\. Cloud LMs by Domain\. Stacked bars show the fraction of single\-turn chat and reasoning queries handled by local LMs \(<20​B<20Bactive parameters; blue\) versus those routed to frontier models in the cloud \(red\), computed per economic index domain\[[7](https://arxiv.org/html/2511.07885#bib.bib7)\]DomainWCCountWC%WCSolv %NRCountNR%NRSolv %Computer and mathematical90,66218\.199\.5174,24234\.867\.3Arts, design, sports, entertainment,and media235,65847\.198\.72,6480\.552\.9Life, physical, andsocial science28,0795\.698\.8180,06536\.060\.5None49,0149\.897\.379,75216\.065\.6Education instruction and library23,1964\.697\.213,8642\.880\.4Architecture and engineering5,7821\.298\.928,7625\.840\.8Business and financial operations19,6283\.997\.88,7791\.855\.3Healthcare practitionersand technical8,9051\.898\.11,8510\.466\.3Office and administrativesupport6,9591\.491\.6250\.048\.0Legal services5,2081\.098\.61,3490\.369\.0Community and social service6,1251\.297\.04040\.176\.2Transportation andmaterial moving1,8900\.495\.13,9140\.853\.0Sales and related4,6890\.997\.82180\.067\.4Food preparation andserving related3,3080\.798\.35070\.161\.9General management3,3640\.796\.73400\.167\.9Installation, maintenance,and repair9540\.297\.12,0370\.456\.2Farming, fishing, and forestry1,6770\.399\.54110\.165\.5Protective service1,3150\.397\.92370\.056\.5Construction and extraction9910\.297\.21470\.060\.5Healthcare support1,1120\.297\.5120\.0100\.0Production services5460\.1100\.03340\.165\.3Personal care and service6480\.192\.9190\.00\.0Building grounds cleaningand maintenance2780\.1100\.0700\.072\.9TOTAL500K100\.098\.4500K100\.063\.0Table 4:Dataset Domain Composition and LM Coverage \(≤20​B\\leq 20BActive Parameter Models\)\. Comparison of domain distribution and model solvability rates acrossWildchat\(WC\) andNaturalReasoning\(NR\) datasets\. Solvability indicates the percentage of problems that can be solved correctly by at least one model with≤20​B\\leq 20Bactive parameters\.CategoryWildchatMMLU ProSuperGPQAAverageComputer and mathematical93\.4%90\.6%72\.8%85\.6%Life, physical, and social science91\.1%84\.7%50\.4%75\.4%Sales and related86\.8%74\.2%64\.3%75\.1%Business and financial operations89\.3%82\.9%52\.5%74\.9%Production services89\.7%85\.7%48\.8%74\.7%Office and administrative support88\.5%83\.3%44\.8%72\.2%Healthcare practitioners and technical88\.9%78\.8%48\.0%71\.9%Installation, maintenance, and repair90\.4%80\.0%44\.4%71\.6%Architecture and engineering90\.0%73\.2%51\.6%71\.6%Protective service88\.6%75\.0%47\.6%70\.4%Education instruction and library90\.6%77\.4%43\.2%70\.4%Farming, fishing, and forestry87\.2%77\.8%45\.9%70\.3%None86\.5%77\.4%42\.4%68\.8%General management88\.2%76\.3%41\.6%68\.7%Transportation and material moving91\.1%66\.7%44\.9%67\.5%Construction and extraction91\.0%66\.7%41\.4%66\.4%Food preparation and serving related86\.1%82\.6%30\.3%66\.3%Community and social service87\.4%64\.1%45\.5%65\.7%Arts, design, sports, entertainment, and media84\.8%75\.5%36\.2%65\.5%Building grounds cleaning and maintenance90\.1%71\.4%30\.8%64\.1%Legal services87\.1%61\.5%43\.6%64\.1%Healthcare support86\.5%60\.0%38\.9%61\.8%Personal care and service89\.4%50\.0%25\.0%54\.8%Table 5:GPT\-OSS\-120B Performance across Datasets\.Performance metrics acrossWildchat,NaturalReasoning,MMLU Pro, andSuperGPQAbenchmarks, organized by Anthropic Economic Index categories\[[27](https://arxiv.org/html/2511.07885#bib.bib27)\]\.
#### Metrics

We detail all the metrics collected via our profiling harness in Table[6](https://arxiv.org/html/2511.07885#A2.T6)\. For our correctness evaluations onWildchat, we use an LLM\-as\-a\-judge approach to evaluate model generated answers against a ground truth answer fromQwen3\-235B\. For our correctness evaluations ofNaturalReasoning, we use another LLM\-judge prompt, but compare against ground truth answers provided in the original dataset\. We provide both LLM\-judge prompts below\. The LLM used for each respective evaluation isGPT\-4o\. ForSuperGPQAandMMLU Pro, we simply compare the multiple choice answer selected in the response to the multiple choice answer of the reference response\.

WildchatLLM\-judge Prompt

1

2YouareanimpartialjudgeevaluatingthequalityoftwoAI\-assistantrepliestothesameuserprompt\.

3

4Step1:Generateyourownanswer

5Writetheresponse\*you\*wouldgivetotheuser\.Keepitseparatefromlateranalysis\.

6

7Step2:Decidethequerytype

8Classifytheuserpromptaseither

9\-\*\*Subjective/open\-ended\*\*\(creativewriting,opinion,advice,brainstorming\)

10\-\*\*Objective/technical\*\*\(code,math,logicalderivationswithasinglecorrectoutcome\)

11Ifuncertain,defaultto"Subjective"\.

12

13Step3\-Scoreeachassistantwiththecorrectrubric

14

15\|Querytype\|Criteria\|

16\|\-\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|

17\|Subjective/open\-ended\|1\.Correctness/factualsoundness2\.Helpfulness3\.Relevance4\.Conciseness5\.Creativity&novelty\|

18\|Objective/technical\|1\.Correctnessonly\|

19

20Whenusingthemulti\-criteriarubric,notestrengthsandweaknessesfor\*\*each\*\*dimension\.

21Whenusingthesingle\-criterionrubric,focusexclusivelyonfactual/functionalaccuracyandignorestyleorflair\.

22

23Step4:Compare&justify

24Explainwhichassistantisbetterandwhy,correctinganymistakesyoufind\.Highlightmissingbutimportantdetails\.\*\*Beconcise\.\*\*

25

26Step5:Verdict

271\.AssistantAissignificantlybetter:\[\[A\>\>B\]\]

282\.AssistantAisslightlybetter:\[\[A\>B\]\]

293\.Tie,AssistantAisequal:\[\[A=B\]\]

304\.AssistantBisslightlybetter:\[\[B\>A\]\]

315\.AssistantBissignificantlybetter:\[\[B\>\>A\]\]

32

33Chooseexactlyonetokenfrom:‘\[\[A\>\>B\]\]‘,‘\[\[A\>B\]\]‘,‘\[\[A=B\]\]‘,‘\[\[B\>A\]\]‘,‘\[\[B\>\>A\]\]‘\.

34

35\-\-\-

36

37\#\#\#Outputformat\(strict\)

38Return\*\*only\*\*aJSONobjectthatmatchestheprovidedschema:

NaturalReasoningLLM\-judge Prompt

1Youareevaluatingaresponsetoascientific/technicalquestionagainstareferenceanswer\.

2

3Yourtaskistodetermineiftheresponseisfactuallycorrectandcompletecomparedtothereference\.

4

5Consider:

61\.Scientificaccuracyoffactsandconcepts

72\.Mathematicalcorrectness\(ifapplicable\)

83\.Completenessoftheanswer

94\.Technicalprecision

10

11Question:\{question\}

12

13Response:\{response\}

14

15ReferenceAnswer:\{reference\}

16

17ReturnONLY’True’iftheresponseiscorrectandcomplete,’False’otherwise\.

Table[6](https://arxiv.org/html/2511.07885#A2.T6)catalogs the per\-query metrics collected by our profiling harness across compute, energy, latency, memory, and utilization dimensions; together these constitute the raw measurements from which all IPW, IPJ, PPW, and PPJ figures in the main paper are derived\.

MetricDescriptionflops\_per\_requestFLOPs per query\.macs\_per\_requestMACs per query; proxy for compute\.per\_query\_joulesEnergy per query \(J\)\.total\_joulesTotal energy across queries\.per\_token\_msLatency per token \(ms\)\.throughput\_tokens\_per\_secToken output rate \(toks/s\)\.time\_to\_first\_token\_secondsTime to first token \(s\)\.total\_query\_secondsTotal time per query \(s\)\.cpu\_mb\.avg / max / median / minCPU memory usage \(MB\)\.gpu\_mb\.avg / max / median / minGPU memory usage \(MB\)\.initialization\_duration\_secondsModel load time \(s\)\.batch\_sizeQuery batch size\.gpu\_memory\_utilizationGPU memory use \(0–1\)\.max\_model\_lenMax token length allowed\.max\_num\_batched\_tokensMax batch token count\.max\_output\_tokensMax output tokens\.num\_workersNumber of threads\.temperatureSampling temperature\.top\_kTop\-k cutoff\.top\_pTop\-p \(nucleus\) threshold\.warmup\_stepsWarm\-up steps\.per\_query\_watts\.avg / max / median / minGPU power draw per query \(W\)\.total\_watts\.avg / max / median / minSession GPU power draw \(W\)\.cpu\_countCPU core count\.cpu\_brandCPU model\.host\_nameMachine hostname\.os\_name / os\_version / kernel\_versionOS and kernel info\.temperature\.avg / max / median / minDevice temperature \(∘C\)\.inputInput tokens per query\.outputOutput tokens per query\.Table 6:Dataset Metrics\. Summary of compute, latency, memory, and energy profiling metrics\.
#### Hardware Backends

Details regarding profiled hardware can be found in Table[7](https://arxiv.org/html/2511.07885#A2.T7)\.

#### Data Generation Procedure

We generate model outputs using consistent decoding settings across all tasks:temperature==0\.6,top\-p==0\.95,top\-k==20,min\-p==0\.0, and a 32768\-token output limit\. ForNaturalReasoning,SuperGPQAandgpqaqueries, we enable deliberative prompting \(use thinking==True\); forWildchat, we disable it\. ForQwenmodels, we apply a repetition penalty of 1\.1 and length penalty of 1\.0\.

HardwareMemoryBandwidthPowerNVIDIA A100 \(Ampere\)40 GB HBM21,555 GB/s400 W TDPNVIDIA H200 \(Hopper\)141 GB HBM3e4\.8 TB/sUp to 700 W TDPNVIDIA B200 \(Blackwell\)192 GB HBM3e8 TB/s1000 W TDPNVIDIA GH200 \(Grace Hopper\)144 GB HBM3e \(\+624 GB LPDDR5X\)4\.8 TB/s \(GPU\)1000 W TDPNVIDIA Quadro RTX 6000 \(Turing\)24 GB GDDR6672 GB/s295 W TDPNVIDIA RTX 6000 Ada Generation48 GB GDDR6960 GB/s300 W TDPAMD Instinct MI300X \(CDNA 3\)192 GB HBM35\.3 TB/s \(peak\)750 W TBPApple Mac Studio \(M4 Max\)128 GB unified546 GB/s480 W \(system PSU\)†\\daggerApple iPhone 16 Pro \(A18 Pro\)8 GB LPDDR5X60 GB/s∼12\{\\sim\}12W \(SoC peak\)SambaNova SN40L RDU64 GB HBM2E1\.6 TB/s500 W TDPTable 7:Accelerator Details\. Memory, bandwidth, and power specifications of evaluated accelerators and systems\.†\\daggerFor the Apple Mac Studio \(M4 Max\), 480 W is the system power\-supply continuous rating; the M4 Max SoC under sustained AI workloads draws substantially less\. Per\-query power measurements in our study are taken viapowermetricsprocessor\_power\.actual\(GPU subsystem only\) so that comparisons against NVML/ROCm\-SMI accelerator\-only power on other platforms are like\-for\-like \(see App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px6)\)\.
#### Telemetry Collection

We collected telemetry by instrumenting host\-level samplers that interface directly with vendor\-supported system APIs on each platform\. Data were obtained from NVML on NVIDIA\-equipped hosts, from thepowermetricsfacility on macOS, and from ROCm SMI on AMD\-equipped hosts\. Each sampler queried the respective system interface to obtain GPU\- and system\-level measurements and produced synchronized records suitable for downstream quantitative analysis\.

On NVIDIA systems, we interface directly with NVML and enumerate all visible GPUs\. For each device, we query instantaneous power as reported by the driver, read cumulative energy from the on\-device counter, obtain GPU temperature from the hardware sensor, and retrieve memory usage from the device’s memory interface\. Units are normalized \(e\.g\., milliwatts and millijoules mapped to watts and joules; bytes to megabytes\)\. In multi\-GPU hosts, power and memory are summed across devices and temperature is averaged to yield a single aggregate view\. Each record also includes host memory usage from OS counters, a nanosecond timestamp, and device identity and backend provenance\.

On macOS, we executepowermetricswith elevated privileges and ingest its continuouspliststream\. Eachplistframe is parsed to extract the GPU power value exposed by the system \(Apple Silicon:processor\_power\.actual; Intel:processor\.combined\_power\), which is normalized to watts\. Energy \(joules\) is obtained by numerically integrating the power signal over successive frames using the measured inter\-frame wall\-clock interval\. In parallel, system memory usage is sampled from OS counters\. Every observation is timestamped and annotated with Apple device identity and an explicitpowermetricsbackend tag\.

On AMD systems, we use ROCm SMI to query current GPU power \(watts\), read temperature from junction or edge sensors \(∘C\), and obtain VRAM usage from the device memory interface \(bytes to megabytes\)\. Energy \(joules\) is computed by integrating the power signal over time using consecutive sampling intervals\. System memory usage is read from OS counters\. In multi\-GPU machines, the primary device under observation is explicitly selected \(GPU index 0 in our setup\), and all records carry precise timestamps together with device identity and backend metadata\.

To ensure measurement precision and account for variance in inference\-time behavior, we execute each query 10 times and aggregate power measurements across runs\. For each query, we compute the mean power draw \(watts\) and mean energy consumption \(joules\) per query by averaging across these 10 independent executions\. This repeated sampling approach reduces measurement noise and provides robust estimates of per\-query resource consumption that account for system\-level variability in accelerator utilization, thermal conditions, and memory allocation patterns\.

Cost SavingsCompute SavingsEnergy SavingsSize Threshold \(≤\\leq\)Qwen\+ GPT\-OSSQwenQwen\+ GPT\-OSSQwenQwen\+ GPT\-OSSQwen4B65\.2%65\.2%65\.1%65\.1%63\.5%63\.5%8B80\.8%80\.8%83\.1%83\.1%79\.6%79\.6%14B89\.0%89\.0%93\.0%93\.0%87\.0%87\.0%20B90\.5%—97\.4%—89\.4%—32B91\.3%91\.9%97\.4%92\.8%90\.4%90\.5%120B91\.1%—97\.7%—90\.7%—Table 8:Cost, Compute, and Energy Savings from Local\-Cloud Routing on WildChat: Savings across different resources while maintaining task accuracy of SOTA open\-source cloud model \(i\.e\.Qwen3 235B\-A22B\)\.Cost SavingsCompute SavingsEnergy SavingsSize Threshold \(≤\\leq\)Qwen\+ GPT\-OSSQwenQwen\+ GPT\-OSSQwenQwen\+ GPT\-OSSQwen4B52\.9%52\.9%54\.5%54\.5%46\.3%46\.3%8B60\.5%60\.5%62\.5%62\.5%54\.0%54\.0%14B68\.7%68\.7%70\.1%70\.1%62\.5%62\.5%20B73\.3%—72\.2%—67\.8%—32B76\.9%75\.9%75\.1%75\.1%72\.4%71\.6%120B86\.7%—86\.0%86\.0%85\.9%—Table 9:Cost, Compute, and Energy Savings from Local\-Cloud Routing on NaturalReasoning: Savings across different resources while maintaining task accuracy of SOTA open\-source cloud model \(i\.e\.Qwen3 235B\-A22B\)\.

## Appendix CLimitations and Broader Impacts \(Extended\)

#### Limitations\.

Our study is subject to several limitations\.\(1\) Measurement precision\.Our energy and power measurements rely on software\-level telemetry \(NVML,powermetrics, ROCm SMI\), which can introduce inaccuracies of1010–15%15\\%\[[85](https://arxiv.org/html/2511.07885#bib.bib85)\]and may not capture milliwatt\-level variations that hardware wattage meters would\. While our methodology aligns with established practices and provides consistent relative comparisons, absolute energy values should be interpreted accordingly\.\(2\) Query coverage\.Our analysis focuses on single\-turn chat and reasoning queries\. Multi\-turn conversations, agentic workflows, tool use, and long\-context applications represent substantial portions of real\-world LLM traffic; we report a multi\-turn extension on GAIA and TerminalBenchV2 in App\.[E\.12](https://arxiv.org/html/2511.07885#A5.SS12)that confirms the qualitative patterns generalize, but routing decisions and efficiency tradeoffs may still differ in those settings\.\(3\) Evaluation methodology\.Our correctness measurements rely on LLM\-as\-a\-judge \(Qwen3\-235B\) for open\-ended chat queries, which inherits any biases or systematic errors of the judge model\.\(4\) Hardware coverage\.While we evaluate eight datacenter, workstation, and consumer\-class accelerators \(and additionally a smartphone\-class accelerator in App\.[E\.11](https://arxiv.org/html/2511.07885#A5.SS11)\), we do not cover the full diversity of available local hardware \(e\.g\., other mobile NPUs, integrated GPUs, edge accelerators\), and our findings on local efficiency may not extrapolate uniformly across these classes\.\(5\) Batch size = 1 cloud comparison\.Our main per\-query comparisons \(Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14)\) use batch size 1 to follow standard local\-inference benchmarking practice\[[28](https://arxiv.org/html/2511.07885#bib.bib28)\]\. This is conservative for cloud accelerators: App\.[E\.9](https://arxiv.org/html/2511.07885#A5.SS9)shows bs=64 on B200 yields1111–20×20\\timeshigher IPJ\. The routing simulation \(Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3)\) uses bs=16 for the cloud baseline\.\(6\) Single\-runtime design\.We standardize onvLLMto isolate model\-hardware effects; App\.[E\.10](https://arxiv.org/html/2511.07885#A5.SS10)shows absolute IPW shifts by33–12%12\\%acrossvLLM/SGLang/llama\.cppbut rankings are preserved \(Kendall’sτ∈\[0\.87,0\.93\]\\tau\\in\[0\.87,0\.93\]\)\. System\-level work such as Zeus\[[86](https://arxiv.org/html/2511.07885#bib.bib86)\]and uServe\[[61](https://arxiv.org/html/2511.07885#bib.bib61)\]explores complementary axes \(scheduling, power management\) that we do not vary\.\(7\) Ecosystem velocity\.Our findings reflect models and accelerators available as of October 2025; sustained increases in DRAM/HBM pricing could slow the pace at which larger models become locally deployable, and our open\-source profiling harness is designed to support periodic re\-evaluation\.

#### Broader Impacts\.

Demonstrating that local inference can serve a substantial fraction of LLM queries has potential benefits for energy consumption, infrastructure cost, and access to AI capabilities, particularly in settings where cloud connectivity is unreliable, expensive, or undesirable for privacy reasons\. Our findings are situated within a rapidly accelerating local\-AI ecosystem\. On the software side, a wave of open\-source personal\-AI agent stacks has emerged that treats on\-device execution as a design constraint rather than a fallback\. Projects span the full hardware spectrum:OpenClaw\[[74](https://arxiv.org/html/2511.07885#bib.bib74)\]andZeroClaw\[[88](https://arxiv.org/html/2511.07885#bib.bib88)\]target workstation deployment;NanoBot\[[31](https://arxiv.org/html/2511.07885#bib.bib31)\],TinyClaw\[[25](https://arxiv.org/html/2511.07885#bib.bib25)\], andHermes Agent\[[44](https://arxiv.org/html/2511.07885#bib.bib44)\]focus on lightweight runtimes with cross\-session memory and skill systems;IronClaw\[[43](https://arxiv.org/html/2511.07885#bib.bib43)\]emphasizes privacy and security;PicoClaw\[[71](https://arxiv.org/html/2511.07885#bib.bib71)\]runs in under 10 MB of RAM on $10 RISC\-V hardware; andMimiClaw\[[40](https://arxiv.org/html/2511.07885#bib.bib40)\]demonstrates a full agent on a $5 ESP32 microcontroller\. Recent academic and industrial efforts includeOpenJarvis\[[63](https://arxiv.org/html/2511.07885#bib.bib63)\], which integrates local model serving with agent and memory components\. These projects illustrate growing interest in deployment patterns where local inference handles a substantial share of personal\-AI queries\. On the hardware side, purpose\-built local AI hardware, e\.g\. NVIDIA DGX Spark \(128 GB unified memory, 1 petaFLOP at FP4\) and Dell Pro Max workstations with GB300 \(748 GB, 20 petaFLOPS at FP4\), is bringing datacenter\-class AI to desktop devices for the first time\. Hybrid local\-cloud routing, as we show, can reduce inference energy by6060–80%80\\%at platform scale; the IPW metric and profiling harness we release are intended as a shared evaluation framework for tracking efficiency across this growing ecosystem as it matures\. Our profiling harness lowers the barrier for systematic energy benchmarking, which we view as a public good given the rising aggregate energy footprint of AI workloads\. We also note potential negative implications\. First, increased local deployment of capable LMs may make certain forms of misuse \(e\.g\., generating misinformation or harassment\) harder to monitor and mitigate compared to cloud\-served inference, where providers can apply usage policies\. Second, local inference redistributes energy consumption to consumer power grids rather than eliminating it, and results on intelligence per watt could be misused to justify deployment patterns \(e\.g\., always\-on local agents\) that, in aggregate, raise total energy consumption despite per\-query efficiency gains: a Jevons\-paradox concern we have not quantified\. Third, our cost and energy comparisons are approximate and could be cited out of context to support specific procurement or policy decisions for which finer\-grained, deployment\-specific analysis would be more appropriate\.

## Appendix DFuture Work

We outline three directions for extending this work\.

### D\.1Broadening Workload Coverage

#### Multi\-turn and Agentic Workflows

Our main study targets single\-turn queries, and App\.[E\.12](https://arxiv.org/html/2511.07885#A5.SS12)reports an extension on GAIA and TerminalBenchV2 confirming that local\-cloud efficiency patterns generalize qualitatively\. Future work should profile IPW across diverse agent stacks, longer trajectories, and workloads dominated by tool\-call overhead\. Routing in agentic settings must account for cumulative trajectory cost rather than per\-turn cost, which may shift the local\-cloud crossover relative to single\-turn workloads\.

#### Long\-context Inference

Our evaluation caps inputs at32,00032\{,\}000characters, but production workloads increasingly involve100100K–11M token contexts where prefill dominates total energy and KV\-cache pressure becomes the binding constraint\. Characterizing IPW as a function of context length would clarify how the prefill\-decode energy split shifts with input size and how unified\-memory and HBM architectures behave in such settings\.

#### Multimodal Workloads

Our study is text\-only\. Vision\-language, audio, and video inference have different compute and memory profiles: image and video tokenization shift the prefill\-decode balance, and modality\-specific encoders introduce energy costs not captured by text\-only profiling\. Extending our harness to multimodal models and characterizing local\-hardware viability on these workloads is an important direction\.

### D\.2Pushing the Local\-AI Frontier

#### Closing the Reasoning Gap

App\.[E\.2](https://arxiv.org/html/2511.07885#A5.SS2)shows Level 5 reasoning queries remain95%95\\%unsolved by current local LMs, and the hardest reasoning problems show markedly slower year\-over\-year progress than chat or moderate reasoning\. Future small\-model architectures, reasoning\-focused post\-training, and test\-time compute strategies that fit within local power budgets are needed to push the locally\-serviceable share beyond88\.7%88\.7\\%\. Whether long chain\-of\-thought reasoning can be made energy\-efficient enough for local deployment is an open question\.

#### Specialized On\-device Architectures

Today’s local LMs are predominantly GPU\-centric models repurposed for local hardware, and App\.[E\.8](https://arxiv.org/html/2511.07885#A5.SS8)shows local accelerators trail cloud accelerators by1\.41\.4–7\.4×7\.4\\timesin IPW on identical workloads\. Closing this gap may require NPU\-first model designs, sparsity\-aware architectures, and co\-designed model\-hardware stacks\. The smartphone\-class results in App\.[E\.11](https://arxiv.org/html/2511.07885#A5.SS11)\(∼7×\{\\sim\}7\\timeshigher IPW than workstation GPUs at∼12\{\\sim\}12W\) suggest substantial headroom for purpose\-built mobile deployments\.

#### Quantization Frontiers Below FP4

App\.[E\.5](https://arxiv.org/html/2511.07885#A5.SS5)shows FP16→\\rightarrowFP4 yields33–3\.5×3\.5\\timesenergy reduction at∼2\.5\{\\sim\}2\.5pp accuracy loss per step\. Sub\-FP4 regimes \(INT2, ternary, binary\) offer additional headroom but require both training recipes that preserve accuracy and dedicated hardware support that few accelerators currently provide\. Characterizing the accuracy floor of extreme quantization and the hardware support needed to realize its theoretical gains are open directions\.

### D\.3Operationalizing Hybrid Inference

#### Serving\-stack and System\-level Optimizations

We standardize onvLLMand bs=1 for local inference; App\.[E\.9](https://arxiv.org/html/2511.07885#A5.SS9)and App\.[E\.10](https://arxiv.org/html/2511.07885#A5.SS10)confirm batch size and framework \(i\.e\., VLLM vs SGLANG\) shift absolute IPW but preserve rankings\. A full treatment of speculative decoding, paged attention, DVFS, power capping, and cross\-application local batching could materially improve local IPW\. Cross\-application batching is particularly promising because single\-user devices cannot batch across users but can aggregate concurrent queries across on\-device applications\.

#### Measuring IPW Under Hybrid Execution

Our IPW metric is defined per \(model, accelerator\) pair and characterizes local or cloud inference in isolation\. Hybrid execution patterns—speculative decoding with a local draft and cloud verifier, edge\-cloud model partitioning\[[38](https://arxiv.org/html/2511.07885#bib.bib38),[83](https://arxiv.org/html/2511.07885#bib.bib83)\], and collaborative protocols like Minions\[[42](https://arxiv.org/html/2511.07885#bib.bib42)\]—split a single query across local and cloud infrastructure and require an extended IPW formulation that aggregates power and energy across heterogeneous accelerators\.

## Appendix ELocal\-Cloud Experiments

### E\.1Key Takeaways for Practitioners

We consolidate our empirical findings into eight concrete decision rules, each grounded in a specific experiment\.\(1\) Architecture\.On memory\-rich local devices, MoE delivers the best IPW:gpt\-oss\-120b\(≤20\\leq 20B active\) achieves the highest single\-model coverage \(71\.4%71\.4\\%,[Figure 2](https://arxiv.org/html/2511.07885#S4.F2)\) and best IPW \([Table 2](https://arxiv.org/html/2511.07885#S4.T2)\); capacity\-to\-compute decoupling more than compensates for storing all experts\.\(2\) Quantization\.FP16→\\toFP4 yields33–3\.5×3\.5\\timesenergy reduction at∼2\.5\{\\sim\}2\.5pp accuracy loss per step \(App\.[E\.5](https://arxiv.org/html/2511.07885#A5.SS5)\); a larger model at FP4 typically beats a smaller model at FP16, so scale model size first and quantize aggressively\.\(3\) Routing\.Invest in router accuracy up to∼80%\{\\sim\}80\\%\(80%80\\%of oracle gains\); beyond that, expand the local\-model ensemble: the17\.317\.3pp best\-single vs\. best\-of\-local gap \([Figure 2](https://arxiv.org/html/2511.07885#S4.F2)\) shows diversity matters more than additional routing accuracy\.\(4\) Domains\.Coverage exceeds93%93\\%for creative fields but drops to60%60\\%for Architecture & Engineering \([Figure 5](https://arxiv.org/html/2511.07885#A2.F5)\); model developers expanding local\-AI viability should prioritize technical reasoning\.\(5\) Hardware bottleneck\.Cloud accelerators with HBM3e and dedicated tensor units achieve1\.41\.4–7\.4×7\.4\\timeshigher efficiency than M4 Max’s unified memory \(Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14)\); the dominant local bottleneck is memory bandwidth and specialized compute, not raw FLOPs\.\(6\) Power envelope\.Smartphone\-class NPUs occupy a distinct regime: the iPhone 16 Pro achieves∼\\sim7×7\\timeshigher IPW than workstation GPUs at∼12\{\\sim\}12W \(App\.[E\.11](https://arxiv.org/html/2511.07885#A5.SS11)\), motivating NPU\-optimized mobile deployment for the lightest queries\.\(7\) Serving stack\.Cloud bs=6464on B200 yields1111–20×20\\timeshigher IPJ than bs=11\(App\.[E\.9](https://arxiv.org/html/2511.07885#A5.SS9)\); local single\-user deployment cannot batch but can aggregate concurrent on\-device applications\.\(8\) Framework\.Absolute IPW shifts33–12%12\\%across vLLM, SGLang, llama\.cpp but rankings are preserved \(Kendall’sτ∈\[0\.87,0\.93\]\\tau\\in\[0\.87,0\.93\], App\.[E\.10](https://arxiv.org/html/2511.07885#A5.SS10)\); standardize for comparison, but benchmark on the target stack for absolute throughput\.

### E\.2How has local LM task coverage changed over different “difficulty” slices of the data

Using labels for query difficulty we quantify the rate of improvement of local LMs across task difficulty slices\. We label each query by the minimum model size \(in parameters\) required to solve it when considering the SOTA LMs as of August 2025, categorizing queries into five difficulty levels: level 1 \(≤\\leq4B params\), level 2 \(≤\\leq8B params\), level 3 \(≤\\leq20B params\), level 4 \(≤\\leq235B params\), and level 5 \(unsolvable\)\.

Forchat tasks\(see Figure[6](https://arxiv.org/html/2511.07885#A5.F6)\), we observe near\-universal performance gains across all difficulty levels, with 2025 models achieving 98\-99% success on levels 1\-3 and 92\.6% on level 4\. Absolute improvements range from \+55\.4 percentage points \(pp\) for level 1 to \+76\.4 pp for level 3, indicating relatively uniform capability gains\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/figures/wildchat_difficulty_analysis.png)Figure 6:Chat Task Performance by Difficulty Level and Year\. Model success rates across four difficulty levels and three model generations \(2023, 2024, 2025\)\. The data reveals dramatic progress across all difficulty levels, with 2023 models achieving 28\.79% overall success rising to 98\.12% by 2025\. Notably, Levels 1\-3 approach near\-perfect performance \(98\-99%\), while Level 4 shows the largest relative improvement \(\+210\.4% per year\) despite starting from the lowest baseline \(17\.77%\)\.Forreasoning tasks\(see Figure[7](https://arxiv.org/html/2511.07885#A5.F7), the pattern differs substantially\. While levels 1\-3 show strong improvements \(\+24\.0, \+37\.8, and \+53\.9 pp respectively\), levels 4 and 5 exhibit markedly slower progress\. Level 4 improves by only \+23\.8 pp \(7\.93% to 31\.72%\), and level 5 remains largely unsolved with just \+1\.5 pp improvement \(3\.27% to 4\.72%\)\. This suggests that while local models have rapidly closed the gap on moderately difficult reasoning tasks, the hardest reasoning problems \(those requiring either massive scale or capabilities beyond current architectures\) remain a significant frontier\. The presence of 134 level 5 problems \(16\.5% of the reasoning dataset\) that remain 95% unsolved indicates substantial headroom for future model development in complex reasoning domains\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/figures/nr_difficulty_analysis.png)Figure 7:Reasoning Task Performance by Difficulty Level and Year\. Model success rates on across five difficulty levels and three model generations\. The benchmark shows a three\-tier saturation pattern: near\-complete \(98\-99% on Levels 1\-2\), approaching saturation \(85\-92% on Levels 3\-4\), and wide\-open frontier \(51% on Level 5\)\.
### E\.3How much has intelligence efficiency changed over time?

Figure[8](https://arxiv.org/html/2511.07885#A5.F8)presents results across single\-turn chat and reasoning queries, evaluating intelligence efficiency across model\-hardware pairs from April 2024 through August 2025\. We measure both perplexity \(left panel\) and accuracy \(right panel\) normalized by energy consumption in joules per query, tracking nine distinct model families \(Llama,Phi,Gemma,Mistral,Falcon,DeepSeek,Qwen, andGPT\-OSS\) deployed on various GPU configurations including NVIDIA A100 \(40GB/80GB PCIe/SXM\), H100 \(80GB SXM\), H200 \(141GB HBM3e\), and L40S \(48GB\) accelerators\. Energy measurements capture end\-to\-end inference costs\. The temporal snapshots at April 2024, August 2024, and August 2025 enable direct comparison of efficiency trajectories, revealing how successive generations of models and hardware migrate from suboptimal regions \(high energy, low performance\) toward optimal regions \(low energy, high performance\) as indicated by the shaded quadrants in each panel\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/perplexity_and_accuracy_trends_image.png)Figure 8:Perplexity and Accuracy per Joule Trends acrossWildchatandNaturalReasoning\.
### E\.4What efficiency gains can effective query routing deliver?

We compute cost per query using pricing available on OpenRouterAI\[[56](https://arxiv.org/html/2511.07885#bib.bib56)\]\. Table[10](https://arxiv.org/html/2511.07885#A5.T10)lists the token pricing used\.

ModelInput Cost \(USD / 1M tokens\)Output Cost \(USD / 1M tokens\)Qwen3\-4B0\.0000\.000Qwen3\-8B0\.0350\.138Qwen3\-14B0\.0600\.124Qwen3\-32B0\.1000\.450Qwen3\-235B0\.2200\.880GPT\-OSS\-20B0\.030\.14GPT\-OSS\-120B0\.150\.60Table 10:Model pricing from OpenRouterAI\.Costs are in USD per 1M tokens for input and output, as of August 2025\.
### E\.5How does model precision affect performance and efficiency?

Model quantization \(reducing numerical precision fromFp16toFp8orFp4\) decreases memory requirements and energy consumption during inference while introducing approximation error that may degrade model accuracy\. To quantify this tradeoff, we evaluate eight open\-source models from theQwen3andGemmafamilies across three precision levels:Fp16\(full precision\),Fp8\(8\-bit floating point\), andFp4\(4\-bit floating point\)\. ForFp8andFp4we use the vendor\-published quantized checkpoints where available, falling back to native PyTorch FP8 / NVIDIA TransformerEngine FP4 conversions when no published checkpoint exists; energy measurements are taken onNVIDIA B200\(cloud\) and replicated onApple M4 Max\(local\) using the same telemetry harness described in App\.[B\.1](https://arxiv.org/html/2511.07885#A2.SS1.SSS0.Px6), with per\-query results averaged across hardware platforms\. For each model\-precision pair, we measure accuracy on three reasoning\-focused datasets:NaturalReasoning\(N=10,000N=10,000\), SuperGPQA \(N=10,000N=10,000\), and MMLU Pro \(N=10,000N=10,000\)\.

Figure[9](https://arxiv.org/html/2511.07885#A5.F9)shows that quantization fromFp16toFp4yields energy reductions of3−3\.5×3\-3\.5\\timeswith accuracy degradation of approximately 2\.5 percentage points per precision step across all models and datasets\. For example, on SuperGPQA,Qwen3\-14Bachieves 54\.5% \(Fp16\), 52\.0% \(Fp8\), and 49\.0% \(Fp4\): a total degradation of 5\.5 percentage points despite a3\.23×3\.23\\timesreduction in energy consumption\. Larger models maintain their relative performance advantage even at lower precision:Qwen3\-14BatFp4\(49\.0% accuracy\) outperformsQwen3\-4BatFp16\(48\.5% accuracy\) on SuperGPQA, indicating that model scale matters more than precision for reasoning tasks\. These results demonstrate thatFp8andFp4quantization enable practical deployment of local models with predictable performance tradeoffs, allowing system designers to select precision levels based on application\-specific requirements while capturing most of the energy savings identified in Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3)\.

![Refer to caption](https://arxiv.org/html/2511.07885v5/model_precision_vs_accuracy_image.png)Figure 9:Minimal Accuracy Degradation Shifting from FP16 to FP4 for Open\-Source Local Models: Evaluation across three reasoning datasets \(N=10,000N=10,000each\) shows2−3%2\-3\\%accuracy loss per precision step, demonstrating thatF​P​8FP8/F​P​4FP4quantization enables efficient deployment with acceptable performance tradeoffs\.
### E\.6How do SOTA open\-source LMs compare to the SOTA closed\-source LMs on Chat and Reasoning Queries?

To evaluate the competitiveness of open\-source models for local deployment, we compare the performance of state\-of\-the\-art open\-source models against leading closed\-source models across single\-turn chat and reasoning queries\. We evaluate three closed\-source frontier models \(GPT\-5\-2025\-08\-07,Gemini\-2\.5\-Pro, andClaude\-Sonnet\-4\-5\) against eight open\-source models ranging fromQwen3\-8BtoQwen3\-235B\-A22B, measuring performance onNaturalReasoning, MMLU Pro, and SuperGPQA\. Table[11](https://arxiv.org/html/2511.07885#A5.T11)shows that the best open\-source model \(Qwen3\-235B\-A22B\) achieves 71\.8% average accuracy across benchmarks, trailing the best closed\-source model \(GPT\-5\-2025\-08\-07, 77\.9%\) by 6\.1 percentage points\.

Performance gaps vary substantially by task type, as shown in Table[12](https://arxiv.org/html/2511.07885#A5.T12)\. On MMLU Pro and SuperGPQA, open\-source models nearly match closed\-source performance:Qwen3\-235B\-A22Bachieves 82\.3% versus 87\.4% on MMLU Pro \(5\.1% gap\) and 63\.1% versus 66\.5% on SuperGPQA \(3\.4% gap\)\. However, onNaturalReasoning, the gap widens to 12\.9% \(70\.0% versus 82\.9%\), indicating that closed\-source models maintain a substantial advantage on naturalistic reasoning tasks\. Local models with deployment constraints \(≤20​B\\leq 20Bactive parameters\) face larger gaps: the best local model \(Qwen3\-14B\) trails closed\-source models by 11\.8–13\.2% across benchmarks, with the smallest gap on MMLU Pro \(11\.8%\) and the largest onNaturalReasoning\(13\.2%\)\. These results demonstrate that while open\-source models at frontier scale \(235B parameters\) approach closed\-source performance on knowledge and reasoning benchmarks, practical local deployment using smaller models \(14B parameters\) requires accepting 11–13% accuracy degradation relative to closed\-source alternatives\.

ModelTypeWildChatNaturalReasoningMMLU ProSuperGPQAAveragegpt\-5\-2025\-08\-07Closed81\.9%82\.9%86\.5%64\.4%78\.9%gemini\-2\.5\-proClosed89\.5%77\.9%87\.4%66\.5%80\.3%claude\-sonnet\-4\-5Closed88\.1%76\.9%86\.4%60\.1%77\.9%Qwen3\-235B\-A22B\(Best OSS\)OpenN/A∗70\.0%82\.3%63\.1%71\.8%Qwen3\-32BOpen76\.1%69\.7%77\.9%56\.5%70\.1%gpt\-oss\-120bOpen89\.2%65\.0%78\.3%55\.3%72\.0%Qwen3\-30B\-A3BOpen47\.3%64\.3%76\.9%57\.4%61\.5%Qwen3\-14BOpen48\.9%60\.0%75\.6%56\.2%60\.2%gpt\-oss\-20bOpen77\.3%67\.3%73\.4%48\.9%66\.7%Qwen3\-8BOpen50\.2%57\.9%73\.3%51\.8%58\.3%

Table 11:Model performance comparison across benchmarks\.Scores represent performance onWildchat,NaturalReasoning,MMLU Pro, andSuperGPQAbenchmarks\.∗Qwen3\-235B\-A22B is used as the reference model forWildchatevaluation\.MetricWildChatNaturalReasoningMMLU ProSuperGPQAClosed Best89\.5%82\.9%87\.4%66\.5%Best Closed Modelgemini\-2\.5\-progpt\-5gemini\-2\.5\-progemini\-2\.5\-proOpen Best89\.2%\*70\.0%82\.3%63\.1%Best Open Modelgpt\-oss\-120bQwen3\-235B\-A22BQwen3\-235B\-A22BQwen3\-235B\-A22BGap−0\.3%\-0\.3\\%−12\.9%\-12\.9\\%−5\.1%\-5\.1\\%−3\.4%\-3\.4\\%Local Best \(≤20​B\\leq 20BActive\)89\.2%67\.3%80\.3%50\.5%Best Local Modelgpt\-oss\-120bgpt\-oss\-20bgpt\-oss\-120bQwen3\-14BLocal Gap−0\.3%\-0\.3\\%−5\.6%\-5\.6\\%−7\.1%\-7\.1\\%−16\.0%\-16\.0\\%Table 12:Closed\-Source vs\. Open\-Source Performance Gap by Task\.Comparison between closed\-source, open\-source \(all sizes\), and local \(≤20​B\\leq 20Bactive parameters\) models\.\*Excludes Qwen3\-235B\-A22B \(reference model for WildChat\)\.
### E\.7How does performance on chat and reasoning queries connect to U\.S\. GDP?

To assess the economic relevance of local model performance improvements, we compute GDP\-weighted accuracy for each model by weighting its performance on each economic category by that sector’s contribution to the 2024 U\.S\. GDP of $29\.18 trillion\[[78](https://arxiv.org/html/2511.07885#bib.bib78)\]\. This metric attempts to quantify what proportion of economic value is relevant and addressable by local LMs, given the local model performances across single\-turn chat and reasoning queries\. Figures[11](https://arxiv.org/html/2511.07885#A5.F11)and[10](https://arxiv.org/html/2511.07885#A5.F10)shows that model improvements translate directly into expanded GDP coverage: on SuperGPQA,Qwen3\-235B\-A22Bachieves 59\.2% accuracy covering $9\.3T in relevant GDP \(31\.9% of total U\.S\. GDP\), while on MMLU Pro, it reaches 84\.5% accuracy covering $7\.6T \(26\.0% of total U\.S\. GDP\)\. The strong positive correlation between model accuracy and GDP coverage across both benchmarks demonstrates that scaling model capabilities systematically expands the set of economically valuable tasks that can be automated\.

Figure 10:Open\-Source Local LMs Performance vs\. U\.S\. GDP \- SuperGPQA and MMLU Pro: Model accuracy on SuperGPQA and MMLU Pro benchmarks plotted against relevant GDP in trillions of dollars\. Both benchmarks show continued performance improvements as training compute scales across models from Qwen3B\-4B to Qwen3B\-A22B\-235B\. For our calculations, we compute the weighted sum of an LM’s accuracy on each U\.S\. Labor category vs\. the U\.S\. GDP associated with that category\.Task type substantially affects GDP coverage: chat tasks inWildchatshow the highest coverage withgpt\-oss\-120breaching 89\.2% accuracy and covering $20\.3T \(69\.6% of U\.S\. GDP\), while reasoning tasks inNaturalReasoningshow lower coverage withQwen3\-235B\-A22achieving 69\.3% accuracy but only covering $6\.8T \(23\.3% of U\.S\. GDP\)\. This disparity reveals that current models excel at creative and conversational tasks that dominate economic activity, but struggle with technical reasoning tasks concentrated in specialized sectors like architecture, engineering, and physical sciences\. The gap between chat coverage \(69\.6%\) and reasoning coverage \(23\.3%\) represents both a limitation of open\-source local models and an economic opportunity: improving reasoning capabilities could unlock an additional $13\.5T in GDP\-relevant tasks, suggesting that advances in technical reasoning would have substantial economic impact beyond current model capabilities\. We caution that this analysis treats benchmark accuracy as a direct proxy for the automation of economically valuable labor in each occupation category, which is an idealization: real\-world deployment also depends on integration with existing tools, user trust, regulatory constraints, and the residual fraction of work in each occupation that is genuinely AI\-amenable\. The figures above are best read as upper\-bound estimates of the addressable scope of local\-LM\-eligible tasks rather than predictions of realized economic impact, and our mapping from[Handa et al\. 2025](https://arxiv.org/html/2511.07885#bib.bib27)’s 22 economic categories to U\.S\. Bureau of Economic Analysis GDP\-by\-industry data\[[78](https://arxiv.org/html/2511.07885#bib.bib78)\]is approximate\.

Figure 11:Open\-Source Local LMs Performance vs\. U\.S\. GDP \- WildChat and Natural Reasoning: Model accuracy on WildChat and Natural Reasoning benchmarks plotted against relevant GDP in trillions of dollars\. Both benchmarks show continued performance improvements as training compute scales across models from Qwen3B\-4B to Qwen3B\-A22B\-235B\. For our calculations, we compute the weighted sum of an LM’s accuracy on each U\.S\. Labor category vs\. the U\.S\. GDP associated with that category\.
### E\.8How do local accelerators compare to cloud accelerators in terms of intelligence efficiency?

To understand the efficiency gap between local and cloud accelerators, we conduct a systematic comparison across identical model configurations\. Tables[13](https://arxiv.org/html/2511.07885#A5.T13)and[14](https://arxiv.org/html/2511.07885#A5.T14)present intelligence per watt and intelligence per joule measurements forQwen3andGPT\-OSSmodel families running onApple M4 Max\(local\),NVIDIA B200\(cloud\), andSambaNova SN40L\(cloud\) accelerators\.

Cloud accelerators achieve consistently higher power efficiency\.Table[13](https://arxiv.org/html/2511.07885#A5.T13)reveals that theNVIDIA B200achieves1\.39×1\.39\\timesto1\.40×1\.40\\timeshigher intelligence per watt than theApple M4 Maxacross allQwen3model sizes evaluated \(4B to 32B parameters\)\. ForQwen3\-32B, the B200 achieves\(2\.75±0\.14\)×10−3\(2\.75\\pm 0\.14\)\\times 10^\{\-3\}intelligence per watt compared to\(1\.97±0\.24\)×10−3\(1\.97\\pm 0\.24\)\\times 10^\{\-3\}for the M4 Max\. TheSambaNova SN40Ldemonstrates even higher efficiency on larger models, achieving\(3\.51±0\.43\)×10−3\(3\.51\\pm 0\.43\)\\times 10^\{\-3\}intelligence per watt onQwen3\-32B\(1\.78×1\.78\\timeshigher than the M4 Max and1\.28×1\.28\\timeshigher than the B200\)\.

Qwen3\-4BQwen3\-8BQwen3\-14BQwen3\-32BSuccess Rate49\.3±1\.8%49\.3\\pm 1\.8\\%57\.5±2\.5%57\.5\\pm 2\.5\\%59\.5±1\.4%59\.5\\pm 1\.4\\%69\.5±2\.3%69\.5\\pm 2\.3\\%Apple M4 MaxIntelligenceper Watt\(1\.40±0\.38\)\(1\.40\\pm 0\.38\)×10−3\\times 10^\{\-3\}\(1\.63±0\.20\)\(1\.63\\pm 0\.20\)×10−3\\times 10^\{\-3\}\(1\.69±0\.31\)\(1\.69\\pm 0\.31\)×10−3\\times 10^\{\-3\}\(1\.97±0\.24\)\(1\.97\\pm 0\.24\)×10−3\\times 10^\{\-3\}NVIDIA B200Intelligenceper Watt\(1\.95±0\.14\)\(1\.95\\pm 0\.14\)×10−3\\times 10^\{\-3\}\(2\.27±0\.18\)\(2\.27\\pm 0\.18\)×10−3\\times 10^\{\-3\}\(2\.35±0\.09\)\(2\.35\\pm 0\.09\)×10−3\\times 10^\{\-3\}\(2\.75±0\.14\)\(2\.75\\pm 0\.14\)×10−3\\times 10^\{\-3\}SambaNova SN40LIntelligenceper Watt———\(3\.51±0\.43\)\(3\.51\\pm 0\.43\)×10−3\\times 10^\{\-3\}Table 13:Local accelerators demonstrate lower power efficiency than cloud accelerators: When running the sameQwen3models, theApple M4 Max\(local\) attains1\.40×1\.40\\timeslower intelligence per watt compared to theNVIDIA B200\(cloud\) andSambaNova SN40L\(cloud\), highlighting the efficiency advantage of purpose\-built cloud accelerators over local accelerators\. Values are mean±\\pm1\-sigma sample standard deviation across 3–5 independent measurement runs per cell\.Energy efficiency gaps widen when accounting for latency\.Table[14](https://arxiv.org/html/2511.07885#A5.T14)extends the analysis to intelligence per joule, which captures end\-to\-end efficiency by incorporating both power consumption and generation latency\. The efficiency advantages of cloud accelerators become more pronounced: theNVIDIA B200achieves1\.6×1\.6\\timesto2\.3×2\.3\\timeshigher intelligence per joule than theApple M4 MaxacrossQwen3andGPT\-OSSmodel variants\. ForQwen3\-8B, the B200 achieves\(8\.71±0\.60\)×10−5\(8\.71\\pm 0\.60\)\\times 10^\{\-5\}intelligence per joule versus\(3\.80±0\.40\)×10−5\(3\.80\\pm 0\.40\)\\times 10^\{\-5\}for the M4 Max, a2\.29×2\.29\\timesefficiency advantage\. TheSambaNova SN40Ldemonstrates the largest efficiency gains, achieving6\.5×6\.5\\timesto7\.4×7\.4\\timeshigher intelligence per joule than the M4 Max:\(3\.12±0\.38\)×10−4\(3\.12\\pm 0\.38\)\\times 10^\{\-4\}versus\(4\.23±0\.45\)×10−5\(4\.23\\pm 0\.45\)\\times 10^\{\-5\}forGPT\-OSS\-120B\.

Qwen3\-8BQwen3\-32BGPT\-OSS\-20BGPT\-OSS\-120BApple M4 MaxIntelligenceper Joule\(3\.80±0\.40\)\(3\.80\\pm 0\.40\)×10−5\\times 10^\{\-5\}\(3\.51±0\.38\)\(3\.51\\pm 0\.38\)×10−5\\times 10^\{\-5\}\(4\.38±0\.31\)\(4\.38\\pm 0\.31\)×10−5\\times 10^\{\-5\}\(4\.23±0\.45\)\(4\.23\\pm 0\.45\)×10−5\\times 10^\{\-5\}NVIDIA B200Intelligenceper Joule\(8\.71±0\.60\)\(8\.71\\pm 0\.60\)×10−5\\times 10^\{\-5\}\(5\.91±0\.42\)\(5\.91\\pm 0\.42\)×10−5\\times 10^\{\-5\}\(7\.34±0\.51\)\(7\.34\\pm 0\.51\)×10−5\\times 10^\{\-5\}\(6\.78±0\.47\)\(6\.78\\pm 0\.47\)×10−5\\times 10^\{\-5\}SambaNova SN40LIntelligenceper Joule—\(2\.27±0\.30\)\(2\.27\\pm 0\.30\)×10−4\\times 10^\{\-4\}—\(3\.12±0\.38\)\(3\.12\\pm 0\.38\)×10−4\\times 10^\{\-4\}Table 14:Cloud accelerators demonstrate superior energy efficiency across all models: TheNVIDIA B200\(cloud\) achieves1\.6×1\.6\\timesto2\.3×2\.3\\timeshigher intelligence per joule than theApple M4 Max\(local\), while theSambaNova SN40L\(cloud\) achieves6\.5×6\.5\\timesto7\.4×7\.4\\timeshigher efficiency\. These results highlight the substantial energy efficiency advantage of purpose\-built cloud accelerators over local hardware acrossQwen3 and GPT\-OSSmodel variants\. Values are mean±\\pm1\-sigma sample standard deviation across 3–5 independent measurement runs per cell\.Architectural differences explain the efficiency gap\.The superior efficiency of cloud accelerators stems from purpose\-built hardware optimizations for LLM inference: high\-bandwidth memory \(HBM3e with 4\.8–8 TB/s bandwidth\), dedicated tensor processing units, and optimized memory hierarchies that maximize throughput per watt\. In contrast, local accelerators like theApple M4 Maxemploy unified memory architectures \(546 GB/s bandwidth\) designed to balance diverse workloads \(including CPU, GPU, and NPU tasks\) under thermal and power constraints typical of consumer devices\. The larger per\-joule gaps \(compared to per\-watt gaps\) reflect that cloud accelerators not only consume less power per unit of accuracy but also complete queries faster, compounding their energy advantage through reduced generation latency\.

Local model capabilities are improving rapidly\.Despite the efficiency disadvantage of local accelerators, Figure[12](https://arxiv.org/html/2511.07885#A5.F12)demonstrates that local LM capabilities have improved dramatically from April 2024 to August 2025\. OnWildchat, the win/tie rate of SOTA local models againstQwen3\-235Bincreased from 28\.0% in April 2024 to 78\.2% in August 2025, a2\.8×2\.8\\timesimprovement in just 16 months\. OnNaturalReasoning, local models improved from 48\.7% to 80\.9% accuracy over the same period, representing a 66% relative improvement\. These trends indicate that while local accelerators remain less efficient per query than cloud infrastructure, the expanding capability of local models enables an increasing fraction of queries to be processed locally, avoiding cloud infrastructure entirely\.

Figure 12:Rapid Improvement of Local LMs across Chat and Reasoning Queries: We evaluate the performance of SOTA local models released between April 2024 and August 2025 onWildchatandNaturalReasoning\. OnWildchat\(left\), local models show a win/tie rate of 78\.2% againstQwen3\-235Bas of August 2025, compared to just 28\.0% in April 2024: a 2\.8×\\timesimprovement in 16 months\. OnNaturalReasoning\(right\), local models achieve 80\.9% accuracy by August 2025, up from 48\.7% in April 2024: a 66% relative improvement\.Local accelerator memory capacity is expanding rapidly\.Figure[13](https://arxiv.org/html/2511.07885#A5.F13)tracks the memory capacity of consumer accelerators from 2012 to 2025, revealing a126\.3×126\.3\\timesimprovement over this period\. Local accelerators that offered 10–20 GB in 2020 now provide 128–512 GB through unified memory architectures like Apple Silicon\. This memory expansion has been the primary enabler of local deployment for increasingly capable models: the transition from sub\-20 GB to 200\+ GB memory removes the key constraint that previously forced workloads to cloud infrastructure\. Models with 8–20B active parameters that now handle the majority of inference queries can run efficiently on current\-generation local hardware, with memory capacity continuing to scale at a pace that suggests even larger models will become locally deployable in coming years\.

Figure 13:Increasing GPU Memory of Consumer Accelerators: Memory capacity \(GB\) for local accelerators\. Over the past decade, local hardware has significantly closed the memory gap with cloud\-grade accelerators, particularly since 2020, driven by advances in high bandwidth memory \(HBM\) components and unified memory architectures\.System\-level benefits offset per\-query efficiency disadvantages\.While cloud accelerators demonstrate1\.4×1\.4\\timesto7\.4×7\.4\\timeshigher intelligence efficiency per query, local deployment provides complementary system\-level benefits that offset this disadvantage\. Local inference avoids datacenter infrastructure costs, network latency, and API pricing, while enabling 88\.7% of queries that local models can handle correctly to bypass cloud compute entirely\. As demonstrated in Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3), intelligent routing between local and cloud infrastructure can achieve 60–80% reductions in total energy, compute, and cost compared to cloud\-only deployment, even when local accelerators are individually less efficient\. These findings suggest that the path to efficient AI infrastructure lies not in local accelerators matching cloud efficiency, but in routing systems that leverage the complementary strengths of both paradigms: local processing for the majority of straightforward queries and cloud infrastructure for the minority requiring frontier model capabilities\.

### E\.9How does cloud batching affect intelligence efficiency?

Our main per\-query comparisons \(Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14)\) use batch size11to follow standard local\-inference benchmarking practice\[[28](https://arxiv.org/html/2511.07885#bib.bib28)\]and to isolate intrinsic model\-accelerator efficiency from system\-level scheduling\. Local devices serving a single user typically operate at bs=1 and cannot batch concurrent queries from different users, so for local accelerators bs=1 reflects realistic operating conditions\. Cloud accelerators, however, can amortize idle GPU power across concurrent queries, and bs=1 is therefore conservative for cloud\. To quantify the magnitude of this effect, we run an ablation onNVIDIA B200sweeping batch size from11to6464across three representative models \(Table[15](https://arxiv.org/html/2511.07885#A5.T15)\)\.

Modelbs=1 IPJbs=64 IPJIPJ GainArchitecture\(×10−5\\times 10^\{\-5\}\)\(×10−5\\times 10^\{\-5\}\)Qwen3\-8B8\.92104\.9311\.8×11\.8\\timesDenseQwen3\-14B7\.2280\.6111\.2×11\.2\\timesDenseGPT\-OSS\-120B6\.72132\.2119\.7×19\.7\\timesMoE \(≤20\\leq 20B active\)Table 15:Cloud Batching Ablation on NVIDIA B200: Per\-query intelligence\-per\-joule at bs=1 versus bs=64\. Per\-query energy drops1212–20×20\\timesat bs=64 \(e\.g\.,Qwen3\-8B:6,4496\{,\}449J→548\\to 548J\) while GPU power scales sublinearly \(255255W at bs=1 to883883W at bs=64\)\. This confirms that bs=1 is conservative for cloud accelerators\. Local accelerators \(M4 Max, iPhone 16 Pro\) serve a single user and cannot batch, so local IPW results are unchanged\.The routing simulation in Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3)accounts for this by operating the cloud baseline at bs=1616, so the reported6060–80%80\\%savings are computed against a batched cloud baseline rather than a bs=1 baseline\. While absolute IPJ values shift substantially with batch size, relative rankings between model\-hardware pairs remain stable, supporting the use of a fixed batch size for comparative analysis\. Local single\-user deployment cannot batch queries from a single user but can in principle aggregate concurrent queries across multiple on\-device applications; we leave a systematic study of cross\-application local batching to future work\.

### E\.10How sensitive are our findings to the choice of inference framework?

We standardize onvLLMacross all platforms in the main study to enable clean attribution of efficiency gains to model and hardware factors rather than serving\-stack choices\. To test whether this choice biases our findings, we benchmark a representative subset of local models onApple M4 Maxacross three popular inference frameworks:vLLM,SGLang, andllama\.cpp\(Table[16](https://arxiv.org/html/2511.07885#A5.T16)\)\.

ModelvLLMSGLangllama\.cppIPW \(×10−3\\times 10^\{\-3\}\)IPW \(×10−3\\times 10^\{\-3\}\)IPW \(×10−3\\times 10^\{\-3\}\)Qwen3\-4B1\.401\.351\.52Qwen3\-8B1\.631\.581\.71Qwen3\-14B1\.691\.621\.78GPT\-OSS\-120B4\.184\.054\.31Table 16:Framework Sensitivity on Apple M4 Max: Intelligence\-per\-watt across three popular inference frameworks for a representative subset of local models\. Absolute IPW values shift by33–12%12\\%across frameworks, but relative rankings between model\-hardware pairs are preserved across the broader20\+20\+\-model,88\-accelerator evaluation: Kendall’sτ∈\[0\.87,0\.93\]\\tau\\in\[0\.87,0\.93\]and Spearman’sρ∈\[0\.89,0\.94\]\\rho\\in\[0\.89,0\.94\]across all framework pairs\. This validates standardizing onvLLMfor comparative analysis and confirms that our key findings are not an artifact of framework selection\. Models that prioritize absolute throughput on a target stack should benchmark on that stack directly\.
### E\.11Are our findings consistent on smartphone\-class accelerators?

To test whether the local\-AI viability story extends to ultra\-low\-power devices, we evaluate theApple iPhone 16 Pro\(Apple A18 ProSoC with integrated NPU, 8 GB LPDDR5X unified memory,∼12\{\\sim\}12W SoC peak power,3535TOPS NPU\) on a representative subset of our query distribution\. Memory constraints \(8 GB\) require aggressive quantization \(FP8 or FP4\) and limit deployable models to≤14\{\\leq\}14B active parameters; within these constraints we evaluate models across three precision levels \(FP16, FP8, FP4\)\. Table[17](https://arxiv.org/html/2511.07885#A5.T17)reports per\-query measurements\.

ModelPrecisionHardwareAcc\.PowerLat\.IPWIPJ\(%\)\(W\)\(s/q\)\(×10−3\\times 10^\{\-3\}\)\(×10−5\\times 10^\{\-5\}\)Smartphone\-class accelerator \(Apple A18 Pro, iPhone 16 Pro\)Qwen3\-4BFP16A18 Pro42\.5±\\pm2\.012\.0±\\pm0\.492\.5±\\pm9\.411\.8±\\pm0\.738\.3±\\pm3\.9Qwen3\-4BFP8A18 Pro40\.5±\\pm1\.811\.0±\\pm0\.355\.2±\\pm5\.612\.4±\\pm0\.766\.7±\\pm6\.8Qwen3\-4BFP4A18 Pro38\.0±\\pm1\.69\.5±\\pm0\.336\.8±\\pm3\.713\.3±\\pm0\.8108\.7±\\pm11\.0Gemma3\-4BFP4A18 Pro32\.0±\\pm1\.58\.8±\\pm0\.333\.5±\\pm3\.511\.6±\\pm0\.7108\.5±\\pm11\.2Granite\-4\.0\-h\-tinyFP4A18 Pro28\.5±\\pm1\.48\.5±\\pm0\.329\.2±\\pm3\.211\.2±\\pm0\.7114\.7±\\pm12\.0Workstation reference \(Apple M4 Max, same model \+ precision for comparison\)Qwen3\-4BFP4M4 Max44\.5±\\pm1\.7240±\\pm183\.5±\\pm0\.41\.85±\\pm0\.1353\.0±\\pm5\.4Workstation reference \(NVIDIA RTX 6000 Ada, same model \+ precision\)Qwen3\-4BFP4RTX 6000 Ada44\.5±\\pm1\.7220±\\pm152\.5±\\pm0\.32\.02±\\pm0\.1380\.9±\\pm8\.1

Table 17:Smartphone\-Class Accelerator Efficiency: Per\-query measurements on Apple A18 Pro \(iPhone 16 Pro,∼12\{\\sim\}12W SoC peak, 60 GB/s memory bandwidth, 8 GB unified memory\) across three precision levels, with workstation references for the same model at the strongest precision the iPhone supports\. Values are mean±\\pm1\-σ\\sigmastandard deviation across 3–5 independent measurement runs\. Accuracy is averaged across the same evaluation subset used in App\.[E](https://arxiv.org/html/2511.07885#A5); latency is wall\-clock per query atbs=1\\text\{bs\}\{=\}1\. The IPW ratio between A18 Pro and workstation GPUs on the same model anchors the∼7×\{\\sim\}7\\timesper\-watt advantage of smartphone\-class accelerators \(13\.3/1\.85=7\.2×13\.3/1\.85=7\.2\\timesvs\. M4 Max;13\.3/2\.02=6\.6×13\.3/2\.02=6\.6\\timesvs\. RTX 6000 Ada\); the IPJ comparison shows the gap widens slightly per joule against M4 Max \(108\.7108\.7vs\.53\.0×10−553\.0\\times 10^\{\-5\}\) but narrows against RTX 6000 Ada \(108\.7108\.7vs\.80\.9×10−580\.9\\times 10^\{\-5\}\), reflecting that the A18 Pro’s99–26×26\\timeshigher latency partially offsets its2020–25×25\\timeslower power\. Modest accuracy degradation on A18 Pro \(∼\\sim6\.5pp on Qwen3\-4B FP4\) reflects CoreML INT4 group quantization plus KV\-cache pressure under 8 GB unified memory\.We find three patterns\. First, smartphone\-class accelerators are remarkably efficient on a per\-watt basis: across precision levels, the iPhone 16 Pro achieves11\.211\.2–13\.3×10−313\.3\\times 10^\{\-3\}accuracy per watt, approximately7×7\\timeshigher IPW than workstation GPUs running the same model \(Table[17](https://arxiv.org/html/2511.07885#A5.T17)\), due to the∼12\{\\sim\}12W SoC envelope versus220220–240240W for workstation GPUs\. Second, this efficiency is conditional on the model fitting within the device’s memory and thermal budget; FP4 quantization is essentially required and FP8 is the practical ceiling for sustained interactive use, with a modest accuracy cost \(∼\\sim6\.5 pp degradation on Qwen3\-4B FP4 vs\. workstation FP4\)\. Third, the per\-watt advantage does not translate uniformly to per\-joule efficiency: the A18 Pro’s99–26×26\\timeshigher per\-query latency \(reflecting its6060GB/s memory bandwidth versus546546GB/s on M4 Max\) partially offsets its lower power draw, so the IPJ gap narrows substantially or inverts depending on the workstation comparator\.

These results suggest that ultra\-low\-power mobile NPUs are viable routing targets for the lightest queries, extending the local\-inference paradigm from workstations and laptops to smartphones in active use\. Concretely, a routing tier that dispatches very\-low\-difficulty queries to on\-device NPUs could further reduce platform\-scale energy beyond the 60–80% reductions reported in Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3), though we leave a full characterization of mobile\-tier routing to future work\.

### E\.12Do our findings generalize to multi\-turn agentic workloads?

Our main study focuses on single\-turn chat and reasoning queries because they constitute the largest share of real\-world LLM traffic\[[10](https://arxiv.org/html/2511.07885#bib.bib10)\], but a substantial and growing fraction of usage involves multi\-turn interactions, tool use, and agentic workflows\. To test whether our local\-versus\-cloud efficiency patterns generalize, we evaluate two multi\-turn benchmarks: GAIA \(165165general\-assistant queries with tool use, run with the OpenHands agent\) and TerminalBenchV2 \(TBv2;8080terminal\-task queries, run with the Terminus 2 agent\) on local \(Apple M4 Max, 128 GB unified memory\) and cloud \(8×8\\timesNVIDIA H10080 GB SXM5 node\) hardware\. Table[18](https://arxiv.org/html/2511.07885#A5.T18)reports per\-query measurements\.

BenchmarkModelHardwareAcc\.PowerLat\.EnergyIPWIPJ\(%\)\(W\)\(s/q\)\(kJ/q\)\(×10−3\\times 10^\{\-3\}\)\(×10−5\\times 10^\{\-5\}\)GAIA \(165 multi\-turn general\-assistant queries with tool use\)GAIAMiniMax\-M2\.58×\\timesH10016\.4±\\pm2\.91558±\\pm954\.69±\\pm0\.417\.31±\\pm0\.620\.105±\\pm0\.0202\.24±\\pm0\.39GAIAQwen3\-235B8×\\timesH1005\.5±\\pm1\.81595±\\pm880\.74±\\pm0\.091\.18±\\pm0\.130\.034±\\pm0\.0124\.66±\\pm1\.55GAIAQwen3\-30B8×\\timesH1008\.6±\\pm2\.2822±\\pm641\.22±\\pm0\.141\.00±\\pm0\.110\.105±\\pm0\.0278\.60±\\pm2\.20GAIAMiniMax\-M2\.5M4 Max14\.2±\\pm2\.7305±\\pm2451\.6±\\pm5\.415\.74±\\pm1\.690\.466±\\pm0\.0920\.90±\\pm0\.18GAIAQwen3\-30BM4 Max6\.4±\\pm1\.9245±\\pm1913\.4±\\pm1\.53\.28±\\pm0\.390\.261±\\pm0\.0801\.95±\\pm0\.61TerminalBenchV2 \(80 multi\-turn terminal\-task queries\)TBv2MiniMax\-M2\.58×\\timesH10039\.7±\\pm4\.41151±\\pm782\.00±\\pm0\.212\.30±\\pm0\.240\.345±\\pm0\.04117\.30±\\pm1\.94TBv2GPT\-OSS\-120B8×\\timesH10030\.4±\\pm3\.91024±\\pm713\.01±\\pm0\.323\.08±\\pm0\.330\.297±\\pm0\.0419\.87±\\pm1\.31TBv2Kimi\-K2\.58×\\timesH10029\.1±\\pm3\.81251±\\pm822\.88±\\pm0\.293\.60±\\pm0\.360\.233±\\pm0\.0338\.08±\\pm1\.10TBv2MiniMax\-M2\.5M4 Max37\.5±\\pm4\.2295±\\pm2222\.3±\\pm2\.46\.58±\\pm0\.711\.271±\\pm0\.1585\.70±\\pm0\.78TBv2GPT\-OSS\-120BM4 Max28\.2±\\pm3\.7268±\\pm2131\.5±\\pm3\.38\.44±\\pm0\.911\.052±\\pm0\.1653\.34±\\pm0\.50

Table 18:Multi\-Turn Agentic Workloads Generalize Single\-Turn Patterns: Per\-query measurements on GAIA \(165 general\-assistant queries with tool use, OpenHands agent\) and TerminalBenchV2 \(80 terminal\-task queries, Terminus 2 agent\) for self\-hosted open\-source models on local \(Apple M4 Max, 128 GB unified memory\) and cloud \(8×\\timesNVIDIA H100 80 GB SXM5 node\) hardware\. Values are mean±\\pm1\-σ\\sigmastandard deviation across 3 independent runs per cell; accuracy uncertainty uses the binomial approximation over benchmark size\. M4 Max models use GGUF quantization via Unsloth to fit within unified memory; cloud models run at native precision via vLLM\. The qualitative patterns from single\-turn evaluation persist: cloud hardware achieves2\.42\.4–3\.0×3\.0\\timeshigher per\-joule efficiency on identical models \(e\.g\., MiniMax\-M2\.5 on TBv2:17\.3017\.30vs\.5\.70×10−55\.70\\times 10^\{\-5\}\), while M4 Max achieves3\.73\.7–3\.8×3\.8\\timeshigher per\-watt efficiency due to its3\.53\.5–5\.3×5\.3\\timeslower power envelope, with only∼\\sim2\.2pp accuracy degradation across both benchmarks\. The 12–15×\\timeshigher M4 Max latency reflects the same memory\-bandwidth bottleneck observed in single\-turn evaluation, with multi\-turn workloads amplifying absolute differences but preserving relative rankings\.The qualitative patterns from single\-turn evaluation persist \(Table[18](https://arxiv.org/html/2511.07885#A5.T18)\): on identical models, cloud hardware achieves2\.42\.4–3\.0×3\.0\\timeshigher per\-joule efficiency than local \(e\.g\., MiniMax\-M2\.5 on TBv2:17\.3017\.30vs\.5\.70×10−55\.70\\times 10^\{\-5\}IPJ\), consistent with Tables[13](https://arxiv.org/html/2511.07885#A5.T13)–[14](https://arxiv.org/html/2511.07885#A5.T14), while local hardware achieves3\.73\.7–3\.8×3\.8\\timeshigher per\-watt efficiency due to its3\.53\.5–5\.3×5\.3\\timeslower power envelope\. Per\-query accuracy on the strongest local model is within∼2\.2\{\\sim\}2\.2pp of the cloud configuration on both benchmarks \(MiniMax\-M2\.5 on TBv2:37\.537\.5vs\.39\.7%39\.7\\%; on GAIA:14\.214\.2vs\.16\.4%16\.4\\%\), suggesting that the routing\-based savings reported in Section[4\.3](https://arxiv.org/html/2511.07885#S4.SS3)carry over qualitatively to agentic workloads\. Absolute savings rates may shift because multi\-turn workloads have substantially longer effective context lengths and more tool\-call overhead than single\-turn chat \(visible here as per\-query energies in thek​JkJrather thanJJrange\)\. We caution that245245multi\-turn queries is a small evaluation and view this experiment as a sanity check rather than a definitive characterization; full multi\-turn IPW characterization across more diverse agent stacks is an important direction for future work\.