Rethinking AI TCO: Why Cost per Token Is the Only Metric That Matters

NVIDIA Blog 04/15/26, 03:00 PM News

Summary

NVIDIA argues that cost per token is the most critical metric for AI Total Cost of Ownership, surpassing traditional measures like FLOPS per dollar, to better reflect real-world inference efficiency and profitability.

<div id="bsf_rt_marker"></div>Traditional data centers only stored, retrieved and processed data. In the generative and agentic AI era, these facilities have evolved into AI token factories. With AI inference becoming their primary workload, their primary output is intelligence manufactured in the form of tokens. This transformation demands a corresponding shift in how the economics of AI infrastructure, including total cost of ownership (TCO), is assessed. Enterprises evaluating AI infrastructure still too often focus on peak chip specifications, compute cost or floating point operations per second for every dollar spent, aka FLOPS per dollar. The distinction that matters is this: <ul> <li>Compute cost is what enterprises pay for AI infrastructure, whether rented from cloud providers or owned on premises.</li> <li>FLOPS per dollar is how much raw computing power an enterprise gets for every dollar spent, but raw compute and real-world token output are not the same thing. </li> <li>Cost per token is an enterprise’s all-in cost to produce each delivered token, usually represented as cost per million tokens.</li> </ul> The first two are merely input metrics. Optimizing for inputs while the business runs on output is a fundamental mismatch. Cost per token determines whether enterprises can profitably scale AI. It’s the one TCO metric that directly accounts for hardware performance, software optimization, ecosystem support and real-world utilization — and NVIDIA delivers the lowest cost per token in the industry. <h2>What Are the Factors That Lower Token Cost?</h2> Understanding how to optimize token cost requires looking at the equation for calculating cost per million tokens. <img loading="lazy" decoding="async" class="alignnone wp-image-92324 size-full" src="https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-scaled.png" alt="An equation describing how to calculate cost per million tokens. Cost per million tokens = [cost per GPU per hour / (tokens per GPU per second x 60 seconds x 60 minutes) ] x 1 million." width="2048" height="1152" srcset="https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-scaled.png 2048w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-960x540.png 960w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-1680x945.png 1680w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-1280x720.png 1280w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-1536x864.png 1536w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-1290x725.png 1290w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-630x354.png 630w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-300x169.png 300w, https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-400x225.png 400w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /> In this equation, many enterprises evaluating AI infrastructure focus on the numerator: the cost per GPU per hour. For cloud deployments, this is the hourly rate paid to a cloud provider; for on-premises deployments, it’s the effective hourly cost derived from amortizing owned infrastructure. The real key to reducing token cost, however, lies in the denominator: maximizing the delivered token output. That denominator carries two business implications. <ul> <li>Minimize token cost: When this increase in token output is reflected through the cost equation, it drives down cost per token, which is what grows the profit margin on every interaction served.</li> <li>Maximize revenue: More tokens delivered per second also translates to more tokens per megawatt, which means more intelligence to use in AI-powered products and services, generating more revenue from the same infrastructure investment.</li> </ul> So focusing only on the numerator means missing what drives the denominator. Think of it as an “inference iceberg”: The numerator sits above the surface, visible and easy to compare. The denominator is everything beneath the surface, which represents key factors that determine real-world token output. Accurately evaluating AI infrastructure starts with asking what lies beneath. <img loading="lazy" decoding="async" class="alignnone wp-image-92321 size-full" src="https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-scaled.jpg" alt="Image describing the "inference iceberg." The top of the iceberg is characterized by peak chip specifications such as FLOPS and high-bandwidth memory (cost per GPU per hour, FLOPS per dollar). The bottom of the iceberg is characterized by extreme codesign across compute, networking, software, memory, storage, software and ecosystem (cost per token, tokens per watt)." width="2048" height="1152" srcset="https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-scaled.jpg 2048w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-960x540.jpg 960w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-1680x945.jpg 1680w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-1280x720.jpg 1280w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-1536x864.jpg 1536w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-1290x725.jpg 1290w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-630x354.jpg 630w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-300x169.jpg 300w, https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-400x225.jpg 400w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /> <ul> <li aria-level="1">Surface-level inquiry: <ul> <li>What is the cost per GPU hour?</li> <li>What are the peak petaflops and high-bandwidth memory capacity?</li> <li>What are the FLOPS per dollar?</li> </ul> </li> <li aria-level="1">In-depth cost analysis: <ul> <li>What is the <a href="https://blogs.nvidia.com/blog/inference-open-source-models-blackwell-reduce-cost-per-token/">cost per million tokens</a>? Specifically, what is the cost per million tokens for large-scale mixture-of-experts (MoE) reasoning models, which represent the most widely deployed type of AI models?</li> <li>What is the delivered <a target="_blank" href="https://developer.nvidia.com/blog/scaling-token-factory-revenue-and-ai-efficiency-by-maximizing-performance-per-watt/">token output per megawatt</a>? For on-premises deployments especially, where capital commitment to land, power and infrastructure is substantial, maximizing intelligence produced per megawatt is critical.</li> <li>Can the <a href="https://blogs.nvidia.com/blog/mixture-of-experts-frontier-models/">scale-up interconnect handle the “all-to-all” traffic of MoE</a> models?</li> <li>Is <a target="_blank" href="https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/">FP4 precision supported</a>? Can the inference stack make use of FP4 while maintaining high accuracy?</li> <li>Does the inference runtime support <a target="_blank" href="https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/">speculative decoding or multi-token prediction</a> to increase user interactivity?</li> <li>Does the serving layer support <a target="_blank" href="https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/">disaggregated serving, KV-aware routing, KV-cache offloading</a> and other optimizations?</li> <li>Does the platform support the unique workload requirements of agentic AI — including ultralow latency, high throughput and large input sequence lengths?</li> <li>Does the platform support the full lifecycle, from training and post-training to high-scale inference, across all model architectures, to ensure infrastructure fungibility and high utilization?</li> </ul> </li> </ul> Every one of these algorithmic, hardware and software optimizations must be active and integrated, or the denominator collapses. A “cheaper” GPU that delivers significantly fewer tokens per second results in a much higher cost per token. AI infrastructure that gets it right across the full stack ensures that every optimization enhances the others. <h2>Why Does Cost per Token Matter Much More Than FLOPS per Dollar?</h2> The following data for the DeepSeek-R1 AI model demonstrates the difference between theoretical and actual business outcomes. Looking at compute cost alone, the NVIDIA Blackwell platform appears to cost roughly 2x more than NVIDIA Hopper — but compute cost says nothing about the output that investment buys. An analysis of mere FLOPS per dollar suggests a 2x NVIDIA Blackwell advantage compared with the NVIDIA Hopper architecture. However, the actual outcome is orders of magnitude different: Blackwell delivers more than 50x greater token output per watt than Hopper, resulting in nearly 35x lower cost per million tokens. <table id="tablepress-36" class="tablepress tablepress-id-36"> <thead> <tr class="row-1"> <th class="column-1">Metric</th><th class="column-2">NVIDIA Hopper (HGX H200) </th><th class="column-3">NVIDIA Blackwell (GB300 NVL72) </th><th class="column-4">NVIDIA Blackwell Relative to Hopper</th> </tr> </thead> <tbody class="row-striping row-hover"> <tr class="row-2"> <td class="column-1">Cost per GPU per Hour ($)</td><td class="column-2">$1.41 </td><td class="column-3">$2.65 </td><td class="column-4">2x</td> </tr> <tr class="row-3"> <td class="column-1">FLOP per Dollar (PFLOPS) </td><td class="column-2">2.8</td><td class="column-3">5.6</td><td class="column-4">2x</td> </tr> <tr class="row-4"> <td class="column-1">Tokens per Second per GPU</td><td class="column-2">90</td><td class="column-3">6,000</td><td class="column-4">65x</td> </tr> <tr class="row-5"> <td class="column-1">Tokens per Second per MW</td><td class="column-2">54K</td><td class="column-3">2.8M</td><td class="column-4">50x</td> </tr> <tr class="row-6"> <td class="column-1">Cost per Million Tokens ($)</td><td class="column-2">$4.20 </td><td class="column-3">$0.12 </td><td class="column-4">35x lower</td> </tr> </tbody> </table>  Note: Data is sourced from NVIDIA analysis and the <a target="_blank" href="https://inferencex.semianalysis.com/inference">SemiAnalysis InferenceX v2</a> benchmark. This massive divergence proves NVIDIA Blackwell delivers a massive leap in business value over the earlier Hopper generation that far outpaces any increase in system cost. <h2>How to Choose the Right AI Infrastructure</h2> Comparing AI infrastructure based on compute cost or theoretical FLOPS per dollar isn’t just insufficient; it doesn’t provide an accurate representation of inference economics. As the data demonstrates, an accurate evaluation of AI infrastructure’s revenue potential and profitability requires a shift from input metrics to cost per token and delivered token output. NVIDIA delivers the industry’s lowest token cost and highest token throughput through <a href="https://blogs.nvidia.com/blog/blackwell-ai-inference/">extreme codesign</a> across compute, networking, memory, storage, software and partner technologies. Moreover, the constant optimization of open source inference software such as vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo built on the NVIDIA platform means that on existing NVIDIA infrastructure, token output continues to increase and the cost per token continues to decline long after it’s acquired. Leading cloud providers and NVIDIA cloud partners are already delivering this advantage at scale. Partners such as <a target="_blank" href="https://x.com/NVIDIADC/status/2044514332508082515?s=20">CoreWeave</a>, <a target="_blank" href="https://x.com/NVIDIADC/status/2044514334437437687?s=20">Nebius</a>, <a target="_blank" href="https://x.com/NVIDIADC/status/2044514336303890477?s=20">Nscale</a> and <a target="_blank" href="https://x.com/NVIDIADC/status/2044514338132709762?s=20">Together AI</a> have <a target="_blank" href="https://www.youtube.com/watch?v=jw_o0xr8MWU&t=3982s">deployed NVIDIA Blackwell infrastructure</a> and optimized their stacks to bring enterprises the lowest token cost available today, with the full benefit of NVIDIA’s hardware, software and ecosystem codesign behind every interaction served.

Original Article

View Cached Full Text

Cached at: 05/08/26, 07:57 AM

# Rethinking AI TCO: Why Cost per Token Is the Only Metric That Matters Source: [https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/](https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/) Traditional data centers only stored, retrieved and processed data\. In the generative and agentic AI era, these facilities have evolved into AI token factories\. With AI inference becoming their primary workload, their primary output is intelligence manufactured in the form of tokens\. This transformation demands a corresponding shift in how the economics of AI infrastructure, including total cost of ownership $TCO$, is assessed\. Enterprises evaluating AI infrastructure still too often focus on peak chip specifications, compute cost or floating point operations per second for every dollar spent, aka FLOPS per dollar\. The distinction that matters is this: - **Compute cost**is what enterprises pay for AI infrastructure, whether rented from cloud providers or owned on premises\. - **FLOPS per dollar**is how much raw computing power an enterprise gets for every dollar spent, but raw compute and real\-world token output are not the same thing\. - **Cost per token**is an enterprise’s all\-in cost to produce each delivered token, usually represented as cost per million tokens\. The first two are merely input metrics\. Optimizing for inputs while the business runs on output is a fundamental mismatch\. Cost per token determines whether enterprises can profitably scale AI\. It’s the one TCO metric that directly accounts for hardware performance, software optimization, ecosystem support and real\-world utilization — and NVIDIA delivers the lowest cost per token in the industry\. ## **What Are the Factors That Lower Token Cost?** Understanding how to optimize token cost requires looking at the equation for calculating cost per million tokens\. ![An equation describing how to calculate cost per million tokens. Cost per million tokens = [cost per GPU per hour / (tokens per GPU per second x 60 seconds x 60 minutes) ] x 1 million.](https://blogs.nvidia.com/wp-content/uploads/2026/04/inference-equation-token-5115300-scaled.png) In this equation, many enterprises evaluating AI infrastructure focus on the numerator: the cost per GPU per hour\. For cloud deployments, this is the hourly rate paid to a cloud provider; for on\-premises deployments, it’s the effective hourly cost derived from amortizing owned infrastructure\. The real key to reducing token cost, however, lies in the denominator: maximizing the delivered token output\. That denominator carries two business implications\. - **Minimize token cost**: When this increase in token output is reflected through the cost equation, it drives down cost per token, which is what grows the profit margin on every interaction served\. - **Maximize revenue**: More tokens delivered per second also translates to more tokens per megawatt, which means more intelligence to use in AI\-powered products and services, generating more revenue from the same infrastructure investment\. So focusing only on the numerator means missing what drives the denominator\. Think of it as an “inference iceberg”: The numerator sits above the surface, visible and easy to compare\. The denominator is everything beneath the surface, which represents key factors that determine real\-world token output\. Accurately evaluating AI infrastructure starts with asking what lies beneath\. ![Image describing the "inference iceberg." The top of the iceberg is characterized by peak chip specifications such as FLOPS and high-bandwidth memory (cost per GPU per hour, FLOPS per dollar). The bottom of the iceberg is characterized by extreme codesign across compute, networking, software, memory, storage, software and ecosystem (cost per token, tokens per watt).](https://blogs.nvidia.com/wp-content/uploads/2026/04/Inference-Iceberg-5115325_004-1-scaled.jpg) - **Surface\-level inquiry:**- *What is the cost per GPU hour?* - *What are the peak petaflops and high\-bandwidth memory capacity?* - *What are the FLOPS per dollar?* - **In\-depth cost analysis:**- *What is the*[*cost per million tokens*](https://blogs.nvidia.com/blog/inference-open-source-models-blackwell-reduce-cost-per-token/)*? Specifically, what is the cost per million tokens for large\-scale mixture\-of\-experts $MoE$ reasoning models, which represent the most widely deployed type of AI models?* - *What is the delivered*[*token output per megawatt*](https://developer.nvidia.com/blog/scaling-token-factory-revenue-and-ai-efficiency-by-maximizing-performance-per-watt/)*? For on\-premises deployments especially, where capital commitment to land, power and infrastructure is substantial, maximizing intelligence produced per megawatt is critical\.* - *Can the*[*scale\-up interconnect handle the “all\-to\-all” traffic of MoE*](https://blogs.nvidia.com/blog/mixture-of-experts-frontier-models/)*models?* - *Is*[*FP4 precision supported*](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)*? Can the inference stack make use of FP4 while maintaining high accuracy?* - *Does the inference runtime support*[*speculative decoding or multi\-token prediction*](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/)*to increase user interactivity?* - *Does the serving layer support*[*disaggregated serving, KV\-aware routing, KV\-cache offloading*](https://developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/)*and other optimizations?* - *Does the platform support the unique workload requirements of agentic AI — including ultralow latency, high throughput and large input sequence lengths?* - *Does the platform support the full lifecycle, from training and post\-training to high\-scale inference, across all model architectures, to ensure infrastructure fungibility and high utilization?* Every one of these algorithmic, hardware and software optimizations must be active and integrated, or the denominator collapses\. A “cheaper” GPU that delivers significantly fewer tokens per second results in a much higher cost per token\. AI infrastructure that gets it right across the full stack ensures that every optimization enhances the others\. ## **Why Does Cost per Token Matter Much More Than FLOPS per Dollar?** The following data for the DeepSeek\-R1 AI model demonstrates the difference between theoretical and actual business outcomes\. Looking at compute cost alone, the NVIDIA Blackwell platform appears to cost roughly 2x more than NVIDIA Hopper — but compute cost says nothing about the output that investment buys\. An analysis of mere FLOPS per dollar suggests a 2x NVIDIA Blackwell advantage compared with the NVIDIA Hopper architecture\. However, the actual outcome is orders of magnitude different: Blackwell delivers more than 50x greater token output per watt than Hopper, resulting in nearly 35x lower cost per million tokens\. **Metric****NVIDIA Hopper $HGX H200$****NVIDIA Blackwell $GB300 NVL72$****NVIDIA Blackwell Relative to Hopper**Cost per GPU per Hour $$$$1\.41$2\.652xFLOP per Dollar $PFLOPS$2\.85\.62xTokens per Second per GPU906,000**65x**Tokens per Second per MW54K2\.8M**50x**Cost per Million Tokens $$$$4\.20$0\.12**35x lower***Note: Data is sourced from NVIDIA analysis and the*[*SemiAnalysis InferenceX v2*](https://inferencex.semianalysis.com/inference)*benchmark\.* This massive divergence proves NVIDIA Blackwell delivers a massive leap in business value over the earlier Hopper generation that far outpaces any increase in system cost\. ## **How to Choose the Right AI Infrastructure** Comparing AI infrastructure based on compute cost or theoretical FLOPS per dollar isn’t just insufficient; it doesn’t provide an accurate representation of inference economics\. As the data demonstrates, an accurate evaluation of AI infrastructure’s revenue potential and profitability requires a shift from input metrics to cost per token and delivered token output\. NVIDIA delivers the industry’s lowest token cost and highest token throughput through[extreme codesign](https://blogs.nvidia.com/blog/blackwell-ai-inference/)across compute, networking, memory, storage, software and partner technologies\. Moreover, the constant optimization of open source inference software such as vLLM, SGLang, NVIDIA TensorRT\-LLM and NVIDIA Dynamo built on the NVIDIA platform means that on existing NVIDIA infrastructure, token output continues to increase and the cost per token continues to decline long after it’s acquired\. Leading cloud providers and NVIDIA cloud partners are already delivering this advantage at scale\. Partners such as[CoreWeave](https://x.com/NVIDIADC/status/2044514332508082515?s=20),[Nebius](https://x.com/NVIDIADC/status/2044514334437437687?s=20),[Nscale](https://x.com/NVIDIADC/status/2044514336303890477?s=20)and[Together AI](https://x.com/NVIDIADC/status/2044514338132709762?s=20)have[deployed NVIDIA Blackwell infrastructure](https://www.youtube.com/watch?v=jw_o0xr8MWU&t=3982s)and optimized their stacks to bring enterprises the lowest token cost available today, with the full benefit of NVIDIA’s hardware, software and ecosystem codesign behind every interaction served\.

Rethinking AI TCO: Why Cost per Token Is the Only Metric That Matters

Similar Articles

Price per token is the wrong cost metric for agents

@levie: Token costs will become a dominant topic in enterprises going forward with AI. Just got out of a dinner with many Fortu…

Price per 1M tokens is meaningless

How NVIDIA’s Inference Software Stack Powers the Lowest Token Cost

At what point does AI token usage become a business problem?

Submit Feedback

Similar Articles

Price per token is the wrong cost metric for agents

@levie: Token costs will become a dominant topic in enterprises going forward with AI. Just got out of a dinner with many Fortu…

Price per 1M tokens is meaningless

How NVIDIA’s Inference Software Stack Powers the Lowest Token Cost

At what point does AI token usage become a business problem?