Visual Guide to Large Language Model Deployment

Introduction

When self-deploying large models, the key metrics you truly need to care about can be summarized in one sentence: Is the first token fast enough (TTFT)? Are the subsequent tokens generated smoothly (TPOT/ITL)? Does throughput degrade under high concurrency? Does the VRAM have enough capacity with sufficient buffer (VRAM + KV cache)? What is the actual cost per million tokens?

Who should read this article:

• Understand the real VRAM and bandwidth requirements for different parameter sizes to avoid purchasing misconfigured hardware
• Select suitable models and quantized versions for RAG, long document, and Agent business scenarios
• Troubleshoot high-concurrency throughput bottlenecks, stuttering token generation, and severe slowdowns with long contexts
• Identify issues like TTFT spikes, P99 jitter, VRAM over-provisioning, and OOM-related freezes
• Calculate the true TCO per million tokens to evaluate the breakeven point between self-hosting and commercial APIs

Reading Guide:

• If you’re preparing to purchase hardware or evaluate costs: Section 4 (VRAM Calculation), Section 6 (Cost Accounting), and Section 8 (Hardware Selection Comparison)
• If you’re tuning performance or troubleshooting throughput bottlenecks: Sections 2/3 (Latency & Throughput), Section 7 (Advanced Optimization), and Section 9 (Real-World Measurement Pitfalls)
• If you’re making technology choices for specific business scenarios (RAG / Agent / Conversation), building test toolchains and monitoring systems: Section 10 (Common Measurement & Monitoring Tools)
• If your model service is about to go into production: Section 5 (Service Quality & Probes)

1. Looking at Real-World Examples

Let’s take the Ling-3.0-Flash, newly released in late July, as an example. This is a MoE architecture model. Unlike dense models like Qwen 3.8 27b, a key characteristic of MoE models is that the total parameters differ from the activated parameters. It has 124B total parameters but only activates about 5.1B per token, positioning it as a cost-effective, fast execution layer for production-grade Agents, with a price that is only a fraction of Qwen 3.8 27b.

In actual deployment, changing the workload or context length for the same model can result in throughput and latency differing by several times. This section uses complete measurements from a real server to translate each metric into concrete numbers.

Test Environment & Subject

• Hardware Platform: NVIDIA DGX Spark (GB10 Grace-Blackwell), 121 GB unified memory (CPU/GPU share the same memory space, no additional system memory buffer)
• Model Under Test: Ling-3.0-flash-INT4 (124B MoE / 5.1B activated, KDA+MLA hybrid attention)
• Inference Engine: vLLM (with CUDA Graphs and MTP speculative decoding enabled), –max-model-len 16384
• Measurement Basis: Consistently count using usage.completion_tokens from the API response

First encountering terms like TTFT, single-stream Decode speed, high-concurrency aggregate throughput, VRAM, and cold starts can be confusing. The good news is that after reading this article, you will gain a comprehensive understanding of edge model deployment.

2. Latency Metrics (Most Directly Perceived by Users)

1. Prefill Time (Prefill Duration) Large model inference is divided into two phases:

• Prefill: Processes your entire input prompt at once, performing a “forward scan” for attention calculations. This phase is computation-intensive — GPU compute is fully utilized.
• Decode: Generates the response token by token. This phase is memory-access intensive — the GPU waits for data to be moved from VRAM, and compute utilization is often only partial.

Prefill Time is the time taken from when the model reads the entire prompt to when it starts outputting the first token. It directly influences TTFT, so the two are often mentioned together, but they have different definitions (see below).

2. TTFT (Time to First Token, First Token Latency)

Definition: The time from when a request is sent to the server until the first output token is received.

This is the most critical latency metric for interactive dialogue scenarios, directly corresponding to “how long after I ask a question until I see the first character appear.” A 70B model on a single H100 GPU typically has a TTFT of a few hundred milliseconds to two seconds for short prompts; it increases significantly with long prompts (tens of thousands of words).

Note: TTFT = Prefill Time + Queue Wait Time. If the system is busy and your request has to queue, a large part of the TTFT is actually “queuing” rather than “slow model computation.”

3. TPOT / ITL (Inter-Token Latency, i.e., “Decoding Latency”) After the first two words, characters are generated one by one. There are two common metrics for this “character generation speed”:

• TPOT (Time Per Output Token): The average time interval between consecutive generated tokens. This is the steady-state token generation speed.
• ITL (Inter-Token Latency): Often used interchangeably with TPOT, focusing on the latency between individual tokens.

Industry documents and benchmarks (like MLPerf Inference, vLLM/SGLang communities) primarily use TTFT + TPOT. They describe “steady-state token generation speed,” ignoring the first token and input length. On a single H100 running an INT4 quantized 70B model, a common range is 50~120 tokens/second (about 8~20 ms per token), varying with context length.

Important Rule: The decode phase is limited by VRAM bandwidth, so generation speed decreases as context length increases (because the KV cache grows larger, requiring more data to be moved each time). This is the “long-context slowdown” phenomenon essential for self-deployment.

This slowdown happens earlier and is steeper than most people expect. Section 9 has a measured curve: on the same machine and model, when the prompt increases from 300 tokens to 15K, decode speed drops from 39.0 to 18.2 tok/s — halved within the 16K window, and further to 7.1 at 49K, which is 5.6 times slower.

4. End-to-End Latency & Percentiles (P95 / P99)

• End-to-End Latency: The total time from sending a request to receiving the entire response, equal to TTFT + total generation time. More important for tasks like file summarization and code generation.
• P95 / P99 Latency: Don’t just look at the average! Averages mask tail anomalies. P95 is the worst latency of the slowest 5% of requests; P99 is the worst of the slowest 1%. For performance in production environments, always look at P95/P99, not just the mean. If P99 is much higher than P50, it indicates occasional system jitter (GC, queue congestion, OOM retries), which is more dangerous than “good average performance.”

3. Throughput Metrics (How Much Work the System Can Handle)

1. Tokens per Second (TPS) The most common and most easily misunderstood metric. You must distinguish three different bases:

• Input TPS: Tokens processed during the prefill phase.
• Output TPS: Tokens generated during the decode phase (per request).
• System Throughput: Total tokens processed/generated by the entire system per second (aggregate).

On a single H100, the decode throughput upper limit for a 70B INT4 model is around 100+ tokens/s; a small 7B model on a consumer 4090 can reach over 200 tokens/s (because smaller models have a higher compute proportion and are more compute-bound than bandwidth-bound).

2. Requests per Second (QPS) Measures “how many concurrent conversations can be served simultaneously.” In interactive scenarios, one QPS often corresponds to dozens of concurrent conversations. The relationship between QPS and TPS depends on concurrency: 10 concurrent conversations × 50 tok/s each = System 500 tok/s.

3. Effect of Continuous Batching / Chunked Prefill Early inference frameworks used “static batching”: had to wait for all requests in a batch to finish before starting the next, causing the GPU to idle while waiting for users to think, severely degrading throughput. Modern frameworks (vLLM’s continuous batching, SGLang, TGI) insert new requests during token intervals, keeping the GPU busy.

The key indicator to evaluate whether this optimization works is: Under the same VRAM, does system throughput still scale linearly as concurrency increases? Systems without continuous batching see a cliff-like drop in TPS as concurrency rises; those with it can handle dozens or even hundreds of concurrent requests.

But there are two kinds of cliffs that produce identical curves: one where continuous batching isn’t effective, and another where it’s just capped by concurrency slot limits like –max-num-seqs. Section 9.2 places two curves from the same machine side by side — at the same 8 concurrency, one gives 104.9 tok/s with TTFT increasing 24x, the other 167.9 tok/s, the only difference being one parameter. When you see a cliff, first check the slots, then suspect the framework.

4. System Throughput vs. Single-Machine Throughput For multi-card deployments, look at linear scaling efficiency: Is N cards ≈ N× throughput? In practice, communication overhead often reduces this to 70~90%. The metric is Efficiency Ratio = Measured Multi-Card Throughput ÷ (Single-Card Throughput × Number of Cards). Below 0.7 indicates that the parallelism strategy or interconnect (NVLink/PCIe) is the bottleneck.

4. Resource Metrics (Can Your Machine Handle It?)

1. VRAM Usage (Capacity is the First Hurdle) Total VRAM = Memory used by weights + Memory used by KV cache + Runtime fragmentation overhead.

• Rough calculation for weight memory: Parameter count (in billions) × precision bytes. BF16/FP16 uses 2 bytes per parameter, INT4 uses about 0.7 bytes (including some metadata, often calculated as 0.75).
• Common reference ranges:
• Rule of Thumb: After calculating the theoretical value, multiply by a safety factor of 1.2 to reserve space for KV cache and fragmentation. A 70B INT4 model fits on a single card, but slightly higher concurrency can cause OOM due to insufficient KV cache — this explains “why it crashes even though it fits.”

2. KV Cache Calculation Formula (Context Length vs. VRAM Usage) KV cache stores attention calculation results to avoid recomputation — the larger the context window and higher the concurrency, the more VRAM it consumes. The general formula: KV Cache Memory (bytes) = 2 × Number of Layers × Number of KV Heads × Head Dimension × Context Length × Bytes per Element Where 2 represents Key and Value caches. Using Llama-3-70B as an example (80 layers, GQA, 8 KV heads, head_dim=128, BF16 i.e., 2 bytes):

• Per token: 2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 0.31 MB/token
• 4096 length → ~1.3 GB; 128K length → ~40 GB

This means: with long contexts (≥32K) and high concurrency, KV cache can exceed the weight memory itself, becoming the primary VRAM bottleneck. This is also why many deployments use KV cache quantization or offload to CPU for long contexts.

The opposite error is also common and more subtle: pre-allocating the maximum context length × maximum concurrency for the KV pool, while only using a few percent in practice. Section 9.3 is a real test of 24x over-provisioning (pool: 1,098,590 tokens, actual usage ~46,000, GPU KV cache usage 4.2%); after adjusting this parameter, the freed memory led to higher aggregate throughput. –gpu-memory-utilization is often tuned for “fitting,” not “running stably” — it’s worth double-checking when copying configurations.

3. GPU Utilization & VRAM Bandwidth Utilization

• GPU Utilization (%): The proportion of compute units (SMs) being occupied. Even 30% utilization during the decode phase can be normal (because it’s bottlenecked by VRAM bandwidth) — don’t draw conclusions based solely on utilization.
• VRAM Bandwidth Utilization: LLM decoding is typically memory-access intensive; this is the real bottleneck determining generation speed. When comparing hardware, look at bandwidth (GB/s): H100 SXM ~3 TB/s, H200 ~4.8 TB/s, A100 ~2 TB/s, RTX 4090 ~1 TB/s. A few times difference in bandwidth leads to a few times difference in generation speed for the same model.

4. Precision & Quantization Loss

• FP16/BF16: Lossless, best quality, but doubles VRAM usage, pushing the decode phase towards a memory bottleneck.
• INT8: Halves VRAM usage; quality is almost unchanged in most scenarios.
• INT4 (AWQ/GPTQ): Reduces VRAM to 1/4. Usually imperceptible for dialogue tasks, but may incur 1~5 percentage point quality degradation in complex reasoning/long-chain thinking tasks.

Quantization Distortion Metrics: Use authoritative benchmarks (MMLU, GSM8K, HumanEval, MAUSt, etc.) to compare scores before and after quantization. More aggressive quantization leads to greater score drops. Before choosing INT4, always test the performance drop on task types relevant to your business.

5. Service Quality & Reliability Metrics (Truly Important in Production)

1. P95/P99 Latency & Queue Wait Time Even if “pure computation latency” is fast, you must look at queue time — how long a request waits in line before running. This is the core conflict in multi-user, shared-GPU scenarios: a single request is fast, but everyone slows down when there are many users.

Monitoring Dashboard Recommendation: Watch TTFT-P50/P95, TPOT-P50/P95, and Queue Time-P50/P95 separately.

2. Error Rate & OOM Frequency

• OOM (Out of Memory) retries can cause occasional requests to have explosive latency, a common reason for P99 spikes.
• Metrics: Error Rate (including timeouts, OOM, parsing failures), Retry Count, Parsing Error Rate
• /health returning 200 does not mean the engine is generating tokens. Most inference framework health checks probe the API server, but the actual work is done by the engine process behind it — they can die independently. Liveness probes must test one actual generation (send a 1-token request to see if it returns), not just check the port or process.
• But the liveness condition shouldn’t use “request timeout”. In a system that queues, requests are supposed to wait under full load — using timeout for liveness means traffic peaks will restart your service. Liveness should check “is the engine still progressing” (e.g., whether Avg generation throughput in engine logs is 0 for multiple consecutive sampling periods), not whether a single request is fast.

3. Availability / Uptime

• Percentage of time the service is normally available externally (99.9% ≈ less than 44 minutes of downtime per month).
• Use health checks and graceful restarts to avoid an OOM crashing the entire process.

4. Timeout Configuration & Streaming Experience Enable streaming output recommended: return tokens to the user as they are generated, so the user doesn’t wait for the entire segment to be generated, making perceived latency close to TTFT. Set reasonable request timeouts to prevent a single long request from hanging the connection.

6. Cost Metrics (Is the Money Worth It?)

1. Cost per Token The most straightforward comparison metric: Total Cost (hardware + operational) ÷ Total Tokens Processed. Note API prices differ for input vs. output; unify the basis for comparison. Self-deployment typically only breaks even when monthly call volume reaches billions of tokens, depending on your hardware acquisition cost and electricity.

2. Tokens per Dollar & Tokens per kWh

• Used for horizontal comparison of hardware efficiency, especially for choosing which GPU to buy or whether to rent cloud.
• For self-hosting, also factor in power consumption (one H100 SXM consumes 700W, a 4090 ~450W); higher tokens/kWh means more cost-effective long-term.

3. Hardware Depreciation & TCO (Total Cost of Ownership) Self-deployment cost = Hardware purchase amortized (3~4 year depreciation) + Electricity + Data center/network + Operations personnel. Self-hosting is cost-effective for short-term high-frequency calls with idle team GPUs; renting cloud is more flexible for low-frequency or high-peak-volatility scenarios.

7. Metrics Specific to Advanced Optimization Techniques

These are your “feel gauges” during tuning; with optimizations enabled, monitor these data points:

1. Speculative Decoding (Draft Verification) Uses a small draft model to guess a sequence of tokens, which the large model then verifies in one go. Key Metrics:

• Speedup Ratio: Measured throughput ÷ Baseline throughput, typically 1.5~3x (most noticeable when decode is the bottleneck and concurrency is low-to-medium).
• Accept Rate: Proportion of draft tokens accepted by the large model, determining the speedup effect; the closer the draft model is to the target, the higher the accept rate.
• Cost: Requires deploying an extra small model (additional weight memory), and offers limited help for high-concurrency, prefill-dominated scenarios.
• Measurement Pitfall (easier to fall into than the above): With speculative decoding enabled, calculating tok/s based on SSE chunk count systemically underestimates — accepted draft tokens and verifying tokens are packed into the same chunk. Section 9.5 shows an error where actual 41 tok/s was reported as 20.5, which coincidentally matched the “speculative decoding off” reference value, seeming completely credible. The correct method: stream_options: {“include_usage”: true}, read usage.completion_tokens.

2. Prefix Caching (Prompt Caching) Stores the already-computed KV cache of the first few common tokens in the prompt for reuse (e.g., the same system prompt, same long document). Key Metrics:

• Hit Rate: When hit, that portion of Prefill is skipped. In real business (Agent scenarios, RAG fixed documents), this can reach 50%~70%.
• Latency Reduction: When hit, Prefill for that request can drop to 1/x, overall TTFT can decrease by 40%~90%; but when completely missed, overhead slightly increases.
• Note cache has lifecycle management; low hit rate indicates low prefix repetition in your scenario, making this optimization not cost-effective.

3. Quantization Quality Degradation As mentioned, use scores like MMLU/GSM8K/HumanEval to measure quality loss from INT8/INT4, ensuring it’s within acceptable range.

4. Multi-Card Parallel Scaling Efficiency

• When a single card isn’t enough, use DP (Data Parallelism replication) / TP (Tensor Parallelism weight splitting) / PP (Pipeline Parallelism).
• Metric: The aforementioned linear scaling efficiency ratio; for multiple cards, also look at communication overhead proportion (NVLink models better than PCIe). Models 70B and above generally require multiple cards.

5. Long-Context Specific Metrics For long-context deployment, additionally monitor: KV cache proportion (% of total VRAM), long-text decode slowdown ratio (how much TPS drops from 2K vs 32K), and whether KV cache quantization/offload, FlashAttention, PagedAttention, and other VRAM optimization techniques are enabled.

8. Real Hardware Reference Values & Selection Comparison

Self-Deployment Starting Experience:

• 7B and below: A single 24GB consumer card is sufficient; prioritize TPOT and QPS.
• 70B tier: Requires H100/A100 80GB or dual A800/H20; INT4 saves VRAM but requires quality verification.
• Long Documents / Enterprise RAG: Focus on total KV cache size, prefix cache hit rate, and long-context slowdown.
• Multi-User Online Services: Focus on continuous batching, P95/P99, queue time, and OOM.

9. Measured Sample: Real-World Measurement of Ling-3.0-Flash

Previous sections defined metrics and reference ranges. But in actual deployment, changing the workload or context length for the same model can result in throughput and latency differing by several times. This section uses complete measurements from a real server (same as Section 1) to translate each metric into concrete numbers.

Test Environment & Subject (Same as Section 1: NVIDIA DGX Spark, Ling-3.0-flash-INT4, vLLM, –max-model-len 16384)

1. Latency & Throughput: Distinguishing “Computation Time” from “Queue Time”

Core Conclusion: When monitoring alerts on TTFT spikes, first check system concurrency queue and queue time; don’t blindly blame insufficient model compute.

2. Concurrency Scaling: Identifying “False Throughput Cliffs”

• False Cliff: Under seqs 4 configuration, throughput stops growing and latency spikes at 8 concurrency, appearing as “continuous batching failure,” but is actually due to server parameters limiting concurrency slots.
• Scaling Cost: Expanding from 1 to 16 concurrency increases total throughput 6.7x (36.3 → 242.1 tok/s), but single-request generation speed drops from 36.3 to 15.1 tok/s. Single-machine deployment requires balancing “single-stream experience” with “system throughput.”

3. KV Cache VRAM Trap: Over-Provisioning is Counterproductive Setting –gpu-memory-utilization to 0.80 per conventional experience allocated a 21 GiB KV Cache pool (~1.1 million tokens). But when running 4 complex requests, only about 46,000 tokens were used (utilization only 4.2%, VRAM over-provisioned 24 times).

Core Conclusion: Blindly maxing out the KV pool squeezes system runtime buffers. If monitoring shows GPU KV cache usage long-term in single digits, actively lowering the utilization parameter can free up to 12 GB VRAM, and throughput and single-stream speed actually improve.

4. Long-Context Slowdown Curve: Prefill vs. Decode Trends are Opposite In a “Needle In A Haystack” long-text measurement (generating 128 Tokens):

• Prefill remains constant: From 3K to 49K Tokens, Prefill speed stays stable at 2,400+ tok/s (advantage of hybrid linear attention architecture is clear).
• Decode continues to degrade: As context lengthens, the memory-access overhead of moving KV Cache each step increases dramatically, speed already halves around 11K.
• Pitfall Key Point: When evaluating long-text performance, you must place measurement points in the middle of the context window (e.g., 8K~16K); never only measure the start and end, and don’t mistake stable Prefill for stable Decode.

5. Speculative Decoding: Identifying Client-Side Measurement Pitfalls

• Speedup Upper Bound Limited by Accept Length: Measured MTP accept rate is 65.4%, average tokens output per verification is 1.65, so speedup benefit upper bound is ~1.6x.
• Measurement Pitfall: With speculative decoding enabled, multiple accepted tokens are merged into a single SSE Chunk for return. If you simply calculate speed based on chunk count on the client side, you get a false figure of 20.5 tok/s (mistakenly thinking speculative decoding failed); the correct method is reading usage.completion_tokens in the response, actual speed is 41 tok/s.

6. Quality Evaluation: Two Major Blind Spots in Metric Interpretation

• Thinking Chain Truncation Trap: In IFEval evaluation, enabling thinking mode caused scores to plummet from 0.789 to 0.207. Investigation revealed max_tokens was set to 2048, forcing truncation of the thinking chain before completion. Insufficient output limit configuration appears as “low scores” in quality testing, not direct errors.
• Function Calling Must Examine Layered Performance: BFCL-v3 overall score 0.744, but broken down: single-turn calling is 0.887 (sufficient for production), while multi-turn autonomous loop is only 0.438. When building Agent systems, driving should be done by outer workflow orchestration, not fully relying on the model’s long-chain autonomous planning.

10. Common Measurement & Monitoring Tools

Tool Recommendations:

• Inference Engine Built-in Tools
  • vllm bench serve / SGLang benchmarking suite: Quickly test TTFT, TPOT, system throughput, and speculative decoding accept rate.
• Professional Performance Benchmarking Tools
  • NVIDIA GenAI-Perf: Standardized large model benchmarking tool, outputting MLPerf-compliant TTFT/TPOT percentiles and cost accounting per million tokens.
• Quality & Precision Evaluation Frameworks
  • EvalScope / lm-evaluation-harness: Used for benchmark precision regression before/after quantization and parameter tuning (covers IFEval, HumanEval, GSM8K, etc.).
• Hardware & Runtime Monitoring
  • Prometheus + Grafana: Connect to the /metrics endpoint exposed by inference frameworks, continuously monitoring P95/P99 latency, queue time, and VRAM usage.
  • nvidia-smi / gpustat: Real-time observation of GPU compute utilization, VRAM allocation, and power consumption.

Related Links:

• Measured Model: https://huggingface.co/inclusionAI/Ling-3.0-flash
• DGX Spark Reference Recipe: https://github.com/sudoingX/dgx-spark-ling
• lm-evaluation-harness: https://github.com/eleutherai/lm-evaluation-harness
• vllm: https://github.com/vllm-project/vllm

The definitions and reference values in this article are compiled from open-source inference community (vLLM/SGLang/TGI) metric definitions, the MLPerf Inference benchmark system, and current mainstream hardware specifications. All numbers marked “sample” in Section 9 and the self-check checklist come from measurements of Ling-3.0-flash-INT4 on DGX Spark from 2026-08-17 to 08-18; configurations and scripts are reproducible. Specific values will vary with model version, quantization scheme, framework parameters, and environment — actual measurement is the standard.