@charles_irl: Tried to squeeze the most important bits about the entire stack for cloud deployment of transformer inference, from app…

Tried to squeeze the most important bits about the entire stack for cloud deployment of transformer inference, from application layer concerns to hardware, debugging, and o11y, into one talk. Had to operate at a very high tok/s!

https://t.co/CFlfGyCSOs https://t.co/B8hysXfUiv

TL;DR: This article covers the complete technical stack for deploying transformer inference in the cloud, from application scenarios and workload definitions, to models, inference engines, hardware, observability and performance optimization, and looks ahead at future trends.

Why Inference Matters

Inference is often seen as the “little brother” of training, but commercially it is the real revenue center. Training is a cost center – you spend money to produce a model, but the model itself is hard to monetize directly. Inference directly serves users and generates revenue. Even when raising venture capital, you need to show revenue progress. Moreover, training itself increasingly relies on inference to generate outputs and interact with the world (e.g., reinforcement learning), and may consume more compute than pre-training. Inference also spans the entire tech stack – applications, linear algebra, GPU template libraries (CUTE), layout algebra, electronics, and even cooling – giving full-stack engineers a broad playground. Finally, inference demand is about to surge, presenting a major opportunity for engineers.

Three Prototypes of Language Model Applications

The engineering constraints of inference services depend on the application type. I categorize language model applications into three types:

Chatbot+

• Examples: ChatGPT, Claude Code
• Characteristics: The other end is a real person, with human reaction times, latency tolerance, and attention span. Not just chat, but also representing users to interact with other computer systems via text.
• Engineering constraints: Low latency (strict TTFT and token interval time), high traffic variability.

Background Agent

• Examples: Devon coding agent, Resolves S, Ramps Inspect, Open Claw
• Characteristics: Users may directly generate many tokens in the prompt, but often don’t wait for the result (e.g., asking an agent to implement a feature and open a PR while in a meeting). Latency constraints are more relaxed – seconds to hours are acceptable.
• Engineering constraints: High latency tolerance, but may involve large output tokens, requiring high throughput or batch processing.

Data Processing

• Examples: Reductto processing platform (indexing emails for Jmail project)
• Characteristics: Processing PDFs, documents, extracting structured information from unstructured data. Downstream is storage (file systems, databases).
• Engineering constraints: High volume, high latency tolerance, but traffic is bursty with large gaps between sparse and dense writes.

Workload Definition: SLA/SLO and Key Metrics

Product developers and application engineers hand off via workload definitions. Service Level Agreements (SLA) or Service Level Objectives (SLO) describe failure rate, performance (speed), and cost. Key metrics include:

• Queries Per Second (QPS): User-controlled, hard to predict, with volatility and seasonality. The higher the ratio of average to peak traffic, the harder the system is to serve.
• Input tokens (prefill phase) and output tokens (decode phase): Output length is determined by the model (when to generate the end token), not deterministic – similar to database query performance varying with data. Can be estimated but not precise.
• Prefix reuse: The proportion of input tokens that have been seen before. Previous computation results (KV cache) can be cached, reducing GPU compute but potentially increasing latency. Especially useful when latency tolerance is high (background agents, data processing).
• Latency budget: Time to First Token (TTFT) and per-output-token time (token interval time). TTFT is the time to return the first token; token interval is the generation speed for subsequent tokens.

Components of an Inference System

Serving inference requires the following components to work together:

Model

• A trained transformer model (e.g., LLaMA, GPT). Needs to be exported to an inference-friendly format (e.g., ONNX, TensorRT, static graph).
• Model choice (architecture, size) directly impacts compute and memory requirements for inference.

Inference Engine

• The software layer that coordinates model inference. Common engines: vLLM, TensorRT-LLM, TGI (Hugging Face), llama.cpp (local deployment), etc.
• The engine handles scheduling, batching (dynamic batching, continuous batching), KV cache management, prefix caching, etc.
• When choosing an engine, consider supported models, hardware, optimization techniques (e.g., FlashAttention, PagedAttention).

Hardware

• Key: GPU (NVIDIA A100/H100/B200, etc.), TPU, other accelerators. Currently GPUs remain the primary inference hardware.
• Hardware affects compute speed (float16/int8/int4 precision) and memory bandwidth (determines KV cache size and decode speed).
• Cooling, power, networking (multi-node communication) are also practical considerations.

Deployment

• Primarily cloud (AWS, GCP, Azure) or on-premises data centers. Need to configure instance types, scaling strategies (horizontal/vertical), load balancing.
• Consider cost: on-demand vs. reserved instances vs. spot instances. Inference engines can cooperate with auto-scaling to handle traffic fluctuations.

Observability

• Logs, metrics (latency, throughput, GPU utilization, cache hit rate), traces.
• Need to be able to identify bottlenecks: compute-bound (GPU compute), memory-bound (KV cache overflow), or network-bound (cross-node communication).

Performance Optimization

• To meet SLOs or reduce costs: lower latency, increase throughput, optimize cost.
• Common techniques:
  • Quantization: From FP16 to INT8/INT4, reduces memory and compute, but may lose precision.
  • Continuous batching: Dynamically combine requests into batches, increasing GPU utilization.
  • KV cache optimization: Prefix caching, PagedAttention, KV cache compression/offloading.
  • Speculative decoding: Use a smaller model to predict tokens, large model verifies, accelerating decoding.
  • Distributed inference: Tensor parallelism, pipeline parallelism, sequence parallelism.
  • Model pruning/distillation: Reduce parameters.

Future Thoughts

• Inference demand will grow rapidly, from proprietary model providers and the open-source community.
• Hardware will continue to evolve: faster memory, larger HBM, dedicated inference chips.
• Engines and optimization techniques will become more automated, lowering the barrier for full-stack engineers.
• Cost optimization will become a core competency, especially for high-frequency, high-latency-tolerant applications.

Source: https://www.youtube.com/watch?v=ZUdIsRZhWXI