Kimi K3 is a 2.8-trillion-parameter multimodal MoE (16 of 896 experts active per token) with a context window of up to 1M tokens. Its architecture departs from a standard transformer in several ways. Each changes what a serving engine has to do. This post looks at Kimi K3’s architecture and how vLLM serves it.

The Inference Engine Guide for K3 Deployment

We’re thrilled to announce Efficient Day-0 vLLM Support for Kimi K3, one of the most powerful open-weight models ever released. Last week, we previewed the production-scale integration work; today, Moonshot AI’s weights are public and the support is live.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. The challenge of K3 is making KDA, MXFP4 MoE execution, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a serving engine you can actually run. This post is the practical guide.

TL;DR

Kimi K3 is a 2.8-trillion-parameter multimodal MoE (16 of 896 experts active per token) with a context window of up to 1M tokens, built on Kimi Delta Attention, Attention Residuals, the Stable LatentMoE design, and native MXFP4 (4-bit) weights.

🔷Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s (a 3.14× improvement) with DSpark, on NVIDIA GB300 NVL72.

🔷Broad feature support at launch: speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, on NVIDIA (B200, B300, GB200, GB300) and AMD ROCm.

🔷vLLM supports DSpark, a state-of-the-art speculative decoding algorithm for Kimi K3, trained with vLLM and open-sourced by Inferact.

🔷Serving K3’s hybrid recurrent-plus-attention design required new vLLM core infrastructure: prefix caching over recurrent (KDA) state — a change that now benefits every hybrid linear model.

Kimi K3’s architecture, and how vLLM serves it

K3 departs from a standard transformer in four ways, and each one changes what a serving engine has to do.

Kimi K3 architecture innovations, from original release blog post https://www.kimi.com/blog/kimi-k3

Kimi K3 architecture innovations, from original release blog post https://www.kimi.com/blog/kimi-k3

Kimi Delta Attention: a hybrid recurrent + full-attention stack. Most of K3’s layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable.

A single hybrid KV-cache manager holds both kinds of memory under one scheduler: paged KV blocks for full-attention layers, and compact recurrent-state blocks for KDA layers. The hardest part is prefix caching over recurrent state: a KDA layer has no per-token KV to hash, so vLLM separates the physical state-block size from the prefix-match granularity and reuses state snapshots at block boundaries, so long shared prompts still hit cache.

Kimi K3 interleaves Kimi Delta Attention (linear) layers with periodic full-attention layers; vLLM’s hybrid cache manages recurrent state and paged KV together.

Kimi K3 interleaves Kimi Delta Attention (linear) layers with periodic full-attention layers; vLLM’s hybrid cache manages recurrent state and paged KV together.

Attention Residuals: AttnRes mixes the outputs of previous layers with softmax attention, letting each token’s representation adapt to its content across the depth of the network. vLLM implements a fused CUDA kernel that computes logits, softmax across previous layers, and aggregates in a single launch, folding the residual add and output RMSNorm into the same kernel.

Stable LatentMoE: Each token is routed to 16 of 896 experts. Experts are sharded with expert parallelism; vLLM offers two MoE backends tuned for different topologies: TRT-LLM-Gen for tensor-parallel (TP>1) and MegaMoE for disaggregated/expert-parallel (DEP), plus optional Expert-Parallel Load Balancing (EPLB).

MXFP4 weights and native vision. K3’s weights are quantization-aware trained in 4-bit MXFP4, so low precision is native rather than a post-hoc conversion.

Serving semantics: K3 ships without a Jinja chat template; its tokenizer renders messages through a Python program instead. vLLM implements the same render program in both the Python and Rust frontends, so tool calling, reasoning output, and structured output behave identically across both.

Built for production

Serving a 2.8T hybrid MoE well means being fast for one user, efficient for thousands, and sticky for agents. vLLM ships Kimi K3 ready on all three.

Ultra-low latency: To reach ultra-low latency on a 2.8T model without accuracy loss, speculative decoding is the natural choice. This which is why vLLM supports DSpark from day 0, and why we trained and released our own DSpark speculator for K3. DSpark uses a block-diffusion backbone to generate multiple speculative tokens in one parallel pass, so drafting cost stays flat as the block deepens. We made the draft MLA-native, mirroring K3’s own attention, so draft and target share a similar KV layout.

Kimi K3 DSpark positional acceptance rates across various datasets

Kimi K3 DSpark positional acceptance rates across various datasets

Measured with the same build, prompt, and method: 118 → 370 tok/s per user, a 3.14× improvement on 16 NVIDIA GB300 NVL72 GPUs (111 → 331 tok/s on TP8). The gain tracks how predictable the output is — for coding and other low-entropy tasks, DSpark accepts around 4.73 tokens per step; for high-entropy tasks such as creative writing, around 2.61 — so the speedup is a measurement at standard settings, not a universal constant. Both the draft model and the inference support are open source as of this release.

Large-scale serving (prefill/decode disaggregation): for high-throughput fleets, vLLM serves K3 with expert and data parallelism across nodes, and with prefill/decode (PD) disaggregation, which runs prefill-heavy and decode-heavy work on separate replicas so each is sized for its own bottleneck. The validated topology routes TEP8 prefill → DEP16 decode, with KV/state moved between stages over NIXL.

Prefill/decode disaggregation flow

Prefill/decode disaggregation flow

Agentic serving: Agentic workloads (long multi-turn sessions, tool loops, large shared system prompts) require detailed KV reuse. vLLM’s Mooncake integration lets K3 offload and reuse KV across turns and requests, so a repeated codebase, document, or agent scratchpad doesn’t pay full prefill every turn. Combined with K3’s prefix caching over recurrent state, this keeps long agent sessions responsive as context accumulates.

Performance and benchmarks

Everything below ran through a served OpenAI-compatible endpoint with the release parsers.

🔷GSM8K: 0.976

🔷GPQA-Diamond: 0.939

🔷OCRBench: 0.889

🔷MMMU Pro Vision: 0.818

On reasoning and knowledge, vLLM tracks Kimi K3’s reported quality closely, which means the kernels, parsers, and caches preserve model quality end to end.

Kimi K3 decode throughput on batch size 1, measured on 2 GB300 trays with 8 GB300 GPUs.

Kimi K3 decode throughput on batch size 1, measured on 2 GB300 trays with 8 GB300 GPUs.

One field note: K3 reasons a lot before it answers. When a score comes in low, check for truncated answers before assuming wrong ones, and consider increasing the max_tokens budget generously.

Deployment

Quick start (the 2.8T MoE requires at least one 8× B300 node to run at all):

vllm serve moonshotai/Kimi-K3 \

–tensor-parallel-size 8 \

–trust-remote-code \

–load-format fastsafetensors \

–enable-prefix-caching \

–max-num-seqs 512 –max-model-len 32768 \

–enable-auto-tool-choice –tool-call-parser kimi_k3 \

–reasoning-parser kimi_k3

Enable DSpark speculative decoding:

–speculative-config ‘{“model”:“Inferact/Kimi-K3-DSpark”,“method”:“dspark”,“num_speculative_tokens”:7,“attention_backend”:“FLASHINFER_MLA”,“draft_sample_method”:“probabilistic”,“rejection_sample_method”:“block”}’

Deployment notes, each from a real incident:

🔷–enable-prefix-caching turns prefix caching on, which is not on by default for K3.

🔷 Tool calling: validate on your own traffic; production agents should retry or fall back when tool_calls comes back empty, and consider strict/structured tool calling.

🔷 –moe-backend: deep_gemm_mega_moe for any DEP environment, flashinfer_trtllm for TP>1.

🔷 –all2all-backend: flashinfer_nvlink_one_sided for NVLink, deepep_v2 for RDMA.

FAQ

How many GPUs do I need? At least one 8× B300 (or GB300) node to run at all; most production deployments run multi-node with expert and data parallelism, over RDMA or NVLink.

Does K3 support prefix caching, and is it on by default? It supports prefix caching over both full-attention KV and recurrent KDA state, but it is not on by default — pass –enable-prefix-caching.

Does vLLM support K3 on AMD GPUs? Yes. ROCm support ships at launch, with broader tuning on the roadmap.

Full recipes, kernels, and the complete write-up:

🔷 Model: huggingface.co/moonshotai/Kimi-K3

🔷 DSpark draft model: huggingface.co/Inferact/Kimi-K3-DSpark

🔷 Recipes & Docker images: recipes.vllm.ai/moonshotai/Kimi-K3

🔷Full blog: vllm.ai/blog/2026-07-27-k3

The infrastructure built for K3 now belongs to every model that comes after it.

We can’t wait to see what you serve.