serving

llm-d 是一个开源、Kubernetes 原生的分布式 LLM 推理栈，支持 vLLM、SGLang 等引擎，可在 GPU、TPU、XPU 等异构硬件上提供生产级服务，并宣称大幅提升吞吐量与降低 TTFT。

DeepSeek open-sources DeepInfer, the high-performance inference framework powering its large-scale production services, featuring optimized kernels, MoE expert parallelism, speculative decoding, batch-invariant inference, and support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Decapybara 是 MegaCapybara LLM 推理服务器的开源 C++20 与 CUDA 重构实现，面向 NVIDIA RTX 5090 Blackwell 硬件，支持 OpenAI 与 Anthropic 兼容 API、分层 KV 缓存及一整套 Blackwell CUDA 内核。

Prime Intellect 今日发布 Prime Inference，提供面向前沿开源模型的低延迟、高可靠的推理服务，涵盖无服务器端点与预留容量，已在 NVIDIA Blackwell GPU 集群上为 GLM-5.3 等模型提供近乎零工具调用错误率和 100% 可用性的生产级部署。

An article explaining static, dynamic, and continuous batching strategies in LLM inference, and why serving LLMs differs from traditional ML inference.

Escha Labs has released a 2-bit quantized version of the Qwen3.8-27B AI model, enabling it to run on a single 24GB consumer GPU with up to 64k context while maintaining performance comparable to FP8 references.

A 10-week, 30-minutes-a-day roadmap for engineers to learn LLM inference serving, covering vLLM, SGLang, load testing, quantization, and optimization techniques to build a reproducible benchmark.

QuixiAI reports running DeepSeek v4 Flash 0731 on 4x A100 with SlimServe, achieving 175 tok/s for single requests and 1k tok/s for 64 concurrent requests.

User seeks community advice on reducing VRAM usage and freeing OS RAM when serving DeepSeek-V4-Flash-0731 on two DGX Spark machines with vLLM, sharing detailed configuration and memory measurements.

The author ports vLLM's serving stack to C++20, creating a 66 MiB binary with no Python at inference, token-for-token verifiable against vLLM and competitive in throughput on tested hardware.

This paper introduces RAG-Stack, a framework that co-optimizes RAG serving performance and answer quality by efficiently exploring the joint algorithm-system configuration space. It finds Pareto frontiers that cover significantly more quality-performance space than existing configuration-search methods.

xMIx is a serving-native platform that enables deploying mechanistic interpretability applications in production LLM serving systems with minimal overhead, achieving near-native performance by attaching MI functions to model layers and activating them dynamically at runtime.

A new open-source dashboard for serving LLMs, developed at MIT, is being released.

Lithos announces its inference engine serving Kimi K2.7 Code, achieving over 1,000 tokens/sec per user on a single 8×B200 node at native precision, 3.4–5.7× faster than major providers.

A survey that systematically reviews system-aware KV cache optimization techniques for efficient large language model serving, organizing existing work into execution/scheduling, placement/migration, and representation/retention dimensions.

A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.

This article summarizes a presentation by Junda Chen on disaggregated inference for LLMs, explaining why goodput (throughput meeting latency SLOs) matters more than raw throughput, and how separating prefill and decode phases improves performance. It also highlights the influence on NVIDIA Dynamo.

This paper introduces a distribution-aware, prediction-free scheduling framework for LLM inference that replaces explicit length prediction with soft priority boosting using statistical signals. The method co-optimizes scheduling and cache-aware preemption to reduce tail latency, achieving up to 35-50% reduction in P99 TTLT compared to SRPT with perfect length knowledge.

A user asks how llama.cpp can serve multiple users each with full context length, noting that it seems to only share the context pool rather than providing dedicated context per user.

vLLM announces native support for Google DeepMind's DiffusionGemma, a 26B discrete diffusion language model that generates 256-token blocks in parallel, enabling low-latency inference at 1200+ tok/s on a single H200.