serving

Escha Labs has released a 2-bit quantized version of the Qwen3.8-27B AI model, enabling it to run on a single 24GB consumer GPU with up to 64k context while maintaining performance comparable to FP8 references.

A 10-week, 30-minutes-a-day roadmap for engineers to learn LLM inference serving, covering vLLM, SGLang, load testing, quantization, and optimization techniques to build a reproducible benchmark.

QuixiAI reports running DeepSeek v4 Flash 0731 on 4x A100 with SlimServe, achieving 175 tok/s for single requests and 1k tok/s for 64 concurrent requests.

User seeks community advice on reducing VRAM usage and freeing OS RAM when serving DeepSeek-V4-Flash-0731 on two DGX Spark machines with vLLM, sharing detailed configuration and memory measurements.

The author ports vLLM's serving stack to C++20, creating a 66 MiB binary with no Python at inference, token-for-token verifiable against vLLM and competitive in throughput on tested hardware.

This paper introduces RAG-Stack, a framework that co-optimizes RAG serving performance and answer quality by efficiently exploring the joint algorithm-system configuration space. It finds Pareto frontiers that cover significantly more quality-performance space than existing configuration-search methods.

xMIx is a serving-native platform that enables deploying mechanistic interpretability applications in production LLM serving systems with minimal overhead, achieving near-native performance by attaching MI functions to model layers and activating them dynamically at runtime.

A new open-source dashboard for serving LLMs, developed at MIT, is being released.

Lithos announces its inference engine serving Kimi K2.7 Code, achieving over 1,000 tokens/sec per user on a single 8×B200 node at native precision, 3.4–5.7× faster than major providers.

A survey that systematically reviews system-aware KV cache optimization techniques for efficient large language model serving, organizing existing work into execution/scheduling, placement/migration, and representation/retention dimensions.

A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.

This article summarizes a presentation by Junda Chen on disaggregated inference for LLMs, explaining why goodput (throughput meeting latency SLOs) matters more than raw throughput, and how separating prefill and decode phases improves performance. It also highlights the influence on NVIDIA Dynamo.

This paper introduces a distribution-aware, prediction-free scheduling framework for LLM inference that replaces explicit length prediction with soft priority boosting using statistical signals. The method co-optimizes scheduling and cache-aware preemption to reduce tail latency, achieving up to 35-50% reduction in P99 TTLT compared to SRPT with perfect length knowledge.

A user asks how llama.cpp can serve multiple users each with full context length, noting that it seems to only share the context pool rather than providing dedicated context per user.

vLLM announces native support for Google DeepMind's DiffusionGemma, a 26B discrete diffusion language model that generates 256-token blocks in parallel, enabling low-latency inference at 1200+ tok/s on a single H200.

Modular's kernel team is optimizing serving for MiniMax M3's 1M-token context and native multimodality, with open weights dropping soon for immediate deployment on Modular.

vLLM-Omni v0.22.0 is a major upgrade adding robust support for NVIDIA Cosmos world models, production TTS (Qwen3-TTS, Qwen3-Omni, VoxCPM2), faster diffusion model serving (Wan 2.2, HunyuanVideo 1.5, LTX-2.3), and broader quantization and hardware coverage with 339 commits from 124 contributors.

The author benchmarks serving 1,000 LoRA adapters on one GPU using vLLM, finding that active adapter count and traffic shape are the real bottlenecks, and provides recommendations for tuning max_loras.

MinT is a managed infrastructure system that enables efficient training and serving of millions of LLMs by keeping base models resident and moving lightweight LoRA adapters, scaling across model architectures, storage, and policy management.

vLLM 0.20.0rc1 releases with major throughput, quantization, speculative decoding, and multi-hardware support enhancements for scalable LLM serving.