Tag
FluidPD introduces SLO-aware in-place elasticity for prefill-decode disaggregated LLM serving, using FluidToken to offload prefill work during transient imbalance and FluidRole to reassign workers between phases during sustained shifts, improving SLO attainment over static SGLang by up to 94.6 percentage points on Azure production traces.
This paper synthesizes research on efficient distributed LLM serving, connecting vLLM and llm-d as complementary layers and proposing an Inference Execution Planner for future scheduler development.
ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that predicts expert activations from prefill signatures and routes requests to minimize latency, implemented in vLLM and achieving 5.9-13.9% reduction in median TPOT on up to 40 GPUs.
The UC San Diego Hao AI Lab blog provides a clear explanation of DistServe's Prefill Decode Disaggregation, tracing its acceptance from 2024 to 2025 and linking to related technologies like NVIDIA Dynamo, llm-d, Ray Serve LLM, LMCache, and MoonCake, making it a great starting point for learning LLM inference.
This blog post from Anyscale explains the intuition behind Prefill-Decode (PD) disaggregation for LLM serving, showing how separating prefill and decode phases onto dedicated GPUs can achieve up to 2.7x better goodput and 67% cost savings when using Ray and vLLM on AMD MI325X, while also discussing when PD disaggregation does not help.
This paper presents ZCube, a novel network architecture developed by Z.ai, Harnets.AI, and Tsinghua University to address topology-induced congestion in Prefill-Decode disaggregated LLM inference clusters. Production deployments on GLM-5.1 coding workloads achieved a 33% reduction in network CapEx, 15% throughput improvement, and 40.6% reduction in TTFT P99 latency.