inference-systems

FluidPD introduces SLO-aware in-place elasticity for prefill-decode disaggregated LLM serving, using FluidToken to offload prefill work during transient imbalance and FluidRole to reassign workers between phases during sustained shifts, improving SLO attainment over static SGLang by up to 94.6 percentage points on Azure production traces.

GLM has developed its own inference infrastructure to support recursive self-improvement in AI systems.

This paper characterizes the serving behavior of masked diffusion language models (dLLMs) using real hardware measurements, identifying key differences from autoregressive models and deriving design principles for efficient inference systems.

This paper investigates how trace-driven evaluation can mislead assessments of MoE expert caching, identifying replay semantics, workload contamination, and operating regimes as confounding axes that can reverse policy rankings. After correcting these issues, it shows that a large offline-optimal gap overstates the gains actually recovered by lightweight causal caching mechanisms.