expert-parallelism

The article promotes a presentation on Elastic Expert Parallelism in vLLM at the PyTorch Conference North America 2026, discussing how to dynamically add or remove GPUs in Mixture-of-Experts deployments with minimal downtime.

This article explains in detail MoE (Mixture of Experts) inference engineering, corrects misconceptions about activated parameters and deployment costs, and delves into technical details such as router selection, runtime grouping, GPU execution, memory management, and expert parallelism.

The article explains the engineering aspects of Mixture-of-Experts (MoE) inference, detailing token routing, expert batching, GPU distribution, and performance trade-offs for efficient serving.

A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.

This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.

This paper introduces Director, a distributed MoE serving system that minimizes end-to-end latency using prediction-driven, online proactive expert placement. It employs a lightweight predictor and a relaxation-based optimizer to achieve up to 55% latency reduction for models like Mistral, DeepSeek, and Qwen.

A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.

MACS is a training-free inference framework that mitigates the straggler effect in expert parallelism for multimodal MoE MLLMs by introducing entropy-weighted load and dynamic modality-adaptive capacity mechanisms.

Federation of Experts (FoE) restructures mixture-of-experts blocks into clusters that process KV heads independently, eliminating inter-node communication bottlenecks and improving inference throughput and latency by up to 5.2x while maintaining generation quality.