Tag
A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.
This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.
This paper introduces Director, a distributed MoE serving system that minimizes end-to-end latency using prediction-driven, online proactive expert placement. It employs a lightweight predictor and a relaxation-based optimizer to achieve up to 55% latency reduction for models like Mistral, DeepSeek, and Qwen.
A former Berkeley PhD who leads SGLang at xAI explains how they serve Grok on 100K GPUs using split prefill/decode, expert sharding, and communication/computation overlap to achieve DeepSeek-API-killing prices.
MACS is a training-free inference framework that mitigates the straggler effect in expert parallelism for multimodal MoE MLLMs by introducing entropy-weighted load and dynamic modality-adaptive capacity mechanisms.
Federation of Experts (FoE) restructures mixture-of-experts blocks into clusters that process KV heads independently, eliminating inter-node communication bottlenecks and improving inference throughput and latency by up to 5.2x while maintaining generation quality.