Tag
An introduction to a learning track on building load balancers from scratch, covering algorithms like round robin, least connections, consistent hashing, and practical pitfalls such as health check intervals.
GORGO introduces a proxy architecture for LLM inference that jointly optimizes network latency, prefill cost, and queueing delay using evolutionary strategy tuning on a new synthetic dataset, improving p95 TTFT by 6.9-15.5% and end-to-end latency by 14.3-30.9%.
The article provides a detailed explanation of Mixture of Experts (MoE) in transformers, covering routing, load balancing, and recent innovations like fine-grained experts. It also highlights the significance of Noam Shazeer's research contributions and his move from Google to OpenAI.
This paper proposes φ-balancing, a principled framework for load balancing in Mixture-of-Experts models that directly targets population-level expert balance using convex duality and mirror descent, achieving more stable expert utilization and outperforming prior methods on reasoning and code generation benchmarks.
A practical blueprint for designing a backend system capable of handling 1 million concurrent users, covering architecture decisions like language selection, load balancing, database sharding, multi-layer caching, and resilience patterns.
MACS is a training-free inference framework that mitigates the straggler effect in expert parallelism for multimodal MoE MLLMs by introducing entropy-weighted load and dynamic modality-adaptive capacity mechanisms.