Tag
Cloudflare optimized the memory usage of their Pingora-based load-balancing service by refining the pingora-ketama consistent hashing library in Rust, reclaiming over 100TB of RAM globally.
This paper proposes PPO-STGNN, a reinforcement learning algorithm that combines proximal policy optimization with spatio-temporal graph neural networks to optimize DAG task scheduling in heterogeneous cloud-edge-end environments, improving load balancing and reducing completion time.
This paper analyzes a one-year production trace from Chutes to study LLM serving workloads, revealing temporal evolution and user-model interactions to improve serving system benchmarking.
A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.
This paper presents EasyBalance, a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference that schedules and jointly executes workloads from different layers to mitigate GPU idling without modifying expert-device mappings, reducing idle time by over 40% in experiments.
The paper presents a two-level algorithm for allocating a conserved capacity budget across locations and service classes, proving it conserves the budget and converges in one iteration, and evaluates it for defending CDNs under volumetric attack.
Matt Godbolt explains how Compiler Explorer runs on AWS in 2026, covering CloudFront, load balancing, auto-scaling fleets, and infrastructure-as-code with Terraform.
This paper presents ReBA (Relax Within, Balance Across), a geometry-guided load-balancing method for vision-language mixture-of-experts models. It addresses modality-specific routing imbalance by balancing image and text separately and grouping correlated visual tokens, improving robustness across resolution and composition shifts.
FVAttn is a training-free sparse attention system that uses runtime load balancing to improve distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism for video generation, achieving up to 4.41x attention speedup and 2.11x inference speedup over FlashAttention on step-distilled Wan2.2 I2V while maintaining competitive video quality.
An introduction to a learning track on building load balancers from scratch, covering algorithms like round robin, least connections, consistent hashing, and practical pitfalls such as health check intervals.
GORGO introduces a proxy architecture for LLM inference that jointly optimizes network latency, prefill cost, and queueing delay using evolutionary strategy tuning on a new synthetic dataset, improving p95 TTFT by 6.9-15.5% and end-to-end latency by 14.3-30.9%.
The article provides a detailed explanation of Mixture of Experts (MoE) in transformers, covering routing, load balancing, and recent innovations like fine-grained experts. It also highlights the significance of Noam Shazeer's research contributions and his move from Google to OpenAI.
This paper proposes φ-balancing, a principled framework for load balancing in Mixture-of-Experts models that directly targets population-level expert balance using convex duality and mirror descent, achieving more stable expert utilization and outperforming prior methods on reasoning and code generation benchmarks.
A practical blueprint for designing a backend system capable of handling 1 million concurrent users, covering architecture decisions like language selection, load balancing, database sharding, multi-layer caching, and resilience patterns.
MACS is a training-free inference framework that mitigates the straggler effect in expert parallelism for multimodal MoE MLLMs by introducing entropy-weighted load and dynamic modality-adaptive capacity mechanisms.