ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

Hugging Face Daily Papers Papers
mixture-of-experts llm-serving decode-routing expert-locality vllm prefill-decode-disaggregation performance-optimization

Summary

ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that predicts expert activations from prefill signatures and routes requests to minimize latency, implemented in vLLM and achieving 5.9-13.9% reduction in median TPOT on up to 40 GPUs.

In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.
Original Article
View Cached Full Text

Cached at: 07/02/26, 07:47 AM

Paper page - ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

Source: https://huggingface.co/papers/2607.00466

Abstract

ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that improves performance by predicting expert activations and routing requests accordingly.

Inprefill-decode(PD)disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existingdecode routers balance only load; formixture-of-experts(MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, anexpert-locality-awaredecode routerfor PD-disaggregated MoE serving. From a request’s prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balancedK-meanspartitions signature space across decode workers; online,locality-band routingsends each request to the least-loaded worker among those best matching its signature. Asignature cache, co-indexed with theKV cacheat KV-block granularity, keeps signatures exact under prefix caching. Implemented invLLMand evaluated on deployments of up to 40 GPUs, ELDR reduces medianTPOTby 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.00466

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.00466 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.00466 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.00466 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

dMoE: dLLMs with Learnable Block Experts

arXiv cs.CL

dMoE proposes block-level expert routing for diffusion LLMs, reducing the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% performance and achieving 76-80% memory reduction with 1.14-1.66× speedup.