expert-routing

论文提出 Spectral-MoE，一种在稠密模型转 MoE 时为不同层动态分配专家利用率（utilization budget）的方法，包括静态和动态两种分配器，在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率，确立了逐层专家利用率这一新的优化方向。

The paper proposes DivMoE, the first framework enabling fine-grained Mixture-of-Experts upcycling from dense models via domain-specialized expert initialization and diversity-constrained routing, fixing the routing collapse and accuracy regressions that plagued prior fine-grained upcycling methods across 15 benchmarks.

JevSoup introduces a training-free framework for routing and combining LoRA experts without additional training, achieving performance gains over existing methods on multiple tasks.

This paper presents ESRL, a reinforcement learning framework that enhances Mixture-of-Experts model training by exploring expert-routing paths, achieving improved performance on mathematical, scientific, and coding tasks.

This paper investigates whether bilingual Mixture-of-Experts (MoE) language models develop linguistically structured expert routing. It finds that interpretable linguistic organization emerges within MoE routing patterns, and that curriculum training influences specialization in language balance.

Presents TEXAS, a method for downstream adaptation of Mixture-of-Experts LLMs that discovers task-relevant experts via correctness-conditioned activations and applies token-level supervision allocation, improving performance across multiple benchmarks.

This paper introduces TIER-MoE, a risk-guided subspace mixture-of-experts model for multimodal biomedical classification that estimates sample-specific modality reliability from out-of-fold predictions and routes modalities to experts, improving performance and calibration on four public datasets.

OPERA proposes a multi-agent ensemble framework that treats expert weight assignment as an offline policy learning problem for universal biomedical image analysis, enabling test-time adaptation without retraining and consistently improving performance across 9 datasets and 30+ baselines.

Proposes PADD, a framework for distilling knowledge from dense teachers into mixture-of-experts (MoE) students, addressing the challenge of learning routing policies without a router in the teacher. The method involves four stages and shows improvements on mathematical reasoning benchmarks.

This paper proposes dMoE, a block-level mixture-of-experts framework for diffusion large language models that aggregates token-level expert distributions into block-level routing, reducing activated experts and memory usage while maintaining performance.

A technical deep-dive into common causes of failed pretraining runs in large language models, including causality-breaking issues in expert routing and numerical precision bugs, with examples from Llama 4, Gemini 2 Pro, and GPT-4.

The paper introduces an information-theoretic framework for communication-efficient expert routing in sparse mixture-of-experts models, treating the gate as a stochastic channel and deriving practical mutual information estimators to analyze accuracy-rate tradeoffs over finite expert banks.