Tag
This paper introduces TIER-MoE, a risk-guided subspace mixture-of-experts model for multimodal biomedical classification that estimates sample-specific modality reliability from out-of-fold predictions and routes modalities to experts, improving performance and calibration on four public datasets.
OPERA proposes a multi-agent ensemble framework that treats expert weight assignment as an offline policy learning problem for universal biomedical image analysis, enabling test-time adaptation without retraining and consistently improving performance across 9 datasets and 30+ baselines.
Proposes PADD, a framework for distilling knowledge from dense teachers into mixture-of-experts (MoE) students, addressing the challenge of learning routing policies without a router in the teacher. The method involves four stages and shows improvements on mathematical reasoning benchmarks.
This paper proposes dMoE, a block-level mixture-of-experts framework for diffusion large language models that aggregates token-level expert distributions into block-level routing, reducing activated experts and memory usage while maintaining performance.
A technical deep-dive into common causes of failed pretraining runs in large language models, including causality-breaking issues in expert routing and numerical precision bugs, with examples from Llama 4, Gemini 2 Pro, and GPT-4.
The paper introduces an information-theoretic framework for communication-efficient expert routing in sparse mixture-of-experts models, treating the gate as a stochastic channel and deriving practical mutual information estimators to analyze accuracy-rate tradeoffs over finite expert banks.