Tag
论文提出 Spectral-MoE,一种在稠密模型转 MoE 时为不同层动态分配专家利用率(utilization budget)的方法,包括静态和动态两种分配器,在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率,确立了逐层专家利用率这一新的优化方向。
ToMoE is a method that converts dense large language models into mixture-of-experts models through dynamic structural pruning, achieving strong performance without weight updates and outperforming state-of-the-art pruning and MoE techniques.
A proof-of-concept training script enables converting Gemma 4 31B dense model into an additive-MoE architecture, inspired by a Korean-specific finetune from JDONE-Research.