dense-to-moe

Tag

Cards List
#dense-to-moe

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

arXiv cs.LG ↗ · 2026-10-15 Cached

论文提出 Spectral-MoE,一种在稠密模型转 MoE 时为不同层动态分配专家利用率(utilization budget)的方法,包括静态和动态两种分配器,在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率,确立了逐层专家利用率这一新的优化方向。

0 favorites 0 likes
#dense-to-moe

What do you think about ToMoE v2 paper, converting dense model to MoE model at near lossless accuracy? I feel Qwen3.8-27B-A16B or something along those lines would be amazing, though there are architectural hurdles, as well as need folr training data.

Reddit r/LocalLLaMA ↗ · 2026-09-28 Cached

ToMoE is a method that converts dense large language models into mixture-of-experts models through dynamic structural pruning, achieving strong performance without weight updates and outperforming state-of-the-art pruning and MoE techniques.

0 favorites 0 likes
#dense-to-moe

Mutating Gemma 4 31B Dense in to a native Gemma 4 additive-MoE model

Reddit r/LocalLLaMA ↗ · 2026-05-29

A proof-of-concept training script enables converting Gemma 4 31B dense model into an additive-MoE architecture, inspired by a Korean-specific finetune from JDONE-Research.

0 favorites 0 likes
← Back to home

Submit Feedback