@FinanceYF5: MoE models may waste about half of expert computations on tokens that don't need experts 1/ Half of experts are working for nothing MoE models already seem efficient, but a paper finds that many tokens don't need expert processing at all. ZEDA teaches the model to "save when possible," skipping up to 50% of expert computations.
Summary
A paper discovers that about 50% of expert computations in MoE models are wasted on tokens that don't need expert processing. The proposed ZEDA method teaches the model to skip these computations, saving up to half of expert calculations.
View Cached Full Text
Cached at: 05/25/26, 04:30 AM
🧵MoE large models may have half of their expert computations actually spent on tokens that don’t need experts at all
1/ ⚡️Half of the experts are working in vain
MoE models already seem quite compute-efficient, but research papers have found: many tokens don’t need expert processing at all.
ZEDA teaches the model to “save where it should,” skipping up to about 50% of expert computations. 👇 https://t.co/5vtoJ8Gcq3
2/ The problem is not the number of parameters
In traditional MoE, each token calls a fixed number of experts.
Even simple tokens like “hello” or “today” go through the same expensive expert computation.
3/ ZEDA adds a zero expert
ZEDA introduces a class of zero expert to the router.
They do nothing, their output is always 0, effectively telling the model: this token can skip the expert.
4/ Not retraining, but self-distillation
The original MoE is frozen as a Teacher.
A new model, as a Student, learns when it can use fewer experts and still approximate the teacher’s output, thus transforming into a dynamic MoE with low cost.
5/ The results are straightforward
On Qwen3-30B-A3B and GLM-4.7-Flash, ZEDA reduces expert computation by about 50%.
Average accuracy drops only slightly, with real inference speed improving by about 20%.
6/ The key is “uncertainty”
The model doesn’t simply allocate compute based on task difficulty.
When the teacher-student divergence increases, it uses more experts; when the token is very certain, it uses fewer experts.
7/ This is not pruning
ZEDA is more like giving MoE a “compute budget awareness.”
Future models will not only decide what to answer, but also whether each token is worth thinking hard about.
Paper: Post-Trained MoE Can Skip Half Experts via Self-Distillation
That’s all.
If you like this topic:
- Follow me (@FinanceYF5)
- Like + repost the first post below
Similar Articles
@rohanpaul_ai: A large MoE model may be wasting half its expert compute on tokens that barely need expert help. In this paper 50% of e…
A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.
Post-Trained MoE Can Skip Half Experts via Self-Distillation
ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.
@aehyok: Share an open-source project FreeToken, a local inference engine specifically for running ultra-large Mixture-of-Experts (MoE) models on consumer-grade computers. Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → R…
FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.
Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss
该论文研究在 MoE 模型 prefill 阶段减少路由专家对短答案问答 F1 的影响，发现大部分损失来自提示词首尾少数敏感 token，并提出将约 4% 的 token 保持全专家以几乎无损地降低 TTFT。
Converting dense models into Mixture-of-Experts
一位独立开发者在 RTX 4060 (8GB) 上尝试将稠密模型直接转换为稀疏 MoE 模型（Qwen2.5-0.5B 和 SmolLM2-360M，32 专家 top-8），无需从头预训练，并分享了计算量节省与性能损失的评测结果。