Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!

Reddit r/LocalLLaMA 09/03/26, 09:58 PM Papers

Summary

A paper introduces an inference-time optimization for sparse MoE models by adjusting expert selection in late transformer layers, reducing reasoning tokens by 8.5% and latency by 10.9% without retraining, while maintaining accuracy.

I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models. The idea: Instead of retraining anything, we just tweak the router at runtime. Specifically, we expand the expert selection budget (N≥KN≥K) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ ! What we found — "Succinct Convergence": When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories. Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B): 📉 8.5% reduction in mean reasoning tokens ⚡ 10.9% drop in latency (p=6.5×10−6) 🎯 Accuracy unchanged (84.5% vs 84.0% native, p=0.77 — statistically indistinguishable) 🆓 Zero training cost — pure inference-time routing modification Links: 📄 Paper: https://zenodo.org/records/22255483 there you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark. In the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0 I'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.

Original Article

Similar Articles

@rohanpaul_ai: A large MoE model may be wasting half its expert compute on tokens that barely need expert help. In this paper 50% of e…

X AI KOLs Timeline

A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.

Proposed architecture for inferencing sparse MOE models increasing Active parameters using layered + linear decay. Succinct reasoning without any model training or fine tune. [p]

Reddit r/MachineLearning

A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.

Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

Reddit r/LocalLLaMA

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

arXiv cs.CL

The paper introduces correlated expert placement and token shuffling to exploit router-learned expert correlations, cutting all-to-all communication time by 1.16–2.63× and end-to-end MoE pretraining step time by up to 1.41× in Megatron-LM without altering routing decisions or model parameters.

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

arXiv cs.LG

ACE introduces a method to consolidate redundant adapters across experts in MoE large language models for more efficient parameter-efficient fine-tuning, achieving up to 1.48× training speedup without increasing peak memory.