moE

InclusionAI introduces Ling-3.0-tiny, a 7.9B-parameter hybrid reasoning MoE model with only 1.3B active parameters per token, optimized for efficient local and edge deployment.

inclusionAI released Ling-3.0-flash, a native hybrid reasoning model with 124B total/5.1B active parameters using a hybrid linear attention architecture (KDA+MLA) and sparse MoE. It matches or outperforms its 1T-class predecessor Ring-2.6-1T while being far more compute-efficient, with built-in agentic and long-context optimizations.

Alibaba announces Qwen3.8-Max, a 2.4T-parameter MoE frontier model with open weights coming next week, claiming autonomous operation for 16 days and significantly lower cost than GPT-5.6 Sol and Claude Fable 5.

KAT-Coder-V2.5-Dev is an open-weight MoE coding model with 35B total parameters (3B active), achieving state-of-the-art results on agentic coding benchmarks through SFT and RL training.

A detailed recipe for running the unpruned GLM-5.2 model (744B parameters, 256 experts) across 4 NVIDIA DGX Spark nodes with 200K context, achieving up to 60.5 tok/s aggregate. Includes performance benchmarks, credits, and patches.

openPangu-2.0-Flash is a 92B-parameter MoE model with 6B activated parameters, trained on Ascend, featuring 512k context length and fast thinking capabilities. It achieves strong performance on reasoning and coding benchmarks, using architectural innovations like MLA attention and multi-token prediction.

This paper explores structured pruning and knowledge distillation techniques for compressing large Mixture-of-Experts (MoE) models during pre-training. It demonstrates that progressive pruning and combined distillation strategies, such as multi-token prediction distillation, improve downstream performance, exemplified by compressing Qwen3-Next-80A3B to a more efficient 23A2B model.