moE

A detailed recipe for running the unpruned GLM-5.2 model (744B parameters, 256 experts) across 4 NVIDIA DGX Spark nodes with 200K context, achieving up to 60.5 tok/s aggregate. Includes performance benchmarks, credits, and patches.

openPangu-2.0-Flash is a 92B-parameter MoE model with 6B activated parameters, trained on Ascend, featuring 512k context length and fast thinking capabilities. It achieves strong performance on reasoning and coding benchmarks, using architectural innovations like MLA attention and multi-token prediction.

This paper explores structured pruning and knowledge distillation techniques for compressing large Mixture-of-Experts (MoE) models during pre-training. It demonstrates that progressive pruning and combined distillation strategies, such as multi-token prediction distillation, improve downstream performance, exemplified by compressing Qwen3-Next-80A3B to a more efficient 23A2B model.