Tag
The blog post identifies a silent expert death failure mode in ultra-sparse MoE training and introduces LM Loss as Auxiliary Loss (LLAL) to rescue lower-layer experts, improving validation loss and downstream performance.