Tag
This paper argues that retraining-free MoE compression checkpoints should be seen as compressed initializations that benefit from a small post-compression adjustment. It shows that with only 3,000 examples and one epoch, LM fine-tuning recovers a substantial portion of performance loss and is more cost-effective than knowledge distillation.