switch-distillation

This paper introduces Switch Distillation, a novel mid-training objective that selectively applies knowledge distillation based on teacher confidence to improve reasoning and preserve factual recall in smaller language models.