Tag
This paper investigates on-policy distillation (OPD) and finds that it reliably transfers compositional reasoning skill but transfers minimal new factual knowledge, showing that OPD teaches students to organize and compose what they already know rather than expand parametric memory.
This paper proposes Adaptive Entropy Distillation (AED), a method that dynamically calibrates token-level imitation strength in knowledge distillation using teacher entropy, achieving superior performance on instruction-following and mathematical reasoning benchmarks.
This paper reexamines the role of temperature in large language model distillation, revealing that temperature asymmetrically benefits forward KL divergence over reverse KL, allowing simple KL methods to match state-of-the-art distillation approaches at higher temperatures.