Tag
This paper investigates whether KL divergence is necessary for on-policy distillation of large language models, showing that preserving update direction suffices and introducing Consensus Multi-Teacher On-Policy Distillation (C-MOPD) to enhance multi-teacher learning.
The paper examines correctness-gated multi-teacher distillation, finding decision shifts, lost label functionality, and an inconclusive grounding audit, with no incremental benefit over hard filtering.
This paper investigates generalization in on-policy distillation for large language models, showing that it transfers reasoning behaviors and that teacher-student origin alignment is crucial, with multi-teacher combinations causing capability trade-offs.