Tag
Presents counterfactual recoverability as an outcome-grounded decision variable for on-policy distillation, showing that selectively supervising based on recoverability outperforms divergence-only methods across AIME and GPQA benchmarks.
This paper introduces CHERRY, a set of techniques for compute-efficient language models including selective token supervision, depth compression via recurrent unrolling, and a mixture of compressed experts, achieving significant efficiency gains on a Korean foundation model.