Tag
ROSS introduces selective supervision on historical rollouts to improve language model post-training, enhancing performance in math, code, and software engineering tasks without additional policy rollouts.
The paper shows that Direct-On-Policy Distillation's token-level log-ratio reward can remain unchanged even when the teacher checkpoints' divergences vanish, motivating S²D-OPD, a method that masks supervision at low-divergence states using teacher-reference JSD. S²D-OPD improves held-out accuracy on AIME and HMMT benchmarks in 7 of 8 teacher-student settings across models from 1.7B to 8B parameters without extra forward passes.
Presents counterfactual recoverability as an outcome-grounded decision variable for on-policy distillation, showing that selectively supervising based on recoverability outperforms divergence-only methods across AIME and GPQA benchmarks.
This paper introduces CHERRY, a set of techniques for compute-efficient language models including selective token supervision, depth compression via recurrent unrolling, and a mixture of compressed experts, achieving significant efficiency gains on a Korean foundation model.