off-policy-training

论文提出 SAPD（Step-Aligned Privileged Distillation），一种无需 rollout 的自蒸馏方法，将固定示范转化为与推理步骤对齐的分布式监督，在数学推理基准上媲美在线强化学习并实现约 2 倍训练加速。

ROSS introduces selective supervision on historical rollouts to improve language model post-training, enhancing performance in math, code, and software engineering tasks without additional policy rollouts.