SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Hugging Face Daily Papers Papers
reinforcement-learning knowledge-distillation advantage-fusion rlvr grpo language-models

Summary

SAF-OPD introduces a Stable Advantage Fusion framework that combines RLVR with on-policy distillation, addressing magnitude and temporal mismatches to improve training stability and performance on math and code benchmarks.

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.
Original Article
View Cached Full Text

Cached at: 08/03/26, 01:32 PM

Paper page - SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Source: https://huggingface.co/papers/2607.29209 Published on Jul 31

·

Submitted byhttps://huggingface.co/dingyii

dingyion Aug 3

Abstract

Reinforcementlearningwithverifiablerewards(RLVR)broadcastsasingleresponse-levelrewardtoeverytoken,whileon-policydistillation(OPD)scoreseachtokenagainstastrongerteacherforadenseadvantagebutcapsperformanceatteacherqualityanddiscouragesexplorationbeyondit.TheircomplementaritymakescombiningRLVRandOPDpromising,butwefindthatfusingthetwoadvantageswithafixedcoefficienttriggersentropycollapsefromtwomiscalibrations:amagnitudemismatch,wheretoken-levelOPDadvantagescanspikefarbeyondtheboundedRLVRadvantageanderaseitssignal,andatemporalmismatch,wheresustainedfull-strengthOPDkeepspullingthestudenttowardtheteacherandlimitsexplorationneededtosurpassit.WeproposeSAF,aStableAdvantageFusionframeworkthatresolvesbothissuesviaalightweight,four-stagepipelineappliedonlytotheOPDadvantage:asparsify-then-compressmechanismformagnitudecontrolpairedwithawarm-up-then-annealmechanismfortemporalcontrol,witheachstageindependentlyswitchableandaddingnegligibleoverhead.InstantiatingRLVRwithGRPO,weevaluateSAFacrosssevenmathematicalreasoningandcodegenerationbenchmarkswithQwen3-1.7B/4B/8B:SAFavoidsentropycollapseandconsistentlyoutperformsfixed-coefficientGRPO+OPDfusion,improvingtheaggregatescoreby0.51-2.70%acrossallsixmodel-domainsettingswhileachievingmorestabletraining.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.29209

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.29209 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.29209 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.29209 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.

β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Hugging Face Daily Papers

This paper introduces β-OPSD, a generalization of on-policy self-distillation that frames it as a policy optimization family with a controllable KL penalty. The method uses distillation to approximate expensive policy optimization, improving stability and reasoning performance on mathematical benchmarks.