SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
Summary
Sign-Gated On-Policy Distillation (SG-OPD) enhances standard on-policy distillation by using a binary verifier as a trust signal for teacher supervision, improving performance on competition-level math reasoning benchmarks.
View Cached Full Text
Cached at: 06/12/26, 06:51 AM
Paper page - SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
Source: https://huggingface.co/papers/2606.09304
Abstract
Sign-Gated On-Policy Distillation improves upon standard on-policy distillation by incorporating a binary verifier to filter teacher signals, resulting in better performance on mathematical reasoning tasks.
On-policy distillation(OPD) trains astudenton its own trajectories withdense per-token supervisionfrom a strongerteacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice:trajectory-level alignmentbetween thestudentand theteacher, and uniformtoken-level reliabilityof theteacher’s preferences. We therefore propose Sign-GatedOn-Policy Distillation(SG-OPD), which uses abinary verifieras a trust signal for theteacherat two complementary granularities:phased teacher samplingmixes in verifier-endorsedteacherrollouts at cold-start, and asign-consistency gateextrapolates thedistillation updateon tokens where theteacheragrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.09304
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.09304 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.09304 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.09304 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
OmniOPD introduces a logit-free on-policy distillation method that uses chunk-level semantic similarity and speculative verification to train student models with black-box teachers, achieving up to +28.64% improvement on math benchmarks over standard OPD.
On-Policy Delta Distillation
The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.
OPRD: On-Policy Representation Distillation
OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.
Pass the Baton: Trajectory-Relayed On-Policy Distillation
Proposes Relay On-Policy Distillation (Relay-OPD) that addresses prefix failure in on-policy distillation by having the teacher briefly take over at detected handoff triggers to correct reasoning trajectories. Achieves consistent improvements on mathematical reasoning benchmarks and reduces training trajectory length by over 50%.
DOPD: Dual On-policy Distillation
DOPD proposes a dual on-policy distillation paradigm that dynamically routes token-level supervision between privileged teacher and student policies based on advantage gaps and probabilities, addressing privilege illusion and improving capability transfer in LLMs and VLMs.