On-Policy Delta Distillation

Hugging Face Daily Papers Papers
on-policy-distillation reinforcement-learning delta-signal knowledge-distillation reasoning-llms post-training

Summary

The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD^2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD^2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:40 AM

Paper page - On-Policy Delta Distillation

Source: https://huggingface.co/papers/2607.15161

Abstract

On-policydistillationisanalternativepost-trainingmethodinreinforcementlearningthatalleviatestheconstraintsimposedbyrewardmodelsbyprovidingtoken-levelsupervisionfromateachermodel.Althoughon-policydistillationhasbeenstudiedandappliedacrossvarioussettings,itsfundamentaldesignremainsunderexplored.Inthispaper,weintroduceanewdistillationreward,termedthedeltasignal,insteadofdirectlyimitatingtheteacher’soutputdistribution.Thedeltasignalisdefinedasthedifferencebetweentheteachermodelanditsbasemodelpriortoinstructiontuningforreasoningcapability.Itthereforecapturesthechangesinducedbyreasoningtuningandprovidesamoredirectsignalfortransferringreasoningcapabilities.Usingextensiveempiricalevidence,weshowthatthedeltasignalsubstantiallyimproveson-policydistillationandrefertothenewdistillationmethodasOn-PolicyDeltaDistillation(OPD^2).Experimentsacrossmathematics,science,andcode-reasoningbenchmarksdemonstratethatOPD^2consistentlyoutperformsconventionalon-policydistillation,enablingreasoningLLMstoachievestrongperformancewithonlyashortpost-trainingperiod.Codewillbeavailableathttps://github.com/naver-ai/opd2

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.15161

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.15161 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.15161 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.15161 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.