DAPD: Dual-Anchored Policy Distillation

Hugging Face Daily Papers Papers
distillation policy-distillation language-model post-training self-distillation

Summary

This paper introduces Dual-Anchored Policy Distillation (DAPD), a framework to address privilege illusion in on-policy self-distillation for language models. DAPD improves Qwen3-4B by +2.00 points on average across tasks, with gains persisting at larger scales.

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
Original Article
View Cached Full Text

Cached at: 08/04/26, 09:38 AM

Paper page - DAPD: Dual-Anchored Policy Distillation

Source: https://huggingface.co/papers/2608.01735

Abstract

On-policy(self)distillation(OPSD)isincreasinglyadoptedforlanguage-modelpost-training.Itstrengthenstheteacherwithprivilegedinformationbutcaninduceaprivilegeillusion:thestudentlearnsprivilege-dependentbehavioritcannotreproducefromitsinference-timecontext,yetbehavesasifthetraining-timeprivilegedinformationremainedavailable,ultimatelydegradingperformance.Inthispaper,weidentifyinformationasymmetrybetweentheprivilegedteacherandthestudentatinferenceastherootcauseofthisfailureinOPSD.Toresolvethisasymmetry,weproposeDual-AnchoredPolicyDistillation(DAPD),aunifiedframeworkwithtwolevelsofanchoring.Dual-PathAnchoring(DPA)introducesaself-conditionedbridgeandalignsreferenceandrolloutbehavioralongtwomatched-informationpaths,preventingprivilege-dependentbehaviorfrombeingtransferredtotheinference-timestudent.Dual-SourceAnchoring(DSA)appliesthesepathsinbothreference-to-rolloutandrollout-to-referencedirections,reducingrelianceonprivilegedreferenceguidancewhilepreservingcorrectnesssupervision.ExtensiveexperimentsshowthatDAPDsignificantlyalleviatesprivilegeillusion,outperformingOPSDonQwen3-4Bby+2.00pointsonaverageacrosstasks.Notably,itsgainspersistacrossscales,reaching+2.69at4Band+2.78at32B.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.01735

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.01735 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.01735 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.01735 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DOPD: Dual On-policy Distillation

Hugging Face Daily Papers

DOPD proposes a dual on-policy distillation paradigm that dynamically routes token-level supervision between privileged teacher and student policies based on advantage gaps and probabilities, addressing privilege illusion and improving capability transfer in LLMs and VLMs.

Adaptive Supervised Anchoring for On-Policy Self-Distillation

arXiv cs.LG

This paper proposes an adaptive supervised anchoring framework for on-policy self-distillation, addressing the problem of rollout-conditioned signal degradation in language model training. The method separates rollout-conditioned distribution matching from canonical-context supervision, improving task acquisition while preserving general capabilities.

Tail-Aware Top-$k$ On-Policy Distillation

arXiv cs.LG

This paper proposes Tail-Aware Top-k On-Policy Distillation (TA-OPD) to address the loss of tail probability in on-policy distillation for language models, improving downstream accuracy on benchmarks.

Trace-Based On-Policy Distillation for Masked Diffusion Language Models

arXiv cs.CL

A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.