Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
Summary
The paper introduces Influence-Directed Adaptive On-Policy Distillation (IDA-OPD) to solve the diversity bottleneck in sampled-token on-policy distillation, enhancing diversity transfer in reasoning models without full-vocabulary teacher data.
View Cached Full Text
Cached at: 09/03/26, 03:52 AM
Paper page - Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
Source: https://huggingface.co/papers/2608.29846
Abstract
Influence-Directed Adaptive On-Policy Distillation improves diversity transfer in reasoning model distillation by selectively preserving entropy-expanding updates and replacing entropy-contracting ones with adaptive advantage shrinkage.
Sampled-tokenon-policy distillation(OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet it frequently suffers from diversity distillation failure: the student’s pass@1 improves while itspass@kplateaus, failing to inherit the teacher’s diversity. To explain this, we introduceFirst-Order Local Entropy Influence, a signed first-order proxy that decouples each update’s entropy effect into the teacher--student log-probability gap and the student’s local probability structure, and empirically linksentropy contractionto negative-influence positions. Motivated by this, we propose Influence-Directed AdaptiveOn-Policy Distillation(IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones withdivergence-adaptive advantage shrinkage, using only the teacher’s sampled-token log-probability. Experiments onreasoning-oriented distillationshowIDA-OPDconsistently improvespass@k, inheriting the teacher’s diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD’s pass@1, all without full-vocabulary teacher information.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.29846
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.29846 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.29846 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.29846 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DOPD: Dual On-policy Distillation
DOPD proposes a dual on-policy distillation paradigm that dynamically routes token-level supervision between privileged teacher and student policies based on advantage gaps and probabilities, addressing privilege illusion and improving capability transfer in LLMs and VLMs.
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
The paper analyzes on-policy distillation, revealing it primarily suppresses low-probability tokens rather than relying on teacher guidance, and introduces OPSA, a supervision-free method that significantly enhances reasoning performance.
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
DiffusionOPD proposes a multi-task training paradigm for diffusion models that uses online policy distillation to efficiently combine task-specific teachers into a unified student, achieving state-of-the-art results on all evaluated benchmarks.
On-Policy Delta Distillation
The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
This paper introduces SA-OPD, a spurious-signal-aware on-policy distillation framework that filters misleading token-level teacher supervision based on input-grounding and optimization impact, improving LLM and VLM distillation performance.