Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Summary
This paper investigates generalization in on-policy distillation for large language models, showing that it transfers reasoning behaviors and that teacher-student origin alignment is crucial, with multi-teacher combinations causing capability trade-offs.
View Cached Full Text
Cached at: 08/24/26, 08:28 AM
Paper page - Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Source: https://huggingface.co/papers/2608.16647 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
On-policy distillation transfers reasoning behaviors rather than specific answers, with generalization strongly tied to teacher-student origin alignment and multi-teacher combinations causing capability trade-offs.
On-policy distillation(OPD) transfers teacher capabilities by supervising trajectories sampled from the student’s own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domaindistribution shifts tocross-domain transferand themulti-teachersetting. We find that OPD transfers a teacher’sreasoning behaviorrather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student:same-originpairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereascross-originpairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher’s influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosingmulti-teacherOPD.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.16647
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.16647 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.16647 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.16647 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
The paper investigates on-policy distillation of large language models, demonstrating that a single training query can achieve substantial state coverage and alignment, suggesting the method is algorithm-starved rather than data-starved.
Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation
This paper identifies a failure mode in multi-teacher on-policy distillation for agentic language models where the student becomes over-reliant on tool-calling, and proposes Soft Clamp, a per-token divergence calibration method that reduces over-calling without sacrificing accuracy.
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.
@jiqizhixin: What if the best teacher for a language model isn’t the strongest one? Researchers from Tsinghua University and collabo…
Researchers from Tsinghua University and collaborators systematically investigate on-policy distillation (OPD) for LLMs, revealing that success requires shared thinking patterns and genuinely new capabilities from the teacher, and propose practical strategies like off-policy cold start and teacher-aligned prompt selection.