Hybrid Policy Distillation for LLMs
Summary
Introduces Hybrid Policy Distillation (HPD), a unified knowledge distillation approach that balances forward and reverse KL divergences and combines off-policy data with lightweight on-policy sampling, improving LLM compression across math, dialogue, and code tasks.
View Cached Full Text
Cached at: 04/23/26, 10:03 AM
# Hybrid Policy Distillation for LLMs Source: [https://arxiv.org/abs/2604.20244](https://arxiv.org/abs/2604.20244) [View PDF](https://arxiv.org/pdf/2604.20244) > Abstract:Knowledge distillation \(KD\) is a powerful paradigm for compressing large language models \(LLMs\), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime\. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log\-likelihood objective at the token level\. We further propose Hybrid Policy Distillation \(HPD\), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode\-seeking, and combines off\-policy data with lightweight, approximate on\-policy sampling\. We validate HPD on long\-generation math reasoning as well as short\-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales\. The code related to this work is available at[this https URL](https://github.com/zwhong714/Hybrid-Policy-Distillation)\. ## Submission history From: Wenhong Zhu \[[view email](https://arxiv.org/show-email/8037def0/2604.20244)\] **\[v1\]**Wed, 22 Apr 2026 06:46:22 UTC \(1,059 KB\)
Similar Articles
DOPD: Dual On-policy Distillation
DOPD proposes a dual on-policy distillation paradigm that dynamically routes token-level supervision between privileged teacher and student policies based on advantage gaps and probabilities, addressing privilege illusion and improving capability transfer in LLMs and VLMs.
OPRD: On-Policy Representation Distillation
OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.
Reasoning Compression with Mixed-Policy Distillation
This paper proposes Mixed-Policy Distillation (MPD), a framework that transfers concise reasoning behaviors from large teacher models to smaller student models, reducing token usage by up to 27.1% while improving performance.
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
ShortOPD proposes a short-to-long on-policy distillation schedule that recovers pruned LLMs for free-form generation by focusing training on effective prefixes, achieving up to 9x improvement over unrecovered models and matching long-horizon distillation with a quarter of the training time.
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
MOPD proposes a multi-teacher on-policy distillation paradigm for LLM post-training, enabling efficient integration of multiple domain capabilities by distilling specialized RL teachers into a student model using its own rollouts. It outperforms existing methods like Mix-RL and Cascade RL, and has been deployed in industrial-scale models.