Hybrid Policy Distillation for LLMs
Summary
Introduces Hybrid Policy Distillation (HPD), a unified knowledge distillation approach that balances forward and reverse KL divergences and combines off-policy data with lightweight on-policy sampling, improving LLM compression across math, dialogue, and code tasks.
View Cached Full Text
Cached at: 04/23/26, 10:03 AM
# Hybrid Policy Distillation for LLMs Source: [https://arxiv.org/abs/2604.20244](https://arxiv.org/abs/2604.20244) [View PDF](https://arxiv.org/pdf/2604.20244) > Abstract:Knowledge distillation \(KD\) is a powerful paradigm for compressing large language models \(LLMs\), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime\. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log\-likelihood objective at the token level\. We further propose Hybrid Policy Distillation \(HPD\), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode\-seeking, and combines off\-policy data with lightweight, approximate on\-policy sampling\. We validate HPD on long\-generation math reasoning as well as short\-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales\. The code related to this work is available at[this https URL](https://github.com/zwhong714/Hybrid-Policy-Distillation)\. ## Submission history From: Wenhong Zhu \[[view email](https://arxiv.org/show-email/8037def0/2604.20244)\] **\[v1\]**Wed, 22 Apr 2026 06:46:22 UTC \(1,059 KB\)
Similar Articles
DOPD: Dual On-policy Distillation
DOPD proposes a dual on-policy distillation paradigm that dynamically routes token-level supervision between privileged teacher and student policies based on advantage gaps and probabilities, addressing privilege illusion and improving capability transfer in LLMs and VLMs.
Lexicographic Multi-Objective On-Policy Distillation
Cohere researchers introduce Lexicographic Multi-Objective On-Policy Distillation (LMOPD), a multi-teacher method that integrates reward-specialized policies under explicit priority orders using gated routing and projected log-policy corrections. Evaluated on 30B-A3B MoE models across three math benchmarks, LMOPD retains far more top-priority accuracy and reasoning gains than baselines while still capturing conciseness improvements.
Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
The paper introduces Latent-MOPD, the first representation-level multi-teacher on-policy distillation method for LLMs, which integrates multiple specialist teachers through both their output distributions and hidden states without extra teacher training, outperforming token-only, representation-only, and averaging baselines across nine math, code, and logic benchmarks.
OPRD: On-Policy Representation Distillation
OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.
Reasoning Compression with Mixed-Policy Distillation
This paper proposes Mixed-Policy Distillation (MPD), a framework that transfers concise reasoning behaviors from large teacher models to smaller student models, reducing token usage by up to 27.1% while improving performance.