knowledge-distillation

Introduces SPOT, a method for on-policy distillation that uses sparse probing and outcome calibration to improve reasoning performance in smaller student models while balancing solution quality and coverage.

This paper studies self-distillation with privileged information (PI) as a lone post-training objective for LLMs, reproducing reported gains on easy tasks but showing it fails on difficult reasoning tasks: per-token loss drops while validation accuracy stagnates or degrades. The authors trace the failure to PI bias, which pulls teacher targets toward a reference trajectory and trains students to be flatter and less decisive without improving reasoning.

This paper investigates the failure modes of converting Qwen3-0.6B-Base attention layers to KDA linear attention on a single GPU, identifying an 'interface injury' where the model predicts option labels rather than content, and proposes a format-targeted KL stage to repair it.

This paper introduces RSTG, a method that selectively applies on-policy distillation to recover learning signals from zero-variance GRPO groups during LLM post-training, achieving substantial gains on math and code reasoning benchmarks.

Proposes Progressive2, a teacher-student progressive co-evolving knowledge distillation method for substantial model compression, addressing large disparity between server and client capabilities. Introduces progressive teacher layer selection and student size reduction with a multi-feature fusion adapter.

SKILL-KD is a contrastive skill distillation framework that improves LLM agents by distilling actionable discrepancies between teacher and student trajectories into textual skill patches, with drift-aware consolidation to iteratively refine skills.

This paper introduces SA-OPD, a spurious-signal-aware on-policy distillation framework that filters misleading token-level teacher supervision based on input-grounding and optimization impact, improving LLM and VLM distillation performance.

This paper proposes DASH-OPD, a discrepancy-aware switching method for on-policy distillation in multi-turn LLM agent training, which adaptively toggles between teacher and student executors based on drift and recovery evidence to improve efficiency and performance.

This paper investigates using an 8-billion parameter LLM to improve autonomous cyber defense, then distills its policy into a lightweight 64,910-parameter RL agent, demonstrating feasibility across CybORG scenarios.

This paper shows that knowledge distillation has asymmetric effects on bias in small language models: it improves context-following on unambiguous tasks but harms refusal calibration on ambiguous ones, and proposes PCCD, a protocol to diagnose such per-item harms that aggregate metrics miss.

An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.

SAF-OPD introduces a Stable Advantage Fusion framework that combines RLVR with on-policy distillation, addressing magnitude and temporal mismatches to improve training stability and performance on math and code benchmarks.

This paper proposes Expert-Guided Mutual Distillation (EGMD) to address domain bias and semantic misalignment in multimodal fake news detection, achieving state-of-the-art accuracy and reducing domain bias by up to 57.3% across four datasets.

Introduces Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves a strong language model by distilling from multiple weaker models using contrast pairs in logit space, consistently outperforming standard on-policy distillation on math and code benchmarks.

Flux-OPD proposes an on-policy distillation paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains, outperforming existing OPD paradigms.

Proposes Relay On-Policy Distillation (Relay-OPD) that addresses prefix failure in on-policy distillation by having the teacher briefly take over at detected handoff triggers to correct reasoning trajectories. Achieves consistent improvements on mathematical reasoning benchmarks and reduces training trajectory length by over 50%.

Masked distillation is a knowledge-distillation framework that trains a student LLM to predict only solution tokens while a reasoning teacher provides feedback, aiming to internalize chain-of-thought computation into model parameters. The method shows task-dependent success, working on GSM8K but requiring small scaffolds for harder tasks like Countdown.

CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.

This paper introduces AMRD, an adaptive multi-teacher relational distillation method for compressing large self-supervised speech emotion recognition models into lightweight student models for edge devices. It addresses teacher reliability variation and relational structure loss, showing improvements on IEMOCAP and CREMA-D datasets.

Nvidia CEO Jensen Huang defended open source AI, stating that knowledge distillation is fundamental to the learning process.