teacher-student

The paper introduces Calibrated On-Policy Distillation (Cal-OPD), a method that estimates the teacher's self-deviation to calibrate teacher-student discrepancies, improving on-policy distillation for mathematical reasoning tasks.

A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.

Proposes Progressive2, a teacher-student progressive co-evolving knowledge distillation method for substantial model compression, addressing large disparity between server and client capabilities. Introduces progressive teacher layer selection and student size reduction with a multi-feature fusion adapter.

An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.

This paper tests the assumption that attention weights reveal what a model actually depends on for its output, finding that attention and causal dependence often disagree. They propose using causal evidence sets obtained via intervention masking as supervision for sparse attention routers, achieving near-perfect accuracy on retrieval tasks where attention-distilled routers fail.

AgentBrew proposes a training-free method for distilling knowledge from strong teacher LLM agents to weak student agents, using a failure-triggered reflection loop and student-aware synthesis to create executable notes, improving performance on coding, math, and tool-use tasks without weight updates.

This paper investigates the position bias phenomenon in online distillation, finding that early tokens provide more useful supervision signals, and proposes the importance-weighted IW-OPD method to improve OPD training.

An educational overview of knowledge distillation, covering its history, core concepts like softmax and temperature, types, scaling laws, and practical examples including DeepSeek-R1.

Sign-Gated On-Policy Distillation (SG-OPD) enhances standard on-policy distillation by using a binary verifier as a trust signal for teacher supervision, improving performance on competition-level math reasoning benchmarks.

Z-Reward is a teacher-student framework that decouples complex reasoning from efficient reward deployment for text-to-image training. It achieves 89.6% human preference accuracy with a 27B teacher and 88.6% with a 9B student, outperforming prior methods.

Prompt-Level Distillation (PLD) extracts reasoning patterns from teacher models into structured instructions for student model system prompts, improving performance on reasoning tasks without fine-tuning overhead.

This paper proposes a principled offline reasoning distillation framework that corrects teacher-student distribution drift, improving reasoning accuracy on math benchmarks without requiring online rollouts.

This paper introduces TESSY, a teacher-student cooperative framework for fine-tuning reasoning models that generates on-policy SFT data by decoupling generation into capability tokens (from teacher) and style tokens (from student), addressing catastrophic forgetting issues when using off-policy teacher data.

OpenAI proposes Teacher–Student Curriculum Learning (TSCL), a framework where a Teacher algorithm automatically selects subtasks for a Student to learn complex tasks, optimizing based on learning curve slope and preventing forgetting. The approach matches or surpasses hand-crafted curricula on decimal addition and Minecraft navigation tasks, enabling solutions previously impossible with direct training.