When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Summary
This paper investigates how on-policy distillation can cause length inflation due to EOS token mismatches between student and teacher models, and proposes a correction method by aggregating EOS probabilities to reduce response length.
View Cached Full Text
Cached at: 09/18/26, 11:01 AM
Paper page - When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Source: https://huggingface.co/papers/2609.20511
https://huggingface.co/papers/2609.20511#%F0%9F%9A%A8-your-opd-run-may-be-penalizing-the-model-for-stopping🚨 Your OPD Run May Be Penalizing the Model for Stopping
In one Qwen3 rollout, the student reaches the correct answer after1,094 tokens, then generates7,098 redundant tokens. We investigate how**on-policy distillation (OPD)**can produce this failure.
🔍**The mechanism:**base students and post-trained teachers can favor different EOS tokens, even within the same declared stopping set. Sampled-token OPD can then suppress the student’s preferred EOS without reliably transferring the teacher’s alternative.
💡**The correction:**aggregate probabilities over functionally equivalent EOS tokens and supervise them as one semantic stopping action. Matching the decoding stopping sets alone is insufficient.
📉**The evidence:**this correction substantially reduces response length and truncation in our Qwen3, Llama, and Gemma experiments on mathematical reasoning.
🧩**Beyond EOS mismatch:**with a pretrained K2-Horizon student, length inflation reappears later in training even after the correction, leaving additional dynamics to explain.
💬 Have you observed similar termination failures in your OPD experiments? We welcome comparisons and discussion!
🔗Project page· 💻Code
Similar Articles
Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation
This paper identifies a failure mode in multi-teacher on-policy distillation for agentic language models where the student becomes over-reliant on tool-calling, and proposes Soft Clamp, a per-token divergence calibration method that reduces over-calling without sacrificing accuracy.
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.
On-Policy Distillation (5 minute read)
This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.
@VukRosic99: When a small model learns from a big one, half the lesson is wasted The setup: a small "student" model writes an answer…
The paper identifies position bias in on-policy distillation for language models, where later tokens in student-generated answers receive degraded supervision. The proposed Importance-Weighted On-Policy Distillation (IW-OPD) weights corrections based on accumulated drift, improving learning speed and final performance.
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.