When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

Hugging Face Daily Papers Papers

Summary

This paper investigates how on-policy distillation can cause length inflation due to EOS token mismatches between student and teacher models, and proposes a correction method by aggregating EOS probabilities to reduce response length.

We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify termination-token mismatch between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, the two models can place their stopping probability on different EOS tokens, even when their declared stopping sets are identical. This mismatch can suppress the student's preferred termination action without reliably transferring the teacher-preferred alternative. We show that aligning the decoding stopping set alone is insufficient, while treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. To further understand how termination behavior evolves over training, we study OPD across different K2-Horizon training stages. This stage-wise analysis shows that termination preferences can shift substantially during training, while also revealing a distinct length inflation late in the OPD run that persists beyond termination alignment. Together, these results identify termination mismatch as an important, but not exhaustive, source of OPD length dynamics. We release an implementation incorporating the proposed termination-handling corrections.
Original Article
View Cached Full Text

Cached at: 09/18/26, 11:01 AM

Paper page - When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

Source: https://huggingface.co/papers/2609.20511

https://huggingface.co/papers/2609.20511#%F0%9F%9A%A8-your-opd-run-may-be-penalizing-the-model-for-stopping🚨 Your OPD Run May Be Penalizing the Model for Stopping

In one Qwen3 rollout, the student reaches the correct answer after1,094 tokens, then generates7,098 redundant tokens. We investigate how**on-policy distillation (OPD)**can produce this failure.

🔍**The mechanism:**base students and post-trained teachers can favor different EOS tokens, even within the same declared stopping set. Sampled-token OPD can then suppress the student’s preferred EOS without reliably transferring the teacher’s alternative.

💡**The correction:**aggregate probabilities over functionally equivalent EOS tokens and supervise them as one semantic stopping action. Matching the decoding stopping sets alone is insufficient.

📉**The evidence:**this correction substantially reduces response length and truncation in our Qwen3, Llama, and Gemma experiments on mathematical reasoning.

🧩**Beyond EOS mismatch:**with a pretrained K2-Horizon student, length inflation reappears later in training even after the correction, leaving additional dynamics to explain.

💬 Have you observed similar termination failures in your OPD experiments? We welcome comparisons and discussion!

🔗Project page· 💻Code

Similar Articles

Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation

arXiv cs.CL

This paper identifies a failure mode in multi-teacher on-policy distillation for agentic language models where the student becomes over-reliant on tool-calling, and proposes Soft Clamp, a per-token divergence calibration method that reduces over-calling without sacrificing accuracy.

On-Policy Distillation (5 minute read)

TLDR AI

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Hugging Face Daily Papers

This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.