Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation

Hugging Face Daily Papers Papers

Summary

This paper identifies a failure mode in classifier-free guidance distillation called Negative Branch Asymmetry, where errors in the positive and negative CFG branches cancel out, and proposes Positive-Direction Matching to supervise branches separately for more robust distilled models.

On-policy distillation (OPD) adapts diffusion models by querying a teacher along trajectories generated by the current student, but how it should behave under classifier-free guidance (CFG), a default component of modern diffusion systems, remains poorly understood. Existing OPD methods naturally extend velocity matching to the CFG-composed prediction, directly matching teacher and student guided velocities. We show that this objective is under-identified at the branch level: positive- and negative-branch errors can compensate in the guided prediction. Through two contrasting cases, we find that naive matching remains effective under shared negative conditioning, where both branch errors decrease jointly. When the model's native CFG schema retains privileged information in the teacher's negative branch that is unavailable to the student, however, this joint reduction breaks down and the composed objective induces antagonistic branch-error dynamics, reducing the positive-branch error while increasing the negative-branch error. We term this failure mode Negative Branch Asymmetry (NBA). To address NBA, we introduce Positive--Direction Matching (PDM), a branch-aware OPD objective that separately constrains the positive prediction and the CFG conditional direction. We apply PDM to dense-to-sparse video control, where naive guided matching is highly sensitive to inference guidance scales, while branch-aware supervision enables more robust and effective knowledge transfer.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:33 AM

Paper page - Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation

Source: https://huggingface.co/papers/2607.24731 What happens when on-policy diffusion distillation meets classifier-free guidance?

A natural approach is to match the teacher’s and student’s guided predictions. But this seemingly reasonable objective hides a subtle ambiguity: errors from the positive and negative CFG branches can cancel each other, making the guided output look correct even when the individual branches are not.

We find that this ambiguity is usually harmless when teacher and student share the same negative conditioning. However, when the teacher retains privileged information in its negative branch, the two branches can start moving in opposite directions during training. We call this failure modeNegative Branch Asymmetry (NBA).

Our solution,Positive–Direction Matching (PDM), supervises the positive prediction and the CFG conditional direction separately. This removes cross-branch error compensation and makes the distilled model substantially more robust to changes in inference guidance scale.

We study this behavior across image-domain diagnostics and dense-to-sparse video control, including pose, depth, and scribble conditions.

We hope this work offers a useful perspective: when distilling CFG-based models, matching the final guided prediction may not be enough—the branches themselves matter.

Similar Articles