@furongh: RL vs. distillation may be a false dichotomy. A post-training abstraction: RL as a compiler for supervision. Use policy…

RL vs. distillation may be a false dichotomy. A post-training abstraction: RL as a compiler for supervision. Use policy optimization to derive where the model should go. Compile the solution into dense targets. Let distillation do the training. Introducing β-OPSD

1/ On-policy self-distillation sounds ideal for reasoning: The model explores its own trajectories. A privileged teacher supervises the states it actually visits. But vanilla OPSD makes one aggressive assumption: Move all the way to the teacher. Now.

That can be brittle.

2/ Our key observation: Vanilla OPSD already sits inside policy optimization: it is exactly the β=1 point of a broader KL-regularized family. β controls how strongly the student stays anchored to its reference policy. A fixed recipe becomes a controllable continuum.

3/

One line of math reveals the geometry:

π*β ∝ πref^(1−1/β) · pteacher^(1/β) β → ∞: stay near the reference. β = 1: match the teacher. The target is no longer just a destination.

It is a path from the student to the teacher.

4/

This changes the recipe.

Vanilla OPSD asks the student to teleport to the teacher distribution.

β-OPSD lets it walk there.

We interpolate reference and teacher logits, then gradually increase teacher influence:

a curriculum over distributions, not examples.

5/

Here is the part I find most interesting:

Once policy optimization tells us what the optimal target should be, we do not need to optimize the RL objective directly. We distill toward its tractable approximation.

Policy optimization specifies where to go. Distillation provides a stable and inexpensive way to get there.

6/

A credit-assignment problem is also hiding inside distillation.

An early reasoning token changes every later prefix. Yet standard OPSD judges it only by its immediate teacher mismatch.

That is myopic.

β-OPSD propagates future mismatch backward with return-to-go.

7/

The brittleness is visible in the numbers.

Qwen3-1.7B, AIME 2024 avg@12:

Base: 50.0 Vanilla OPSD: 44.2 β-OPSD: 53.3, 9.16pp improvement over Vanilla OPSD

Across AIME24, AIME25, and HMMT25, β-OPSD gains +5.74 points over vanilla OPSD and beats GRPO on average.

8/

Both components matter in our ablations.

A smoother target improves AIME24/25 and HMMT25 by +6.03/+5.30/+1.67 points. Replacing local token credit with return-to-go adds +1.12/+5.55/+3.61 points.

The target path and future-aware credit are complementary.

9/

The lesson is not “distillation beats RL.”

We may be organizing post-training around the wrong dichotomy.

RL gives us objectives, geometry, and credit assignment. Distillation gives us dense supervision and stable optimization.

The next wave may deliberately compose both.

10/

Not every RL objective will have a closed-form solution.

But whenever its optimal policy can be characterized—or approximated—we should ask:

Can we distill the solution instead of directly optimizing the objective?

I suspect this pattern will recur far beyond OPSD.

11/

The recipe:

policy objective → target distribution → dense supervision → stable training.

Derive with policy optimization. Train with self-distillation. https://arxiv.org/abs/2607.28582

Joint work with @JimmyXu147587, @minghuiliu95, @juzheng_z, and @tomgoldsteincs.