Paper page - Distilled Reinforcement Learning for LLM Post-training

Source: https://huggingface.co/papers/2607.17247

https://huggingface.co/papers/2607.17247#distilled-reinforcement-learning-for-llm-post-trainingDistilled Reinforcement Learning for LLM Post-Training

Standard reinforcement learning relies on coarse-grained outcome rewards, while on-policy distillation usually encourages the student to imitate the teacher distribution unconditionally. Distilled RL instead uses the teacher to redistribute the policy-gradient signal at the token level, providing selective and fine-grained guidance while preserving reward-driven optimization.

https://huggingface.co/papers/2607.17247#overviewOverview

Distilled RL consists of three components:

1. Reverse importance sampling, which measures the teacher’s relative preference for each student-generated token.
2. Negative sample reset, which disables teacher reweighting on negative-advantage trajectories.
3. Sequence-level geometric normalization, which removes sequence-level scale bias while preserving relative token preferences.

https://huggingface.co/papers/2607.17247#methodMethod

Given a prompt q and a response o\_i sampled from the old student policy, the standard policy ratio is

ri,t(θ)=πθ(oi,t∣q,oi,1:t−1)πold(oi,t∣q,oi,1:t−1)
r_{i,t}(\theta)
=
\frac{
\pi_{\theta}(o_{i,t} \mid q, o_{i,1:t-1})
}{
\pi_{\mathrm{old}}(o_{i,t} \mid q, o_{i,1:t-1})
}

The response-level advantage is estimated using group-normalized rewards:

Ai=Ri−mean({Rj}j=1G)std({Rj}j=1G).
A_i
=
\frac{
R_i - \mathrm{mean}(\{R_j\}_{j=1}^{G})
}{
\mathrm{std}(\{R_j\}_{j=1}^{G})
}.

https://huggingface.co/papers/2607.17247#reverse-importance-samplingReverse Importance Sampling

We measure the teacher’s relative preference for each student-generated token using

ρi,t=πteacher(oi,t∣q,oi,1:t−1)πθold(oi,t∣q,oi,1:t−1).
\rho_{i,t}
=
\frac{
\pi_{\mathrm{teacher}}(o_{i,t} \mid q, o_{i,1:t-1})
}{
\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,1:t-1})
}.

To prevent extreme teacher–student likelihood ratios, we apply symmetric clipping:

ρˉi,t=clip(ρi,t,ϵρ−1,ϵρ).
\bar{\rho}_{i,t}
=
\mathrm{clip}
\left(
\rho_{i,t},
\epsilon_{\rho}^{-1},
\epsilon_{\rho}
\right).

https://huggingface.co/papers/2607.17247#sequence-level-geometric-normalizationSequence-Level Geometric Normalization

The clipped ratios are normalized within each response:

ρ~i,t=ρˉi,texp⁡(1∣oi∣∑s=1∣oi∣log⁡ρˉi,s).
\widetilde{\rho}_{i,t}
=
\frac{
\bar{\rho}_{i,t}
}{
\exp
\left(
\frac{1}{|o_i|}
\sum_{s=1}^{|o_i|}
\log \bar{\rho}_{i,s}
\right)
}.

The normalized ratios satisfy

(∏t=1∣oi∣ρ~i,t)1/∣oi∣=1.
\left(
\prod_{t=1}^{|o_i|}
\widetilde{\rho}_{i,t}
\right)^{1/|o_i|}
=
1.

This normalization removes the sequence-level mean shift in log importance ratios while preserving the teacher’s relative preferences across tokens.

https://huggingface.co/papers/2607.17247#negative-sample-resetNegative Sample Reset

Teacher guidance is applied only to positive-advantage responses:

wi,t={ρ~i,t,Ai>0,1,Ai≤0.
w_{i,t}
=
\begin{cases}
\widetilde{\rho}_{i,t}, & A_i > 0, \\
1, & A_i \leq 0.
\end{cases}

For negative-advantage responses, the update reduces to the original RL objective.

https://huggingface.co/papers/2607.17247#distilled-rl-objectiveDistilled RL Objective

For responses sampled from the old student policy, the final policy optimization objective is

JDistilledRL(θ)=E[1G∑i=1G1∣oi∣∑t=1∣oi∣min⁡(ri,t(θ)wi,tAi,r^i,t(θ)wi,tAi)],
\mathcal{J}_{\mathrm{DistilledRL}}(\theta)
=
\mathbb{E}
\left[
\frac{1}{G}
\sum_{i=1}^{G}
\frac{1}{|o_i|}
\sum_{t=1}^{|o_i|}
\min
\left(
r_{i,t}(\theta) w_{i,t} A_i,
\hat{r}_{i,t}(\theta) w_{i,t} A_i
\right)
\right],

where the clipped policy ratio is

r^i,t(θ)=clip(ri,t(θ),1−ϵlow,1+ϵhigh).
\hat{r}_{i,t}(\theta)
=
\mathrm{clip}
\left(
r_{i,t}(\theta),
1-\epsilon_{\mathrm{low}},
1+\epsilon_{\mathrm{high}}
\right).

Unlike KL-based on-policy distillation, Distilled RL does not treat the teacher as an unconditional imitation target. Instead, the teacher selectively redistributes the reward-driven policy-gradient signal at the token level.

https://huggingface.co/papers/2607.17247#main-resultsMain Results

We evaluate Distilled RL on three student models using Qwen3-8B-GRPO as the teacher. The table below reports the average Pass@1 over ten mathematical reasoning benchmarks.

Student ModelBaseOPDRLOPD+RLDistilled RLDeepSeek-R1-Distill-Qwen-1.5B31.7035.2736.8636.5440.00Qwen3-1.7B39.8645.2144.7644.8946.37Qwen3-4B46.3355.9757.4056.3858.96 Distilled RL consistently improves over standard RL, OPD, and their direct combination across different student scales and teacher–student settings.

https://huggingface.co/papers/2607.17247#requirementsRequirements

https://huggingface.co/papers/2607.17247#softwareSoftware

Clone the repository:

git clone https://github.com/597358816/Distilled-RL.git
cd Distilled-RL

Install the required dependencies:

pip install torch==2.6.0 torchaudio==2.6.0 torchvision==0.21.0 vllm==0.8.3 transformers==4.51.2
pip install ray==2.48.0 tensordict==0.9.1 pydantic==2.11.7
pip install flash-attn
pip install -e .
pip install tensorboard
cd examples
bash XX.sh