Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation

# Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation
Source: [https://arxiv.org/html/2605.12741](https://arxiv.org/html/2605.12741)
Yuwei Zhang1Sha Li2Changlong Yu2Qin Lu2Shuowei Jin2Chengyu Dong2 Haoran Liu1Ilgee Hong311footnotemark:1Xintong Li111footnotemark:1Zhenyu Shi2Bing Yin2Jingbo Shang1 UC San Diego1Amazon2Georgia Institute of Technology3 \{yuz163,jshang\}@ucsd\.edu

###### Abstract

Enabling Large Language Models \(LLMs\) to continuously improve from environmental interactions is a central challenge in post\-training\. While on\-policy self\-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal\. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare\-success regimes\. To bridge this gap, we introduce Reflection\-Enhanced Self\-Distillation \(Resd\), a framework that transforms raw failure feedback into an active source of corrective supervision\. Instead of passively appending feedback,Resdinterprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps\. The enriched context enables the self\-teacher to provide actionable token\-level supervision even in the absence of successful rollouts\. Empirical evaluations on multiple continual learning tasks demonstrate thatResdsubstantially outperforms standard self\-distillation baselines\. Furthermore,Resdachieves significantly faster early\-stage improvement than GRPO with8×8\\timessamples using only a single rollout per prompt, highlighting its superior interaction efficiency\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.12741v1/x1.png)Figure 1:Resdimproves interaction efficiency during training\. The x\-axis is the number of samples\.A fundamental challenge in the post\-training of Large Language Models \(LLMs\) is enabling continuous improvement through environmental interactions\. Traditionally, Reinforcement Learning \(RL\) algorithms, such as PPO or GRPO, have been the standard paradigm for aligning models with desired outcomes\. However, a critical bottleneck in deploying RL for complex, multi\-step tasks is the reliance on sparse reward signals\. In scenarios where successful trajectories are exceedingly rare, standard RL often struggles to effectively bootstrap policies because it lacks dense supervision to guide the model through a vast exploration space\.

To mitigate the sparse reward problem, On\-Policy Distillation \(OPD\) has emerged as a promising alternative\(Agarwalet al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib14); Lu and Lab,[2025](https://arxiv.org/html/2605.12741#bib.bib3); Xiaoet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib33)\)\. Rather than relying on a single scalar reward at the end of a rollout, OPD leverages a stronger or privileged teacher model to compute target probabilities for every single token in the generated trajectory\. This mechanism elegantly translates sparse trajectory\-level outcomes into dense, token\-level learning signals\. However, standard OPD requires maintaining a separate expert model, which incurs high computational costs and risks distribution mismatch between the teacher and the student\. To address this, self\-distillation variants like SDPO instantiate the teacher from the model itself\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1); Zhaoet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib2)\)\. By conditioning the self\-teacher on environmental feedback, SDPO eliminates the need for an external oracle while perfectly aligning the supervisory signal with the model’s own generation distribution\.

While self\-distillation resolves both reward sparsity and distribution mismatch, its success hinges entirely on the quality of the self\-teacher’s supervision\. Current methods predominantly treat environmental feedback as a static, passive conditioning variable\.*We argue that this overlooks a key design choice: the structural representation of the feedback itself\.*In sparse\-reward regimes where successful demonstrations are absent, the teacher is forced to derive supervision almost entirely from failed rollouts\. We demonstrate in Figure[1](https://arxiv.org/html/2605.12741#S1.F1)that simply feeding raw failure outcomes or unstructured environment errors into the teacher context yields only marginal student improvements\.

Motivated by this observation, we propose Reflection\-Enhanced Self\-Distillation \(Resd\), a framework that enriches the teacher context with actively interpreted feedback\. For each failed rollout,Resdfirst generates a retrospective reflection that identifies the likely cause of failure and the correction that would have avoided it\. It then organizes recurring lessons into a persistent playbook, allowing future teacher prompts to reuse feedback\-derived knowledge across training steps\. The resulting enriched teacher context transforms raw failure feedback from a passive diagnostic signal into an actionable and reusable source of corrective supervision, enabling self\-distillation to improve even when successful demonstrations are unavailable\.

Empirically, we evaluateResdon tasks designed to reflect continuous learning beyond common post\-training distributions\. These tasks are novel to the model, often begin in rare\-success regimes, and provide rich execution feedback despite receiving sparse binary rewards\. Across these settings,Resdsubstantially improves over standard self\-distillation baselines and achieves faster early\-stage improvement than GRPO while using only a single rollout per prompt\. These results suggest that structured feedback enables sample\-efficient bootstrapping from failures, while reward\-based optimization remains a complementary tool once sufficient successful rollouts are available\. Our contributions are as follows:

- •We identify*feedback formulation*as a critical design axis for on\-policy self\-distillation especially in sparse\-reward regimes\.
- •We propose*Reflection\-Enhanced Self\-Distillation*\(Resd\), which enriches the teacher context with retrospective reflections and a persistent playbook\. By interpreting failures and preserving recurring lessons across training steps,Resdturns raw feedback from a passive diagnostic signal into reusable corrective supervision\. Empirically, we characterize the sample\-efficiency of feedback\-enhanced self\-distillation under an online streaming protocol\.
- •More broadly, by revealing that self\-improving models are highly sensitive to how feedback is structured, we position our reflection and playbook mechanisms as a plug\-and\-play module that can readily enhance other self\-distillation objectives\.

## 2Preliminaries

Recent advancements such as Self\-Distillation Policy Optimization \(SDPO\)\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1)\)and On\-Policy Self\-Distillation \(OPSD\)\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib2)\)introduce a “self\-teacher” paradigm that replaces sparse outcomes in Reinforcement Learning with dense, token\-level supervisions without the need for a massive separate teacher model\. In these frameworks, a “self\-teacher” policy is typically instantiated as the model’s current or moving\-average weightsθold\\theta\_\{\\text\{old\}\}, operating in a privileged context\. The teacher is conditioned on both the intermediate statest=\(xt,y<t\)s\_\{t\}=\(x\_\{t\},y\_\{<t\}\)and the retrospective feedbackc​\(x,y\)c\(x,y\), producing a corrected, feedback\-informed distributionπθold\(⋅\|x,y<t,c\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|x,y\_\{<t\},c\)\. The base policyπθ\\pi\_\{\\theta\}is then optimized to match this privileged distribution at every decoding step\. We formulate this matching objective universally using*f\-divergences*\. For a given time steptt, let the likelihood ratio for any tokenv∈𝒱v\\in\\mathcal\{V\}be defined asτvt=πθold​\(v\|x,y<t,c\)πθ​\(v\|x,y<t\)\\tau^\{t\}\_\{v\}=\\frac\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(v\|x,y\_\{<t\},c\)\}\{\\pi\_\{\\theta\}\(v\|x,y\_\{<t\}\)\}\. To prevent extreme ratios from destabilizing training, we clip the likelihood ratio to a bounded range:

τ~vt=clip⁡\(τvt,ϵmin,ϵmax\)\\tilde\{\\tau\}^\{t\}\_\{v\}=\\operatorname\{clip\}\\\!\\left\(\\tau^\{t\}\_\{v\},\\;\\epsilon\_\{\\min\},\\;\\epsilon\_\{\\max\}\\right\)The unified self\-distillation objective then minimizes the expected f\-divergence across all tokenst∈\[1,T\]t\\in\[1,T\]:

ℒSD​\(θ\)=𝔼x,y∼πθ,f​\[∑t=1T∑v∈𝒱πθ​\(v\|x,y<t\)⋅f​\(τ~vt\)\]\\mathcal\{L\}\_\{\\text\{SD\}\}\(\\theta\)=\\mathbb\{E\}\_\{x,y\\sim\\pi\_\{\\theta\},f\}\\left\[\\sum\_\{t=1\}^\{T\}\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\|x,y\_\{<t\}\)\\cdot f\(\\tilde\{\\tau\}^\{t\}\_\{v\}\)\\right\]\(1\)wheref​\(τ\)f\(\\tau\)is a convex function\. By selecting differentff, we can recover various divergence measures, such as forward KL, reverse KL and Jensen\-Shannon Divergence \(JSD\), each offering unique optimization properties\. We provide a detailed discussion on the choice offfand its implications in Appendix[A](https://arxiv.org/html/2605.12741#A1)\. Notice that, since we sample trajectories from the base policyπθ\\pi\_\{\\theta\}, the self\-distillation objective is inherently on\-policy\. Such a formulation allows the model to recover forgotten behaviors after midtraining\(Shenfeldet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib4); Chenet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib5); Shenfeldet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib6); Laiet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib7)\)and prevent from exposure bias\(Lu and Lab,[2025](https://arxiv.org/html/2605.12741#bib.bib3); Agarwalet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib8)\)\.

## 3Method

![Refer to caption](https://arxiv.org/html/2605.12741v1/x2.png)Figure 2:Overview of theResdframework\. The student generates a rollout and receives environment feedback\. On failure, a local self\-reflection diagnoses the error and a global playbook curation step distills reusable lessons into a persistent playbook\. The enriched context \(reflection, curated playbook, and any cached solutions\) is fed to the teacher prompt, whose output distribution provides token\-level supervision to the student via a KL loss\.Our work builds on the finding that existing self\-distillation methods rely heavily on successful demonstrations, and their supervision degrades when successful rollouts are scarce\. We propose Reflection\-Enhanced Self\-Distillation \(Resd\), which enriches the teacher’s context with retrospective feedback from failures and a persistent memory of past experiences, enabling corrective supervision even without successful demonstrations\. We first motivate the problem by analyzing SDPO’s failure mode under sparse rewards \(§[3\.1](https://arxiv.org/html/2605.12741#S3.SS1)\), then detail the retrospective reflection and playbook curation mechanism \(§[3\.2](https://arxiv.org/html/2605.12741#S3.SS2)\), and describe how these components are integrated into a memory\-augmented self\-distillation framework \(§[3\.3](https://arxiv.org/html/2605.12741#S3.SS3)\)\.

### 3\.1Self\-Distillation Relies on Successful Peer Demonstrations

The privileged context of the SDPO self\-teacher comprises two complementary signals: environment feedback on the current trajectory and successful peer demonstrations sampled within the same rollout batch\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1)\)\. To disentangle their contributions, we conduct an ablation that varies the rollout batch sizeNNwhile keeping all other hyperparameters fixed\. WhenN=1N\{=\}1, each prompt group samples a single rollout, eliminating the possibility of peer demonstrations entirely and leaving environment feedback as the sole supervisory signal\. As shown in Figure[3](https://arxiv.org/html/2605.12741#S3.F3), SDPO’s performance degrades substantially in this regime, indicating that the teacher struggles to translate failure feedback alone into effective corrective distributions\. This failure mode becomes particularly severe in rare\-success regime\. If the per\-rollout success rate is small, most rollout groups will contain only failures and provide no positive counterfactual demonstration, forcing the teacher to construct token\-level supervision from raw failure feedback alone which will result in less performance improvements through training\.

### 3\.2From Passive Feedback Exposure to Active Feedback Understanding

![Refer to caption](https://arxiv.org/html/2605.12741v1/x3.png)Figure 3:Per\-task accuracy onFinerunder varying rollout batch sizes\. SDPO \(N=1N\{=\}1\) degrades without peer demonstrations, while adding reflection and playbook curation \(SDPO\+Ref,N=1N\{=\}1\) recovers and surpasses theN=8N\{=\}8baseline\.The degradation observed in theN=1N\{=\}1setting suggests that environment feedback alone is not sufficient to support effective self\-distillation\. This is somewhat surprising, as the teacher is still given privileged information about the student’s failed trajectory, and in principle should be able to use this information to produce a more corrective distribution\.*We argue that the issue lies not in the absence of feedback but in how feedback is represented and used\.*Existing self\-distillation methods often expose the teacher to environment feedback as additional context, but leave the interpretation of that feedback entirely implicit\. In rare\-success regimes, this passive use of feedback becomes a bottleneck, raw failure signals are often diagnostic rather than instructional, indicating that a trajectory failed without explaining which decision caused the failure\.

This motivates a shift from passive feedback exposure to active feedback understanding\. For feedback to become useful supervision, the teacher must perform two additional operations\.\(a\)It must retrospectively interpret the failed trajectory, connecting the final feedback signal to the intermediate reasoning that caused the failure\.\(b\)It must preserve useful lessons across trajectories, since many failures are repeated manifestations of the same hidden rule rather than independent mistakes\. In this view, raw feedback is only a starting point, and*it must be transformed into causal explanations and persistent knowledge before it can reliably guide token\-level distillation*\. As shown by the yellow curve in Figure[3](https://arxiv.org/html/2605.12741#S3.F3), equipping the teacher with only these two failure\-derived components \(without access to any successful demonstrations yet\) already recovers much of the performance lost in theN=1N\{=\}1regime\. Further analysis on the distribution of per\-prompt accuracy \(i\.e\. we sample44times for each prompt and measure the proportion of correct trials\) in Figure[4](https://arxiv.org/html/2605.12741#S3.F4)shows that the performance improvement of SDPO \(N=8N\{=\}8\) mostly comes from increasing the number of completely correct prompts, while SDPO\+Ref \(N=1N\{=\}1\) significantly reduces the number of completely incorrect prompts, indicating that reflections help the model learn from failures\.

### 3\.3Reflection\-Enhanced Self\-Distillation \(Resd\)

![Refer to caption](https://arxiv.org/html/2605.12741v1/x4.png)\(a\)SDPO \(N=8\)\.
![Refer to caption](https://arxiv.org/html/2605.12741v1/x5.png)\(b\)SDPO\+Ref \(N=1\)\.

Figure 4:Per\-prompt mean accuracy distribution onFinerover training steps\. All\-wrong cases’ proportion decreases better for SDPO\+Ref\.Building on the above observation, we introduce Reflection\-Enhanced Self\-Distillation \(Resd\), a framework that operationalizes active feedback understanding within the self\-distillation loop\.Resdmaintains two forms of persistent context: a playbook𝒫t\\mathcal\{P\}\_\{t\}, following the broader idea from\(Zhanget al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib9)\), that stores reusable lessons distilled from previous failures, and an optional solution bufferℬ\\mathcal\{B\}that caches successful trajectories when they are available\. At each training step,Resdfirst updates this context using the outcome of the current rollout, and then conditions the teacher on the enriched context to produce token\-level supervision\. This design allows failed trajectories to improve future teacher prompts, rather than being used only as one\-shot raw feedback\.

Retrospective Reflection on Failures\.For a failed trajectory\(x,y\)\(x,y\)with environment feedbackc​\(x,y\)c\(x,y\),Resdinvokes the student model in a diagnostic mode to generate a retrospective reflection\. Formally, given the current playbook𝒫t\\mathcal\{P\}\_\{t\}, the reflector outputs

r,\{ℓt,j\}j=1\|𝒫t\|=Reflect​\(x,y,c,𝒫t;θ\),r,\\\{\\ell\_\{t,j\}\\\}\_\{j=1\}^\{\|\\mathcal\{P\}\_\{t\}\|\}=\\textsc\{Reflect\}\(x,y,c,\\mathcal\{P\}\_\{t\};\\,\\theta\),whererris a natural language explanation of the failure andℓt,j∈\{helpful,harmful,neutral\}\\ell\_\{t,j\}\\in\\\{\\texttt\{helpful\},\\texttt\{harmful\},\\texttt\{neutral\}\\\}indicates whether the existing playbook entrypj∈𝒫tp\_\{j\}\\in\\mathcal\{P\}\_\{t\}helped or misled the current attempt\. The reflection is designed to make the raw feedback more actionable by connecting it to specific reasoning steps\. The entry\-level tags provide a lightweight usage signal for maintaining the playbook over time\.

Playbook Curation from Past Experiences\.The playbook𝒫t=\{p1,…,pM\}\\mathcal\{P\}\_\{t\}=\\\{p\_\{1\},\\dots,p\_\{M\}\\\}is a persistent set of natural\-language entries that encode reusable lessons\. Given a reflectionrr, the curator generates new entries to add:

𝒫t\+1=𝒫t∪Curate​\(𝒫t,r;θ\)\\mathcal\{P\}\_\{t\+1\}=\\mathcal\{P\}\_\{t\}\\cup\\textsc\{Curate\}\(\\mathcal\{P\}\_\{t\},r;\\,\\theta\)whereπθ\\pi\_\{\\theta\}is prompted with the current playbook and the reflection, and produces a set of non\-redundant entries that capture lessons not already present in𝒫t\\mathcal\{P\}\_\{t\}\. Each entrypjp\_\{j\}maintains accumulated counts\(hj,dj\)\(h\_\{j\},d\_\{j\}\)of helpful and harmful tags received from the reflector across steps\. Before each context update, aConciseoperation prunes the playbook by removing entries wheredj≥hjd\_\{j\}\\geq h\_\{j\}\(net harmful\) and, if\|𝒫t\|\|\\mathcal\{P\}\_\{t\}\|exceeds a budgetMmaxM\_\{\\max\}, evicting the least recently tagged entries\. This lifecycle ensures the playbook remains compact and converges toward entries that are repeatedly useful across trajectories\. We detail the fullConcisestrategy, including trigger conditions and two removal variants, in Appendix[D](https://arxiv.org/html/2605.12741#A4)\.

Interleaving Context Update with Policy Optimization\.Algorithm[1](https://arxiv.org/html/2605.12741#alg1)summarizes the fullResdtraining loop\. At each steptt, the studentπθ\\pi\_\{\\theta\}generates a rollouty∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)and receives environment feedbackc​\(x,y\)c\(x,y\)and rewardR​\(x,y\)R\(x,y\)to judge the correctness\. Before the policy update, a context update phase modifies the persistent state:Concisefirst prunes𝒫t\\mathcal\{P\}\_\{t\}, then, if the rollout failed,ReflectandCurateextend the playbook to𝒫t\+1\\mathcal\{P\}\_\{t\+1\}; if it succeeded, the solution buffer is updatedℬ​\(x\)←y\\mathcal\{B\}\(x\)\\leftarrow yand setsr=∅r=\\emptyset\. The teacher model is then synced with student model with an EMA update\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1)\), and conditioned on the enriched context to produce token\-level supervision:

πθold\(⋅∣x,y<t,c,r,𝒫t\+1,ℬ\(x\)\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid x,y\_\{<t\},c,r,\\mathcal\{P\}\_\{t\+1\},\\mathcal\{B\}\(x\)\)and the student minimizes the self\-distillation objectiveℒSD​\(θ\)\\mathcal\{L\}\_\{\\text\{SD\}\}\(\\theta\)\(Eq\.[1](https://arxiv.org/html/2605.12741#S2.E1)\)\. We additionally apply a per\-sample weight to enhance the gradient contribution of successful samples, computed as a function of the batch success rate \(details in Appendix[B](https://arxiv.org/html/2605.12741#A2)\)\. Because𝒫\\mathcal\{P\}andℬ\\mathcal\{B\}persist across steps, the supervision available to the teacher can improve over time even when the current batch contains no successful demonstration\.

Algorithm 1Reflection\-Enhanced Self\-Distillation \(Resd\)0:Student policy

πθ\\pi\_\{\\theta\}, teacher weights

θold\\theta\_\{\\text\{old\}\}, playbook

𝒫←∅\\mathcal\{P\}\\leftarrow\\emptyset, solution buffer

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
1:foreach training stepdo

2:// Rollout & Reward

3:Sample rollout

y∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)
4:Obtain environment feedback

c​\(x,y\)c\(x,y\)and reward

R​\(x,y\)R\(x,y\)
5:// Context Update

6:

𝒫←Concise​\(𝒫\)\\mathcal\{P\}\\leftarrow\\textsc\{Concise\}\(\\mathcal\{P\}\)⊳\\trianglerightprune stale/harmful entries

7:if

R​\(x,y\)<thresholdR\(x,y\)<\\text\{threshold\}then

8:

r←Reflect​\(x,y,c,𝒫;θ\)r\\leftarrow\\textsc\{Reflect\}\(x,y,c,\\mathcal\{P\};\\,\\theta\)⊳\\trianglerightgenerate reflection and tag playbook entries

9:

𝒫←Curate​\(𝒫,r;θ\)\\mathcal\{P\}\\leftarrow\\textsc\{Curate\}\(\\mathcal\{P\},r;\\,\\theta\)⊳\\trianglerightgenerate new playbook entries from reflection

10:else

11:

ℬ​\(x\)←y\\mathcal\{B\}\(x\)\\leftarrow y⊳\\trianglerightcache successful solution for future replay

12:endif

13:// Enriched Teacher Prompt Construction

14:Retrieve buffered solution

ℬ​\(x\)\\mathcal\{B\}\(x\)\(if available\)

15:Construct enriched context: playbook

𝒫\\mathcal\{P\}, reflection

rr, previous trial

yy, feedback

cc, solution

ℬ​\(x\)\\mathcal\{B\}\(x\)
16:// Policy Update

17:Compute teacher distribution

πθold\(⋅∣x,y<t,c,r,𝒫,ℬ\(x\)\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid x,y\_\{<t\},c,r,\\mathcal\{P\},\\mathcal\{B\}\(x\)\)
18:Update

θ\\thetaby minimizing

ℒSD​\(θ\)\\mathcal\{L\}\_\{\\text\{SD\}\}\(\\theta\)
19:

θold←θ\\theta\_\{\\text\{old\}\}\\leftarrow\\theta⊳\\trianglerightsync teacher weights

20:endfor

## 4Experiments

Our experiments are designed to address the following questions:

- •DoesResdimprove over self\-distillation baselines especially in rare\-success regimes? \(§[4\.2](https://arxiv.org/html/2605.12741#S4.SS2),§[4\.3](https://arxiv.org/html/2605.12741#S4.SS3)\)
- •How doesResdcompare to reward\-based RL \(GRPO\) in terms of interaction efficiency? \(§[4\.4](https://arxiv.org/html/2605.12741#S4.SS4)\)
- •How does each component ofResdaffect the performance? \(§[4\.5](https://arxiv.org/html/2605.12741#S4.SS5)\)
- •How doesResdinterpret and organize failure feedback at the instance level? \(§[4\.6](https://arxiv.org/html/2605.12741#S4.SS6)\)

We additionally evaluate instruction\-following preservation and training latency in Appendices[E](https://arxiv.org/html/2605.12741#A5)and[F](https://arxiv.org/html/2605.12741#A6)\.

### 4\.1Experimental Setup

Datasets\.We evaluate on four tasks spanning program synthesis and physical reasoning\.Manufactoria\-HasandBouncingSim\-Easy/Mediumare drawn from the RL\-Grok benchmark\(Sunet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib10)\), whileFineris fromZhanget al\.\([2025](https://arxiv.org/html/2605.12741#bib.bib9)\)\. The RL\-Grok tasks are characterized by near\-zero initial success rates, making them a natural testbed for learning from failure feedback\.Finer, by contrast, exhibits a higher initial success rate, allowing us to assess whetherResdalso benefits regimes where successful demonstrations are more accessible\. Table[1](https://arxiv.org/html/2605.12741#S4.T1)summarizes the datasets\. We trainQwen3\-4B\-Thinking\-2507forManufactoria\-Has,BouncingSim\-Easy, andFiner, andQwen3\-30B\-A3B\-Thinking\-2507for the harderBouncingSim\-Medium\.

Table 1:Overview of evaluation tasks\. “Train” and “Test” denote the number of unique problems in each split\.TaskTrainTestDescriptionManufactoria\-Has\(Sunet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib10)\)742132Write DSL programs that check whether an input tape contains a target patternBouncingSim\-Easy\(Sunet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib10)\)640100Write Python code to simulate 2\-D multi\-object bouncing dynamics \(easy\)BouncingSim\-Medium\(Sunet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib10)\)320100Write Python code to simulate 2\-D multi\-object bouncing dynamics \(medium\)Finer\(Zhanget al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib9)\)1000500Tag financial named entities in SEC filings with XBRL labelsSettings\.We adopt an*online streaming*training protocol in which the model makes a single pass over the training data and each training example is seen at most once\. For every incoming batch, the trainer executes an inner loop of up toK=4K=4update iterations on the same set of prompts\. This protocol is designed to mimic practical self\-improvement scenarios, where deployed models continuously encounter new tasks, observe task outcomes, and must update under a limited interaction budget \(N=1N\{=\}1\)\. Unless otherwise specified, we adopt EMA update for teacher model weights with an update rate of0\.00010\.0001\. We optimize reverse\-KL divergence as the self\-distillation loss[1](https://arxiv.org/html/2605.12741#S2.E1)with a lower clipϵmin=0\.2\\epsilon\_\{\\text\{min\}\}=0\.2and no upper clip, under train data batch size of3232\. We provide further implementation details in Appendix[G](https://arxiv.org/html/2605.12741#A7)\.

Metrics and Baselines\.We report both*mean@4*\(m@4\), the average performance over four independently sampled responses per test problem, and*best@4*\(b@4\), the best performance among the four samples\. We use two evaluation metrics:*Per\-Task Accuracy*, which measures the fraction of problems whose response passes all test cases \(equivalent to pass@1 for a single sample, also used as reward function\), and*Per\-Test\-Case Accuracy*, which measures the average fraction of test cases passed across problems\. Since different validation metrics may peak at different training steps, we select a single checkpoint per run using a rank\-based rule: for each logged step, we rank its four validation metrics, sum the ranks, and report the checkpoint with the highest total rank\. The same selection rule is applied uniformly to all methods\. We additionally includeSDPO\+ss, a variant that also distills on the self\-success instead of skipping them \(disabled as a default behavior in SDPO\), providing denser teacher supervision in higher\-success regimes\.

Table 2:Main results on four tasks\.*m@4*and*b@4*denote mean@4 and best@4\. Per\-Task Accuracy is the fraction of problems whose response passes all test cases; Per\-Test\-Case Accuracy is the average fraction of test cases passed\.Manufactoria\-HasBouncingSim\-EasyBouncingSim\-MediumFiner\(Qwen3\-4B\)\(Qwen3\-4B\)\(Qwen3\-30B\-A3B\)\(Qwen3\-4B\)Methodm@4b@4m@4b@4m@4b@4m@4b@4*Per\-Task Accuracy*Base model0\.381\.520\.000\.001\.003\.0039\.2851\.90SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1)\)0\.953\.791\.003\.002\.006\.0040\.9350\.50SDPO\+ss0\.572\.271\.004\.002\.006\.0050\.2559\.12Resd\(ours\)35\.8065\.914\.258\.007\.2510\.0053\.6661\.12*Per\-Test\-Case Accuracy*Base model25\.3160\.9727\.8533\.5323\.7234\.6769\.6579\.21SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1)\)37\.6673\.5528\.9846\.2024\.3834\.4771\.2778\.21SDPO\+ss37\.9771\.9329\.6847\.7324\.9335\.8076\.1582\.21Resd\(ours\)76\.9596\.5338\.8755\.8034\.7241\.0778\.9884\.32

### 4\.2Main Results

Table[2](https://arxiv.org/html/2605.12741#S4.T2)reports the main results across all four tasks\.Resdachieves the best performance across all tasks and metrics\. The gains are largest in the rare\-success regimes\. OnManufactoria\-Has,Resdimproves per\-task accuracy from0\.57/2\.270\.57/2\.27under SDPO\+ss to35\.80/65\.9135\.80/65\.91in m@4/b@4, while also improving per\-test\-case accuracy from37\.97/71\.9337\.97/71\.93to76\.95/96\.5376\.95/96\.53\.Resdalso consistently improves over SDPO and SDPO\+ss on bothBouncingSimtasks, showing that failure\-derived reflection and playbook memory are useful beyond the DSL setting\. OnFiner, where successful demonstrations are more accessible,Resdstill obtains the strongest results, suggesting that structured feedback remains beneficial even outside the most sparse\-success regime\. Full validation training curves for every run in Table[2](https://arxiv.org/html/2605.12741#S4.T2)are provided in Appendix[C](https://arxiv.org/html/2605.12741#A3)\.

### 4\.3Reflection and Playbook Elicit a More Corrective Teacher

![Refer to caption](https://arxiv.org/html/2605.12741v1/x6.png)Figure 5:\(a\)\(b\) Token\-level distillation loss for training step 20\. Each token is shaded from white \(zero loss\) to dark red \(high loss\)\. \(c\)\(d\) Average per\-token distillation loss across training steps forResdand baselines\.We next examine how enriching the teacher prompt changes the self\-distillation signal\. Figure[5](https://arxiv.org/html/2605.12741#S4.F5)plots both per\-token and average per\-token distillation losses over training steps\. \(1\) Across tasks,Resdoften yields a higher self\-distillation loss than the baselines\. Since this loss measures the discrepancy between the student distribution and the feedback\-conditioned teacher distribution, a higher value suggests that the reflection\- and playbook\-augmented teacher provides a distribution that deviates more from the student’s current behavior\. \(2\)Resdhas more loss mass focusing on reasoning decision tokens, like “create”, “solution”, “processes” and “The goal”\. These are points where the models choose what to do next or how to frame the problem, which demonstrate that the teacher is correcting the reasoning strategy\.

### 4\.4Comparison with GRPO

![Refer to caption](https://arxiv.org/html/2605.12741v1/x7.png)\(a\)Manufactoria\-Has
![Refer to caption](https://arxiv.org/html/2605.12741v1/x8.png)\(b\)BouncingSim\-Easy
![Refer to caption](https://arxiv.org/html/2605.12741v1/x9.png)\(c\)BouncingSim\-Medium
![Refer to caption](https://arxiv.org/html/2605.12741v1/x10.png)\(d\)Finer

Figure 6:Validation performance across training steps forResdand GRPO\. GRPO samples88rollouts per prompt, whileResduses a single rollout\.We compareResdwith GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib38)\), a strong on\-policy reinforcement learning baseline, in Figure[6](https://arxiv.org/html/2605.12741#S4.F6)\. The key difference is the interaction pattern: GRPO uses grouped rollouts with group size88, requiring eight sampled trajectories and reward/feedback evaluations per prompt, whereasResduses a single rollout per prompt and updates its teacher context from that trajectory\. We therefore view this comparison as an interaction\-efficiency comparison rather than a strict equal\-rollout\-budget comparison\. Despite using fewer rollouts per prompt,Resdimproves faster than GRPO in the early stage of training across the tasks in Figure[6](https://arxiv.org/html/2605.12741#S4.F6)\. These results suggest thatResdis well suited for sample\-efficient bootstrapping under limited interaction budgets\. We note thatResddoes not always improve monotonically and the validation curves exhibit occasional dips\. OnFiner, the sharp drop around step5050is primarily caused by response truncation\. OnBouncingSim\-Easy, the later\-stage degradation after around step6060suggests that once the model reaches a higher\-success regime, dense self\-distillation from feedback\-derived contexts becomes less stable than reward\-based optimization\. A practical strategy is to useResdto quickly reach a higher\-success regime, then switch to stabler reward\-based optimization such as GRPO once grouped rollouts provide reliable learning signals\.

### 4\.5Ablation Study on Teacher Prompt Components

Table 3:Ablation of teacher prompt components\. We report per\-test\-case accuracy \(m@4 / b@4\)\.Manuf\.\-HasBSim\-EasyFinerMethodm@4b@4m@4b@4m@4b@4Resd\(full\)76\.9596\.5338\.8755\.8078\.9884\.32w/o reflection51\.4179\.2636\.8555\.3377\.7882\.26w/o playbook60\.6592\.3732\.9553\.6077\.4283\.07w/o solution buffer60\.2480\.1031\.9752\.4078\.3683\.42We ablateResd’s enriched teacher context by removing local reflection, persistent playbook, or solution buffer\. Table[3](https://arxiv.org/html/2605.12741#S4.T3)shows that fullResdachieves the strongest performance across all three tasks\. Removing reflection causes the largest degradation onManufactoria\-Has\(76\.95→\\to51\.41 m@4\), while removing the playbook or solution buffer also lowers performance, suggesting that local reflection provides useful short\-term corrections while the playbook preserves reusable lessons and prevents regressions\. The solution buffer further helps convert partial progress into fully correct solutions\.

### 4\.6Case Study: HowResdInterprets Failure Feedback

Table[4](https://arxiv.org/html/2605.12741#S4.T4)tracesResdon a representativeManufactoria\-Hastask across four consecutive inner\-loop updates, illustrating how local environment feedback is progressively organized into reusable task knowledge\. This case highlights two mechanisms that are difficult to see from aggregate accuracy alone\.*First,Resdexposes learning progress before sparse success appears*: the model fixes syntax, then transition handling, then a more subtle edge case involving empty\-tape behavior\.*Second, the playbook accumulates situation\-specific rules*: different failure modes require different corrections \(e\.g\., skipping non\-pattern symbols vs\. rejecting when the tape is empty\), and the playbook records each rule alongside the situation it applies to\. A cached successful exemplar ultimately demonstrates how these rules coexist in a complete working program\. This supports the view that failure feedback becomes useful not merely by being appended to the prompt, but by being organized into persistent, scoped knowledge\.

Table 4:Case study onManufactoria\-Has: “Accept if the tape contains BRBR\.” In this task, the model must write a finite\-state machine program in a domain\-specific language that processes an input tape—a sequence of colored symbols \(e\.g\.,GRBRBRY,BRGBY\)—one symbol at a time from left to right\. Each state reads the next symbol and branches to a successor state depending on the symbol; a tape is accepted by reaching a designated accept state and rejected otherwise\. Here, the target program must accept any tape containing the contiguous substring BRBR \(e\.g\., acceptGRBRBRY, rejectBRGBY\), requiring the model to handle arbitrary surrounding symbols, restart matching after partial mismatches, and correctly reject when the input ends mid\-match\. “acc\.” denotes per\-test\-case accuracy\. Each row corresponds to one inner\-loop update step, showing howResdprogressively diagnoses and corrects errors through reflection and playbook accumulation\. Full generated programs for each step are provided in Appendix[H](https://arxiv.org/html/2605.12741#A8)\.Stepacc\.Failure reasonReflection / playbook update450\.00Program references a non\-existent state, causing a parse errorReflection extracts a syntax rule: all branch targets must reference states that are explicitly declared in the program\.460\.84Incorrectly rejects tapes whose leading symbols are not part of the pattern \(e\.g\., rejectsGBRBRbecause it starts withG, even though BRBR appears later\)Reflection identifies that states0routes a leadingGto reject instead of continuing the search\. Playbook records: non\-pattern symbols before or between matches should be skipped, not rejected\.470\.96Overcorrection: the fix for skipping non\-pattern symbols introduces an infinite loop between two states when the tape becomes empty at a non\-terminal stateReflection identifies that non\-pattern symbols should be skipped when the tape is non\-empty, but the program should reject when the tape is empty at a non\-terminal state\. The playbook records both situation\-specific rules\.481\.00—A successful solution is cached\.

## 5Related Works

On\-policy self\-distillation\.Knowledge distillation is widely used as an alternative to supervised fine\-tuning \(SFT\) when a stronger teacher model is available\. Classical approaches train a student to match the output distribution or representations of a teacher\(Kim and Rush,[2016](https://arxiv.org/html/2605.12741#bib.bib17); Sanhet al\.,[2019](https://arxiv.org/html/2605.12741#bib.bib18); Teamet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib19); Koet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib20)\)\. However, standard distillation relies on fixed data and suffers from student–teacher mismatch, since the student is evaluated on its own generated trajectories\(Agarwalet al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib14); Xuet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib15); Chenet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib16)\)\. To reduce this distribution shift, recent works study on\-policy distillation\(Lu and Lab,[2025](https://arxiv.org/html/2605.12741#bib.bib3); Xiaoet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib33); Yanget al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib32); DeepSeek\-AI,[2026](https://arxiv.org/html/2605.12741#bib.bib34)\), where the student learns from teacher feedback on its own generations\. This aligns the training distribution with the student policy and is closely related to online imitation learning\. Building on this idea, self\-distillation uses a self\-teacher that has access to privileged information to guide the current policy\. The teacher may condition on environment feedback or execution signals\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib1); Yeet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib11)\), or leverage reference solutions and hints\(Zhaoet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib2); Agarwalet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib8)\)\. Other works explicitly study privileged information distillation in this setting\(Penalozaet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib12); Chenet al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib5)\)\. While these methods are restricted to isolated instance\-level corrections, our approach introduces cross\-sample reflection to aggregate experience and provide a richer training signal\.

Continual learning via self\-reflection and experience accumulation\.Recent work explores training\-free continual learning by extracting reusable strategies from an LLM’s own trajectories\. Models can utilize verbal feedback as episodic memory\(Shinnet al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib21); Madaanet al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib30)\), self\-discover reasoning structures\(Zhouet al\.,[2024](https://arxiv.org/html/2605.12741#bib.bib22); Wanget al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib13)\), or maintain an evolving experience pool to dynamically inject refined strategies into future inference\(Zhanget al\.,[2025](https://arxiv.org/html/2605.12741#bib.bib9); Yoranet al\.,[2023](https://arxiv.org/html/2605.12741#bib.bib23)\)\. Building on this trajectory\-based memory paradigm, our method integrates experience accumulation directly into on\-policy self\-distillation\.

## 6Discussion and Conclusions

In this work, we address the challenge of continuous LLM self\-improvement via environmental interactions, particularly in sparse\-reward regimes\. We introduce Reflection\-Enhanced Self\-Distillation \(Resd\), a framework that utilizes retrospective reflections and a persistent playbook to successfully transform raw failure feedback into active, corrective supervision\. Empirical evaluations across multiple complex reasoning tasks demonstrate thatResdsignificantly enhances interaction efficiency\. By relying on only a single rollout per prompt, it achieves rapid early\-stage convergence, substantially outperforming standard self\-distillation methods and GRPO\.

Resdas a feedback\-enhancement module\.The non\-monotonic validation curves in Figure[6](https://arxiv.org/html/2605.12741#S4.F6)highlight an important property of on\-policy self\-distillation: its effectiveness depends not only on the structure and reliability of the teacher context, but also on the optimization objective\. Recent studies similarly observe that self\-distillation can exhibit unstable dynamics\(Kimet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib35); Liet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib36); Yanget al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib37)\)\. Our results complement this line of work from a different angle\. Rather than proposing a new self\-distillation objective,Resdimproves the information supplied to the self\-teacher by converting raw failure feedback into local reflections and persistent playbook knowledge\. This explains whyResdis especially effective in the early rare\-success regime, where reward\-only methods obtain little signal and standard self\-distillation lacks successful demonstrations\. More broadly,Resdcan be plugged into stronger self\-distillation algorithms: sample routing can decide when to apply distillation\(Liet al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib36)\), reward\-grounded objectives can determine update directions\(Yanget al\.,[2026](https://arxiv.org/html/2605.12741#bib.bib37)\), andResdcan provide the structured feedback context that makes failed trajectories useful for token\-level supervision\.

## References

- \[1\]R\. Agarwal, N\. Vieillard, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. Bachem\(2023\)Gkd: generalized knowledge distillation for auto\-regressive sequence models\.arXiv preprint arXiv:2306\.1364912\.Cited by:[§1](https://arxiv.org/html/2605.12741#S1.p2.1),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[2\]R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem\(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[§2](https://arxiv.org/html/2605.12741#S2.p1.13),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[3\]H\. Chen, N\. Razin, K\. Narasimhan, and D\. Chen\(2025\)Retaining by doing: the role of on\-policy data in mitigating forgetting\.External Links:2510\.18874,[Link](https://arxiv.org/abs/2510.18874)Cited by:[§2](https://arxiv.org/html/2605.12741#S2.p1.13),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[4\]Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu\(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.arXiv preprint arXiv:2401\.01335\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[5\]DeepSeek\-AI\(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[6\]J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. Krause\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§1](https://arxiv.org/html/2605.12741#S1.p2.1),[§2](https://arxiv.org/html/2605.12741#S2.p1.8),[§3\.1](https://arxiv.org/html/2605.12741#S3.SS1.p1.2),[§3\.3](https://arxiv.org/html/2605.12741#S3.SS3.p4.9),[Table 2](https://arxiv.org/html/2605.12741#S4.T2.6.1.11.1),[Table 2](https://arxiv.org/html/2605.12741#S4.T2.6.1.6.1),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[7\]J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. Yang\(2026\)Why does self\-distillation \(sometimes\) degrade the reasoning capability of llms?\.arXiv preprint arXiv:2603\.24472\.Cited by:[§6](https://arxiv.org/html/2605.12741#S6.p2.1)\.
- \[8\]Y\. Kim and A\. M\. Rush\(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[9\]J\. Ko, S\. Kim, T\. Chen, and S\. Yun\(2024\)Distillm: towards streamlined distillation for large language models\.arXiv preprint arXiv:2402\.03898\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[10\]S\. Lai, H\. Zhao, R\. Feng, C\. Ma, W\. Liu, H\. Zhao, X\. Lin, D\. Yi, Q\. Zhang, H\. Liu, G\. Meng, and F\. Zhu\(2026\)Reinforcement fine\-tuning naturally mitigates forgetting in continual post\-training\.External Links:2507\.05386,[Link](https://arxiv.org/abs/2507.05386)Cited by:[§2](https://arxiv.org/html/2605.12741#S2.p1.13)\.
- \[11\]G\. Li, T\. Yang, J\. Fang, M\. Song, M\. Zheng, H\. Guo, D\. Zhang, J\. Wang, and T\. Chua\(2026\)Unifying group\-relative and self\-distillation policy optimization via sample routing\.arXiv preprint arXiv:2604\.02288\.Cited by:[§6](https://arxiv.org/html/2605.12741#S6.p2.1)\.
- \[12\]K\. Lu and T\. M\. Lab\(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2605.12741#S1.p2.1),[§2](https://arxiv.org/html/2605.12741#S2.p1.13),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[13\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.Advances in neural information processing systems36,pp\. 46534–46594\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.
- \[14\]E\. Penaloza, D\. Vattikonda, N\. Gontier, A\. Lacoste, L\. Charlin, and M\. Caccia\(2026\)Privileged information distillation for language models\.arXiv preprint arXiv:2602\.04942\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[15\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[16\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§4\.4](https://arxiv.org/html/2605.12741#S4.SS4.p1.3)\.
- \[17\]I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal\(2026\)Self\-distillation enables continual learning\.External Links:2601\.19897,[Link](https://arxiv.org/abs/2601.19897)Cited by:[§2](https://arxiv.org/html/2605.12741#S2.p1.13)\.
- \[18\]I\. Shenfeld, J\. Pari, and P\. Agrawal\(2025\)RL’s razor: why online reinforcement learning forgets less\.External Links:2509\.04259,[Link](https://arxiv.org/abs/2509.04259)Cited by:[§2](https://arxiv.org/html/2605.12741#S2.p1.13)\.
- \[19\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.
- \[20\]Y\. Sun, Y\. Cao, P\. Huang, H\. Bai, H\. Hajishirzi, N\. Dziri, and D\. Song\(2025\-09\)RL grokking recipe: how does rl unlock and transfer new algorithms in llms?\.External Links:2509\.21016,[Document](https://dx.doi.org/10.48550/arXiv.2509.21016),[Link](https://arxiv.org/abs/2509.21016)Cited by:[§4\.1](https://arxiv.org/html/2605.12741#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.12741#S4.T1.4.2.1.1),[Table 1](https://arxiv.org/html/2605.12741#S4.T1.4.3.1.1),[Table 1](https://arxiv.org/html/2605.12741#S4.T1.4.4.1.1)\.
- \[21\]G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[22\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.
- \[23\]B\. Xiao, B\. Xia, B\. Yang, B\. Gao, B\. Shen, C\. Zhang, C\. He, C\. Lou, F\. Luo, G\. Wang,et al\.\(2026\)Mimo\-v2\-flash technical report\.arXiv preprint arXiv:2601\.02780\.Cited by:[§1](https://arxiv.org/html/2605.12741#S1.p2.1),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[24\]S\. Xu, W\. Fu, J\. Gao, W\. Ye, W\. Liu, Z\. Mei, G\. Wang, C\. Yu, and Y\. Wu\(2024\)Is dpo superior to ppo for llm alignment? a comprehensive study\.arXiv preprint arXiv:2404\.10719\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[25\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[26\]C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan\(2026\)Self\-distilled rlvr\.arXiv preprint arXiv:2604\.03128\.Cited by:[§6](https://arxiv.org/html/2605.12741#S6.p2.1)\.
- \[27\]T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei\(2026\)On\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[28\]O\. Yoran, T\. Wolfson, B\. Bogin, U\. Katz, D\. Deutch, and J\. Berant\(2023\)Answering questions by meta\-reasoning over multiple chains of thought\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5942–5966\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.
- \[29\]Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.\(2025\)Agentic context engineering: evolving contexts for self\-improving language models\.arXiv preprint arXiv:2510\.04618\.Cited by:[§3\.3](https://arxiv.org/html/2605.12741#S3.SS3.p1.2),[§4\.1](https://arxiv.org/html/2605.12741#S4.SS1.p1.1),[Table 1](https://arxiv.org/html/2605.12741#S4.T1.4.5.1.1),[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.
- \[30\]S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover\(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§1](https://arxiv.org/html/2605.12741#S1.p2.1),[§2](https://arxiv.org/html/2605.12741#S2.p1.8),[§5](https://arxiv.org/html/2605.12741#S5.p1.1)\.
- \[31\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Appendix E](https://arxiv.org/html/2605.12741#A5.p1.1)\.
- \[32\]P\. Zhou, J\. Pujara, X\. Ren, X\. Chen, H\. Cheng, Q\. V\. Le, D\. Zhou, S\. Mishra, H\. S\. Zheng,et al\.\(2024\)Self\-discover: large language models self\-compose reasoning structures\.Advances in Neural Information Processing Systems37,pp\. 126032–126058\.Cited by:[§5](https://arxiv.org/html/2605.12741#S5.p2.1)\.

## Appendix ADiscussion on the Self\-Distillation Objective

As established in Section[2](https://arxiv.org/html/2605.12741#S2), our unified self\-distillation objective minimizes the expectedff\-divergence between the teacher distributionπθold\(⋅\|x,y<t,c\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|x,y\_\{<t\},c\)and the student policyπθ\(⋅\|x,y<t\)\\pi\_\{\\theta\}\(\\cdot\|x,y\_\{<t\}\)\. The choice of the generator functionffyields qualitatively different optimization landscapes\. Below we discuss three canonical instantiations and their practical implications\.

#### Forward Kullback\-Leibler \(KL\) Divergence\.

Settingf​\(τ\)=τ​log⁡τf\(\\tau\)=\\tau\\log\\taurecovers the standard distillation objective\. Because the loss is an expectation weighted by the student policyπθ\\pi\_\{\\theta\}yet penalizes deviations throughτ​log⁡τ\\tau\\log\\tau, the resulting gradient is dominated by regions where the teacher assigns substantial probability mass\. This produces*mean\-seeking*behavior: the student is penalized heavily whenever it assigns low probability to tokens the teacher considers valid, encouraging it to cover all modes of the teacher distribution\.

#### Reverse KL Divergence\.

Settingf​\(τ\)=−log⁡τf\(\\tau\)=\-\\log\\tauswaps the directional penalty, yielding*mode\-seeking*behavior\. Here the student incurs large loss when it generates tokens that the teacher deems highly improbable\. In complex reasoning tasks, this property is particularly desirable because it actively suppresses logical hallucinations—tokens that are plausible under the student’s distribution but inconsistent with the teacher’s feedback\-informed reasoning\.

#### Jensen\-Shannon Divergence \(JSD\)\.

Settingf​\(τ\)=τ2​log⁡τ−τ\+12​log⁡\(τ\+12\)f\(\\tau\)=\\frac\{\\tau\}\{2\}\\log\\tau\-\\frac\{\\tau\+1\}\{2\}\\log\\\!\\left\(\\frac\{\\tau\+1\}\{2\}\\right\)yields a smoothed, symmetric objective bounded between0andlog⁡2\\log 2\. By interpolating between the forward and reverse directions, JSD stabilizes training in regimes where the teacher and student distributions are nearly disjoint—precisely the situation that causes extreme gradient explosions under pure forward or reverse KL\. This boundedness makes JSD a robust default when the quality gap between teacher and student is large or highly variable across training\.

## Appendix BSuccess\-Rate Rebalancing

In the self\-distillation loss, every sample in a rollout batch contributes equally by default\. When the batch success rate is very low \(or very high\), the gradient signal is dominated by the majority outcome, diluting the supervision from the minority\. To counteract this imbalance, we weight each sampleiiby a scalarwiw\_\{i\}that depends on the batch success rates=\|\{i:R​\(xi,yi\)≥τ\}\|/Bs=\|\\\{i:R\(x\_\{i\},y\_\{i\}\)\\geq\\tau\\\}\|/B:

wi=\{\(1−s\)αif​R​\(xi,yi\)≥τ\(success\),sβotherwise\(failure\),w\_\{i\}=\\begin\{cases\}\(1\-s\)^\{\\alpha\}&\\text\{if \}R\(x\_\{i\},y\_\{i\}\)\\geq\\tau\\quad\(\\text\{success\}\),\\\\ s^\{\\beta\}&\\text\{otherwise\}\\quad\(\\text\{failure\}\),\\end\{cases\}\(2\)whereα,β\>0\\alpha,\\beta\>0are hyperparameters that control the strength of rebalancing, andτ\\tauis the success reward threshold\. The weights are normalized so that their batch mean equals one,w¯=1B​∑iwi=1\\bar\{w\}=\\frac\{1\}\{B\}\\sum\_\{i\}w\_\{i\}=1, ensuring that the effective learning rate is unaffected\. The weighted self\-distillation loss becomes:

ℒSDw​\(θ\)=1B​∑i=1Bwi​∑t=1Ti∑v∈𝒱πθ​\(v∣xi,yi,<t\)⋅f​\(τ~vt\)\.\\mathcal\{L\}\_\{\\text\{SD\}\}^\{w\}\(\\theta\)=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}w\_\{i\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid x\_\{i\},y\_\{i,<t\}\)\\cdot f\\\!\\left\(\\tilde\{\\tau\}^\{t\}\_\{v\}\\right\)\.\(3\)When success is rare \(s→0s\\to 0\), successful samples receive high weights while failed samples receive low weights, amplifying the distillation signal from the few successful demonstrations\. Conversely, when success is common, the corrective signal from failures is amplified\. In all experiments we setα=β=1\\alpha=\\beta=1\.

## Appendix CTraining Curves for Main Results

Figure[7](https://arxiv.org/html/2605.12741#A3.F7)shows the validation training curves of the runs summarized in Table[2](https://arxiv.org/html/2605.12741#S4.T2)\. Each panel plots validation accuracy over training steps for one task, comparing the baselines andResdunder the same rank\-based checkpoint\-selection rule described in Section[4](https://arxiv.org/html/2605.12741#S4)\.

![Refer to caption](https://arxiv.org/html/2605.12741v1/x11.png)\(a\)Manufactoria\-Has\.
![Refer to caption](https://arxiv.org/html/2605.12741v1/x12.png)\(b\)BouncingSim\-Easy\.
![Refer to caption](https://arxiv.org/html/2605.12741v1/x13.png)\(c\)BouncingSim\-Medium\.
![Refer to caption](https://arxiv.org/html/2605.12741v1/x14.png)\(d\)Finer\.

Figure 7:Validation accuracy over training steps for the runs reported in Table[2](https://arxiv.org/html/2605.12741#S4.T2)\.
## Appendix DPlaybook Concise Strategy

TheConciseoperation in Section[3\.3](https://arxiv.org/html/2605.12741#S3.SS3)maintains the playbook𝒫\\mathcal\{P\}as a compact set of reusable lessons over the course of training\. We describe two concrete instantiations used in our experiments and the trigger conditions under which they are applied\.

#### Bullet categorization\.

At the time of eachConcisecall, every entrypj∈𝒫p\_\{j\}\\in\\mathcal\{P\}is classified based on its tag history\. Lethjh\_\{j\}anddjd\_\{j\}denote the accumulated helpful and harmful counts assigned by the reflector across training steps \(Section[3\.3](https://arxiv.org/html/2605.12741#S3.SS3)\)\. We partition the playbook as

- •Unused:hj\+dj=0h\_\{j\}\+d\_\{j\}=0\(never tagged\),
- •Harmful:dj≥hjd\_\{j\}\\geq h\_\{j\}anddj\>0d\_\{j\}\>0\(net harmful\),
- •Helpful:otherwise\.

In addition, we record the most recent stepτj\\tau\_\{j\}at which each entry was tagged and usestep−τj\\text\{step\}\-\\tau\_\{j\}as a staleness measure\.

#### Concise methods\.

Given a budgetMmaxM\_\{\\max\}on the playbook size, we consider two removal strategies:

- •*Prioritized\.*Remove all unused and harmful entries\. If\|𝒫\|\|\\mathcal\{P\}\|still exceedsMmaxM\_\{\\max\}, randomly drop helpful entries until the cap is met\.
- •*Staleness\.*Remove all harmful entries and all*stale unused*entries \(unused for more than one context\-update step\)\. If\|𝒫\|\|\\mathcal\{P\}\|still exceedsMmaxM\_\{\\max\}, additionally evict the longest\-unused remaining entries in order of increasingτj\\tau\_\{j\}until the cap is met\.

Both strategies guarantee that persistently harmful entries are removed; they differ in how they treat unused entries\. The*staleness*variant is more patient in that unused entries are given one additional step to be tagged before becoming eligible for removal, which helps preserve newly curated lessons that may take a few reflections before receiving a usage signal\.

#### Trigger conditions\.

Conciseis invoked under two conditions: \(i\) a*frequency trigger*, which fires everyFFcontext\-update steps \(defaultF=4F=4\), and \(ii\) a*limit trigger*, which fires whenever\|𝒫\|\>Mmax\|\\mathcal\{P\}\|\>M\_\{\\max\}between scheduled invocations\. In the former case, the playbook is pruned down to its harmful/stale subset without enforcing the budget; in the latter, the budgetMmaxM\_\{\\max\}is enforced\.

#### Default configuration\.

Unless otherwise noted, we use the*staleness*method withMmaxM\_\{\\max\}andFFset to the values in Appendix[C](https://arxiv.org/html/2605.12741#A3)\(training details\)\. This is the configuration used for allResdruns in Table[2](https://arxiv.org/html/2605.12741#S4.T2)\.

## Appendix EPreservation of Instruction Following Capability

Table 5:IFEval accuracy after training on each task\.Per\-Test\-Case Acc\.Per\-Task Acc\.Checkpointm@4b@4m@4b@4Base model87\.6291\.3382\.5087\.00AfterManuf\.\-Has87\.9691\.3383\.7587\.00AfterBSim\-Easy87\.8392\.3383\.2588\.00AfterFiner86\.8890\.3380\.5086\.00A common challenge in continual learning is catastrophic forgetting, where improvements on specific reasoning tasks may degrade the model’s foundational instruction\-following capabilities\. To verify thatResdavoids this trade\-off, we evaluate the models on the IFEval benchmark\[[31](https://arxiv.org/html/2605.12741#bib.bib31)\]before and after training\. As shown in Table[5](https://arxiv.org/html/2605.12741#A5.T5),Resdsuccessfully preserves the model’s general capabilities across all evaluation settings\. For instance, after training on theManufactoria\-Hastask, the per\-task mean@4 accuracy actually increases slightly from 82\.50% to 83\.75%, while the per\-test\-case mean@4 accuracy remains stable at around 87–88%\.

## Appendix FAnalysis on Training Latency

![Refer to caption](https://arxiv.org/html/2605.12741v1/x15.png)Figure 8:Per\-step training latency forBouncingSim\-Easy\.WhileResdsignificantly improves interaction efficiency by requiring only a single rollout per prompt, the generation of retrospective reflections and playbook curation introduces additional inference steps\. To understand the latency implications of these components, we analyze the per\-step training latency ofResdcompared to SDPO and GRPO baselines\. As illustrated in Figure[8](https://arxiv.org/html/2605.12741#A6.F8), standard SDPO exhibits the lowest latency about400400s per\-step, because it only computes the teacher distribution and KL loss for a single trajectory\.Resdintroduces a moderate and stable computational overhead due to the diagnostic inference required for the reflection and curation operations\. However,Resdremains substantially faster than GRPO\. Because GRPO relies on group\-relative optimization, it must repeatedly sample, execute, and evaluate a large batch of trajectories, causing its latency to climb above12001200s per step\.

## Appendix GImplementation Details

#### Infrastructure\.

All experiments are conducted on a single node with 8 NVIDIA H200 GPUs\. Each experiment completes in approximately one day of wall\-clock time\. We use vLLM for rollout generation with tensor parallelism across 4 GPUs, and FSDP for distributed training\. Rollout generation uses temperature1\.01\.0with top\-ppsampling at0\.950\.95\. All models have thinking mode enabled\.

#### Training\.

We train for a single epoch over the training data \(online streaming\)\. Each batch undergoesK=4K=4inner\-loop update iterations\. The batch size is3232and we use a single rollout per prompt \(N=1N\{=\}1\) for allResdexperiments\. We apply a learning rate warmup of 10 steps\. For validation, we sample44responses per test problem to compute mean@4 and best@4 metrics\.

#### Self\-distillation\.

The self\-distillation loss is computed over the top\-100100tokens in the student’s vocabulary at each position\. Success\-rate rebalancing \(Appendix[B](https://arxiv.org/html/2605.12741#A2)\) is applied onBouncingSim\-EasyandFiner\.

#### Context updater \(Resd\-specific\)\.

The playbook is maintained globally \(shared across all training examples\)\. Playbook entries are tagged as helpful or harmful during reflection on both failed and successful samples\. The solution buffer caches successful trajectories for future teacher prompt construction\.

#### Per\-task hyperparameters\.

Unless otherwise noted, we use EMA update rate0\.00010\.0001, lower clipϵmin=0\.2\\epsilon\_\{\\min\}\{=\}0\.2, and no upper clip\. Task\-specific deviations are as follows:

- •Manufactoria\-Has: reverse KL \(α=1\.0\\alpha\{=\}1\.0\), upper clipϵmax=5\\epsilon\_\{\\max\}\{=\}5, max playbook entriesMmax=200M\_\{\\max\}\{=\}200, prioritized concise method, max prompt/response length 49152/20480 tokens\.
- •BouncingSim\-Easy: JSD \(α=0\.5\\alpha\{=\}0\.5\), max playbook entriesMmax=120M\_\{\\max\}\{=\}120, staleness\-based concise method, success\-rate reweighting enabled, max prompt/response length 58368/25600 tokens\.
- •BouncingSim\-Medium: reverse KL \(α=1\.0\\alpha\{=\}1\.0\), EMA update rate0\.010\.01, max playbook entriesMmax=150M\_\{\\max\}\{=\}150, prioritized concise method, max prompt/response length 58368/25600 tokens\.
- •Finer: JSD \(α=0\.5\\alpha\{=\}0\.5\), max playbook entriesMmax=120M\_\{\\max\}\{=\}120, staleness\-based concise method, success\-rate reweighting enabled, max prompt/response length 49152/20480 tokens\.

#### GRPO baseline\.

For GRPO, we use a rollout group size of88per prompt with the same learning rate \(1​e\-​61\\text\{e\-\}6\), batch size \(3232\), and inner\-loop steps \(K=4K\{=\}4\)\. GRPO does not use self\-distillation, context updating, or EMA\.

## Appendix HCase Study: Full Program Outputs

This appendix provides the complete DSL programs generated at each inner\-loop step for the case study in Section[4\.6](https://arxiv.org/html/2605.12741#S4.SS6)\(“Accept if the tape contains BRBR”\)\. Each listing corresponds to one row of Table[4](https://arxiv.org/html/2605.12741#S4.T4)\. We highlight the buggy lines with inline comments\.

#### Step 45 \(acc\. = 0\.00\): Parse error from undefined node reference\.

The program referencesstate0in aPULLER\_YGnode, butstate0is never declared as a node, causing a parse error\.

STARTstart:

NEXTnode\_rb

PULLER\_RBnode\_rb:

\[B\]node\_b1

\[R\]node\_rb

\[EMPTY\]node\_yg

PULLER\_YGnode\_yg:

\[Y\]node\_rb

\[G\]node\_rb

\[EMPTY\]state0\#BUG:’state0’isneverdeclared

PULLER\_RBnode\_b1:

\[R\]node\_b1r1

\[B\]node\_b1

\[EMPTY\]node\_yg

\.\.\.

ENDend

#### Step 46 \(acc\. = 0\.84\): Correct pattern matching logic, but rejects non\-R/B characters\.

The state machine correctly implements substring matching transitions for B and R\. However, all\[EMPTY\]branches route toNONE, which causes immediate rejection when a non\-R/B character \(Y or G\) is encountered—e\.g\., inputGGYYYBBRBRis rejected at states0because the leadingGtriggers\[EMPTY\] NONE\.

STARTstart:

NEXTs0

PULLER\_RBs0:

\[B\]s1

\[R\]s0

\[EMPTY\]NONE\#BUG:rejectsY/Gcharacters

PULLER\_RBs1:

\[R\]s2

\[B\]s1

\[EMPTY\]NONE\#BUG:sameissue

PULLER\_RBs2:

\[B\]s3

\[R\]s0

\[EMPTY\]NONE\#BUG:sameissue

PULLER\_RBs3:

\[R\]end

\[B\]s1

\[EMPTY\]NONE\#BUG:sameissue

ENDend

#### Step 47 \(acc\. = 0\.96\): Overcorrection introduces an infinite loop\.

The model adds aPULLER\_YGnode to handle non\-R/B characters, fixing the step 46 bug\. However, when the tape becomes empty at a non\-terminal state,PULLER\_RBroutes tostate0\_yg, whose\[EMPTY\]routes toNONE—but if there are remaining characters that alternate between the two node types, they bounce back and forth infinitely\.

STARTstart:

NEXTstate0

PULLER\_RBstate0:

\[B\]state1

\[R\]state0

\[EMPTY\]state0\_yg\#Routesnon\-R/BtoPULLER\_YG

PULLER\_RBstate1:

\[R\]state2

\[B\]state0

\[EMPTY\]state0\_yg

PULLER\_RBstate2:

\[B\]state3

\[R\]state0

\[EMPTY\]state0\_yg

PULLER\_RBstate3:

\[R\]end

\[B\]state0

\[EMPTY\]state0\_yg

PULLER\_YGstate0\_yg:

\[Y\]state0

\[G\]state0

\[EMPTY\]NONE\#BUG:causesinfiniteloopwhentape

\#isemptybetweenstate0andstate0\_yg

ENDend

#### Step 48 \(acc\. = 1\.00\): Correct solution\.

The final program correctly handles all cases: non\-pattern symbols are skipped via dedicatedPULLER\_YGnodes, and empty\-tape transitions at non\-terminal states correctly reject\. A cached successful solution from a prior rollout provided the complete exemplar that guided this output\.

STARTstart:

NEXTs0

PULLER\_RBs0:

\[B\]s1

\[R\]s0

\[EMPTY\]s0\_yg

PULLER\_YGs0\_yg:

\[Y\]s0

\[G\]s0

\[EMPTY\]NONE

PULLER\_RBs1:

\[R\]s2

\[B\]s1

\[EMPTY\]s1\_yg

PULLER\_YGs1\_yg:

\[Y\]s1

\[G\]s1

\[EMPTY\]NONE

PULLER\_RBs2:

\[B\]s3

\[R\]s0

\[EMPTY\]s2\_yg

PULLER\_YGs2\_yg:

\[Y\]s2

\[G\]s2

\[EMPTY\]NONE

PULLER\_RBs3:

\[R\]end

\[B\]s1

\[EMPTY\]s3\_yg

PULLER\_YGs3\_yg:

\[Y\]s3

\[G\]s3

\[EMPTY\]NONE

ENDend