Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation

# Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation
Source: [https://arxiv.org/html/2608.04408](https://arxiv.org/html/2608.04408)
###### Abstract

On\-policy distillation \(OPD\) supervises student\-visited trajectories, yet divergence\-based rules cannot determine whether an erroneous prefix remains correctable\. We formulate this decision ascounterfactual recoverabilityand replay each error state through budget\-matched teacher\-continuation and rollback branches\. Based on their relative success, states are categorized as recoverable, irreversible\-but\-avoidable, or ambiguous, and these labels guide whether training retains, rolls back, or conventionally supervises the corresponding trajectory\. On AIME branch diagnostics, the mean continuation\-minus\-rollback effect is 0\.185 for recoverable states and−1\.000\-1\.000for irreversible\-but\-avoidable states, demonstrating opposite intervention preferences\. A branch\-derived recoverability proxy achieves an AUC of 1\.000, substantially outperforming divergence alone at 0\.392\. Across frozen evaluations, recoverability\-aware control achieves the strongest recorded performance, reaching 0\.578 success on held\-out AIME2025 compared with 0\.517 for the best baseline\. It also improves AIME2024–2025 average@32 from 0\.2656 to 0\.3125 and GPQA\-Diamond average@32 from 0\.2702 to 0\.3070\. Component ablations further show that retaining teacher\-correctable prefixes provides the largest individual contribution\. These findings establish recoverability as an outcome\-grounded decision variable for selective supervision in OPD\.

## Introduction

On\-policy distillation \(OPD\) provides dense teacher supervision on student\-visited trajectories, but an error can leave a state that remains correctable or one whose prefix is harmful to retain\. Step\-wise weighting uses teacher–student divergence to attenuate questionable supervision\(Zhong et al\.[2026](https://arxiv.org/html/2608.04408#bib.bib21)\); broader analyses identify compatibility and trajectory dynamics as central OPD conditions\(Li et al\.[2026b](https://arxiv.org/html/2608.04408#bib.bib9)\)\. Yet disagreement describes two policies rather than answering what action should follow an error\.

This distinction is especially important in sequential reasoning and tool\-interaction settings, where an action alters the context for all later actions\. A deviation can be local, permitting teacher correction, or structurally harmful, leaving insufficient budget for recovery\. Treating both identically can discard useful supervision or reinforce a trajectory that cannot recover\.

Existing OPD interventions primarily regulate the amount, location, or schedule of teacher guidance\. SOD reweights step\-level supervision using the evolution of divergence\(Zhong et al\.[2026](https://arxiv.org/html/2608.04408#bib.bib21)\); curriculum turn\-level guidance modifies how teacher and student turns are mixed\(Li et al\.[2026a](https://arxiv.org/html/2608.04408#bib.bib8)\)\. These approaches provide important mechanisms for handling distribution shift, but their observable signals do not directly encode the causal question faced after an error: is continuation better than rollback under the same resources? The answer depends jointly on the state, remaining budget, action history, and environment dynamics\.

![Refer to caption](https://arxiv.org/html/2608.04408v1/divergence_failure_sketch.png)Figure 1:Why divergence\-only suppression is insufficient\. Equally high disagreement can occur on a repairable prefix, where continuation helps, or an unrecoverable prefix, where rollback helps\.We therefore ask a concrete question: under a fixed budget, is a teacher more likely to finish the task when it continues from the error, or when the error is rolled back and the branch is resampled? We call the answer*recoverability*\. This framing is intentionally diagnostic\. It does not presume that a branch oracle is inexpensive, that its labels can be predicted online, or that a policy trained with an oracle dominates conventional OPD\. Rather, it makes the decision underlying a divergence heuristic observable and falsifiable\.

Our approach is based on replayed counterfactual branches\. We reconstruct a selected pre\-error prefix, run teacher continuation and rollback branches with matched generation and environment budgets, and compare their empirical success frequencies\. The resulting labels are paired with a candidate\-conditioned OPD backbone: teacher and student score the same admissible action set, avoiding invalid\-action confounds and making the comparison independent of token alignment\. We further separate branch probes from held\-out branch outcomes when assessing diagnostic prediction, and log the additional costs of oracle label acquisition\.

Our main contributions are:

1. 1\.We specify budget\-matched replay branches for continuation and rollback\. The resulting protocol labels states as recoverable, irreversible\-but\-avoidable, or ambiguous\.
2. 2\.On the recorded AIME branch diagnostic, we find opposite action\-effect signs for recoverable and irreversible\-but\-avoidable states and show that a recoverability proxy is more informative than divergence alone\.
3. 3\.We audit an oracle\-assisted control and evaluate frozen adapters across two held\-out AIME protocols and GPQA\-Diamond\. The recorded oracle is consistently above the E0 controls on these fixed formal evaluations, while component ablations identify retention as the largest single contribution\.

All performance claims remain bounded by the supplied manifests, seed counts, and non\-matched training cost\.

## Related Work

### On\-Policy Distillation and Intervention Granularity

OPD differs from offline distillation because supervision is queried on student\-generated trajectories\. This reduces train–test state mismatch but exposes supervision to compounding errors\. SOD responds by reweighting step\-level supervision according to divergence\(Zhong et al\.[2026](https://arxiv.org/html/2608.04408#bib.bib21)\); turn\-level guidance instead schedules intervention across a multi\-turn rollout\(Li et al\.[2026a](https://arxiv.org/html/2608.04408#bib.bib8)\)\. These methods establish that when and where to supervise matters\. Our question is orthogonal: even at a fixed divergence or turn, does an error state warrant continuation or rollback? The answer depends on an environmental counterfactual, not on disagreement alone\.

The present implementation uses shared admissible action candidates and normalized candidate distributions so that invalid actions do not masquerade as score differences\. Because the Qwen3\.5\-9B student and Qwen3\.5\-27B teacher are loaded from distinct model paths, the implementation conservatively treats their token sequences as unaligned; consequently, the recorded step signal is a sequence\-mean log\-probability gap rather than an exact token\-aligned divergence\. We treat it as a diagnostic feature and never as a theoretically exact reproduction of token\-level SOD\.

### Candidate Actions and Cross\-Tokenizer Comparability

Agent distillation may compare model scores or teacher targets at the token level\. That comparison is fragile when models tokenize the same action text differently or when free generation emits actions outside an environment’s legal interface\. Our candidate\-conditioned backbone instead scores a shared admissible action set and normalizes each model’s scores inside that set\. It therefore makes the selected action auditable while preventing parse or admissibility failures from being confused with recovery effects\.

This direction is an experimental control rather than a universal objective claim\. The candidate\-distribution JS value is useful for measuring local choice disagreement, but it is not called a token\-level KL and it is not assumed to solve distribution shift\. The sequence\-level gap used by the SOD reproduction is explicitly an approximation to the token\-aligned setting\. This clarification matters because the central contribution asks whether an observable disagreement signal is sufficient for a post\-error decision, not whether a new normalization alone improves agent performance\.

### Counterfactual Action Analysis

Counterfactual action outcomes have been used to explain RL decisions\(Amitai, Septon, and Amir[2024](https://arxiv.org/html/2608.04408#bib.bib2)\)\. Our use differs in purpose and protocol\. We replay an observed error state and compare two executable branches under identical remaining\-step and generation constraints\. The resulting label is not a semantic explanation of a policy; it is a decision\-oriented target for asking whether corrective supervision should retain the current prefix\. This distinction matters because a diagnostic that consumes additional teacher rollouts is an oracle upper bound, not an online deployment method\.

Recovery mechanisms for agents can retry an action, revise a plan, or request additional feedback at inference time\. Here the counterfactual label instead controls a training decision: retain a trajectory segment for corrective teacher supervision, roll it back for resampling, or leave it to ordinary SOD when evidence is ambiguous\. We therefore contribute a bounded protocol connecting recovery outcomes to OPD supervision\. We do not claim a general recovery planner, a state\-only predictor, or a learned inference\-time recovery policy\.

![Refer to caption](https://arxiv.org/html/2608.04408v1/recoverability_pipeline_aime_sketch.png)Figure 2:Recoverability\-aware on\-policy distillation pipeline\. The student acts over a shared admissible candidate set; an error state is replayed into budget\-matched continue and rollback/resample branches; observed branch outcomes determine a three\-way oracle control that either retains, rolls back, or defaults to SOD\. The AIME2024\-II\-4 and AIME2025\-I\-1 cards are illustrative task elements from the supplied manifests, not AIME2025 evaluation results\. The lower strip locates the recorded E1–E4 audits\.

## Methodology

### Problem Formulation

We consider an on\-policy student acting in a sequential environment\. At statests\_\{t\}, the environment exposes an admissible action setAtA\_\{t\}, the student selects an action, and the resulting observation becomes part of the next context\. A teacher supplies dense supervision on student\-visited states\. The methodological question is local: after an observed erroneous action, should training retain the realized prefix and use a corrective teacher continuation, or should it roll the action back and obtain a fresh branch? We do not assume that a large student–teacher discrepancy resolves this choice\.

### Counterfactual Recoverability Labels

For a replayed error statess, letB​\(s\)B\(s\)denote the remaining environment\-step, token, temperature, and sampling budget\. We executekkteacher branches that continue from the error andkkbranches that roll back the error, both usingB​\(s\)B\(s\)\. Letpc​o​n​t​\(s\)p\_\{cont\}\(s\)andpr​o​l​l​\(s\)p\_\{roll\}\(s\)be their empirical success frequencies\. We define

Δa​c​t​i​o​n​\(s\)=pc​o​n​t​\(s\)−pr​o​l​l​\(s\)\.\\Delta\_\{action\}\(s\)=p\_\{cont\}\(s\)\-p\_\{roll\}\(s\)\.States with sufficiently successful continuation are recoverable; states for which rollback is strictly preferable are irreversible\-but\-avoidable; all other states are ambiguous\. Ambiguity is a substantive outcome: it records insufficient evidence for a strong intervention rather than forcing a binary label\.

More precisely, with the pre\-registered thresholds used by the labeler,

ℓ​\(s\)=\{recoverable,pc​o​n​t​\(s\)≥0\.50,irreversible\-but\-avoidable,pc​o​n​t​\(s\)≤0\.25and​pr​o​l​l​\(s\)≥0\.50,ambiguous,otherwise\.\\ell\(s\)=\\begin\{cases\}\\text\{recoverable\},&p\_\{cont\}\(s\)\\geq 0\.50,\\\\ \\begin\{subarray\}\{c\}\\text\{irreversible\-but\-\}\\\\ \\text\{avoidable\}\\end\{subarray\},&\\begin\{subarray\}\{c\}p\_\{cont\}\(s\)\\leq 0\.25\\\\ \\text\{and \}p\_\{roll\}\(s\)\\geq 0\.50\\end\{subarray\},\\\\ \\text\{ambiguous\},&\\text\{otherwise\}\.\\end\{cases\}The thresholds define the analysis population; they are not learned parameters and are not tuned on the E2 outcome comparison\.

The protocol has three safeguards\. First, replay reconstructs the pre\-error prefix before either branch is run\. Second, branch checks assert matching remaining budgets\. Third, the E3 diagnostic separates probe branches used to construct a recoverability feature from held\-out branches used as the prediction target\. This avoids evaluating a predictor on the same branch outcomes that define it\. For oracle training, recoverable states retain a corrective horizon, irreversible\-but\-avoidable states trigger rollback/resampling, and ambiguous states fall back to ordinary SOD\.

### Candidate\-Conditioned Distillation Backbone

The diagnostic is built on an admissible\-candidate OPD implementation\. At a statess, the environment provides a shared textual action setA​\(s\)A\(s\)\. Student and teacher score everya∈A​\(s\)a\\in A\(s\)using their own sequence\-average log probabilities, and each score vector is normalized only within that shared set\. Thus, the rollout action is sampled from the student candidate distribution, the teacher target is the teacher argmax candidate, and the training loss is the student sequence negative log likelihood of that target\. This design removes two confounds of free generation: an invalid action cannot receive a spurious comparison score, and raw log\-probability scales from separately scored models are not compared directly\.

For modelm∈\{θ,T\}m\\in\\\{\\theta,T\\\}, letqm​\(a∣s\)q\_\{m\}\(a\\mid s\)be the sequence\-average log probability assigned to action textaa\. With distillation temperatureτd\\tau\_\{d\}, we form

Pm​\(a∣s,A\)=exp⁡\(qm​\(a∣s\)/τd\)∑a′∈A​\(s\)exp⁡\(qm​\(a′∣s\)/τd\)\.P\_\{m\}\(a\\mid s,A\)=\\frac\{\\exp\(q\_\{m\}\(a\\mid s\)/\\tau\_\{d\}\)\}\{\\sum\_\{a^\{\\prime\}\\in A\(s\)\}\\exp\(q\_\{m\}\(a^\{\\prime\}\\mid s\)/\\tau\_\{d\}\)\}\.The student samples rollout actions fromPθP\_\{\\theta\}using a separately configured candidate temperature, while the teacher target isaT∗​\(s\)=arg⁡maxa∈A​\(s\)⁡PT​\(a∣s,A\)a\_\{T\}^\{\*\}\(s\)=\\arg\\max\_\{a\\in A\(s\)\}P\_\{T\}\(a\\mid s,A\)\. The retained\-target loss is

ℒOPD​\(s\)=−log⁡πθ​\(aT∗​\(s\)∣s\)\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(s\)=\-\\log\\pi\_\{\\theta\}\(a\_\{T\}^\{\*\}\(s\)\\mid s\)\.
LetM=12​\(Pθ\+PT\)M=\\frac\{1\}\{2\}\(P\_\{\\theta\}\+P\_\{T\}\)\. Candidate disagreement is logged as

DJS​\(s\)=12​DKL​\(Pθ∥M\)\+12​DKL​\(PT∥M\)\.D\_\{\\mathrm\{JS\}\}\(s\)=\\tfrac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(P\_\{\\theta\}\\\|M\)\+\\tfrac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(P\_\{T\}\\\|M\)\.The SOD reproduction retains the cumulative\-ratio weighting rule over the recorded sequence\-level gapdkd\_\{k\},

w1=1,wk=min⁡\(∏u=1k−1du\+ϵdu\+1\+ϵ,1\+δ\),w\_\{1\}=1,\\qquad w\_\{k\}=\\min\\\!\\left\(\\prod\_\{u=1\}^\{k\-1\}\\frac\{d\_\{u\}\+\\epsilon\}\{d\_\{u\+1\}\+\\epsilon\},\\;1\+\\delta\\right\),with the official bounded amplification form\. Candidate JS and the SOD weight have different roles: JS summarizes local candidate disagreement, whereaswkw\_\{k\}responds to relative changes indkd\_\{k\}along a trajectory\. The recorded Qwen3\.5\-9B/Qwen3\.5\-27B run uses the implementation’s conservative unaligned\-token mode, sodkd\_\{k\}is a sequence\-mean log\-probability gap for the same action text, not an exact token\-level divergence\. NeitherDJSD\_\{\\mathrm\{JS\}\}norwkw\_\{k\}is assumed to equal recoverability\.

This separation motivates the counterfactual experiment\. A scalar disagreement score may be high because the student has entered an unrecoverable region, but it may also be high at a state from which a correct teacher continuation can still repair the trajectory\. Conversely, low divergence does not guarantee that a past action can be safely retained\. The branch experiment therefore evaluates the decision consequence directly instead of treating a correlation with divergence as sufficient evidence\.

### Oracle Control Rule and Training Interface

The oracle converts the label into an explicit training action\. For a recoverable state, it retains the prefix and assigns a corrective horizon; for an irreversible\-but\-avoidable state, it rolls back and resamples; for an ambiguous state, it preserves the ordinary SOD path\. This rule deliberately separates the causal label from the backbone loss\. The counterfactual label determines which trajectory segment is trusted, while the candidate\-conditioned OPD backbone determines how the retained teacher target is optimized\.

Operationally, each training update first produces a student trajectory and records eligible error states\. The online labeler then checks that the state has a remaining budget, runs matched branches, and records the label, branch cost, and any mismatch\. The training controller consumes only this audited record\. A missing label or a mismatch cannot silently become a strong intervention; it reverts to the ambiguous path\. This design makes the oracle an upper\-bound mechanism with observable failure behavior rather than a hidden heuristic\.

Table 1:Methodology components and their role\.

## Experimental Setup

The recorded training and branch experiments use 30 AIME2024 training\-manifest tasks, a Qwen3\.5\-9B student, and a Qwen3\.5\-27B teacher\. Candidate actions are scored through the environment\-provided admissible set\. The E1 gate selected 200 replayable states uniformly over five divergence strata with zero replay failures\. E2 retains 99 non\-ambiguous states \(65 recoverable and 34 irreversible\-but\-avoidable\); 101 states are ambiguous and are reported rather than silently discarded\. E5 freezes the final adapters and uses four pre\-specified formal protocols: a disjoint 30\-task AIME2025 multi\-step set, a larger 60\-task AIME multi\-step set, a 60\-problem AIME2024–2025 free\-text SOD\-style protocol at 32 samples, and GPQA\-Diamond \(198 questions\) at 32 samples\. E0 controls and the oracle are each recorded for seeds 0–2; all reported uncertainty is the sample standard deviation over these three training seeds, not a confidence interval over test items\.

GroupPolicy / methodTasksParseAdmissibleTask successUpdatesFrozen policies on the AIME2024 training manifestFrozen policyStudent \(Qwen3\.5\-9B\)301\.0001\.0000\.200–Frozen policyTeacher \(Qwen3\.5\-27B\)301\.0001\.0000\.500–E0 candidate\-conditioned training controls \(3\-seed mean\)E0 controlVanilla OPD301\.0001\.0000\.513300E0 controlSOD reproduction301\.0001\.0000\.452300E0 controlMatched random mask301\.0001\.0000\.439300E4 oracle feasibility records \(not equal\-budget baselines\)Oracle recordRecoverability oracle \(3\-seed mean\)301\.0001\.0000\.539300Oracle recordRecoverability oracle \(seed 2\)301\.0001\.0000\.541300Table 2:Recorded AIME2024 training\-protocol metrics, organized in benchmark\-table form\. “Tasks” is the 30\-task training manifest; frozen policies are one\-pass records, while the remaining rows are final training summaries\. The separate E5 table reports frozen AIME2025 evaluation\. Neither table is a five\-run or equal\-total\-cost leaderboard\.All methods recorded parse rate, admissible\-action rate, training\-target rate, task\-success traces, and NaN counts\. The table above is training health context rather than a final leaderboard\. E5 adds frozen evaluations reported as 3\-seed means over seeds 0–2 for E0 and the oracle\. These counts support a descriptive stability check but remain too small, and their training costs too different, for a cost\-matched or broad statistical claim\.

### Evaluation Questions and Decision Rules

The study uses five pre\-specified questions\. E0 asks whether the candidate\-conditioned training runs are healthy enough to interpret: parsing and admissibility must be high, NaNs absent, the SOD weight must have non\-degenerate dynamics, and the random\-mask control must match its derived budget\. E1 asks whether the selected error states can be replayed reliably\. E2 asks whether the label classes have opposite branch effects\. E3 asks whether recoverability information predicts the preferred branch better than simple alternatives\. E4 asks whether an oracle labeler can be integrated without exceeding its cost ceiling\. E5 asks whether frozen final adapters retain any advantage on an AIME2025 manifest not used in training\.

This ordering avoids a common failure mode in method papers: optimizing a new control before establishing that its state construction, counterfactuals, and controls are valid\. A passing gate is not an independent scientific result and does not establish novelty\. It is an experiment\-integrity condition\. E5 now supplies frozen outcomes across multiple task sets as matched 3\-seed averages, but it does not convert the records into a fair policy\-improvement study: the oracle still incurs additional teacher and environment calls during training, and seed counts remain limited\.

ParamsMethodMathScienceAverageAIME25AIME24–25 \(60\)GPQA\-D \(198\)SuccessAvg@32Pass@32Avg@32Pass@32Distilled Student Baselines9BVanilla OPD41\.67±2\.3641\.67\\pm 2\.3626\.56±0\.4426\.56\\pm 0\.4439\.17±1\.1839\.17\\pm 1\.1827\.02±0\.3627\.02\\pm 0\.3639\.39±0\.7139\.39\\pm 0\.7134\.76±0\.0734\.76\\pm 0\.07SOD reproduction51\.67±2\.3651\.67\\pm 2\.3624\.69±0\.4424\.69\\pm 0\.4437\.50±1\.1837\.50\\pm 1\.1825\.51±0\.3625\.51\\pm 0\.3637\.37±0\.7137\.37\\pm 0\.7135\.35±1\.0135\.35\\pm 1\.01Random mask48\.33±2\.3648\.33\\pm 2\.3623\.44±0\.4423\.44\\pm 0\.4434\.17±1\.1834\.17\\pm 1\.1823\.99±0\.3623\.99\\pm 0\.3634\.85±0\.7134\.85\\pm 0\.7132\.96±1\.0132\.96\\pm 1\.01Recoverability\-Aware Student9BRecoverability oracle57\.78±1\.92\\boldsymbol\{57\.78\\pm 1\.92\}31\.25±0\.62\\boldsymbol\{31\.25\\pm 0\.62\}45\.56±0\.96\\boldsymbol\{45\.56\\pm 0\.96\}30\.70±0\.52\\boldsymbol\{30\.70\\pm 0\.52\}44\.61±0\.77\\boldsymbol\{44\.61\\pm 0\.77\}41\.98±0\.94\\boldsymbol\{41\.98\\pm 0\.94\}Table 3:Performance comparison of frozen Qwen3\.5\-9B student adapters across the recorded benchmarks\. The best result is inbold, and the second\-best isunderlined\. Values are mean±\\pmsample s\.d\. across training seeds \(E0 and oracle:n=3n=3\)\. AIME2025 uses the disjoint 30\-task multi\-step manifest; AIME24–25 and GPQA\-Diamond use 32\-sample free\-text protocols\. Average is the unweighted mean of the five displayed rates\.

## Results

### Distillation Training Summary

Table[4](https://arxiv.org/html/2608.04408#Sx5.T4)gives a compact view of the final recorded training state\. The oracle has the highest online task\-success mean, but it consumes roughly three times as many teacher tokens as each E0 control\. These online values provide optimization and cost context only; the frozen AIME/GPQA comparisons are reported separately\.

Table 4:Final 300\-update training metrics, reported as mean with sample s\.d\. in parentheses across training seeds \(E0 and oracle:n=3n=3\)\. Success is online rather than held\-out\. Teacher tokens are in thousands\.Figure[3](https://arxiv.org/html/2608.04408#Sx5.F3)complements the endpoint summary with the recorded optimization trajectories\. Online success rises for every method, while training loss decreases and teacher–student agreement increases\. The oracle finishes with the highest online success but also accumulates substantially more teacher tokens; these curves therefore diagnose optimization and cost rather than establish an equal\-budget comparison\.

The most informative contrast is that the oracle pairs the highest final online success \(0\.5390\.539\) with the lowest teacher–student agreement \(0\.5730\.573\); the E0 controls end between0\.6250\.625and0\.7810\.781agreement\. This is consistent with selective correction rather than maximizing imitation at every state\. However, the oracle changes both branch selection and interaction volume\. It neither has the lowest terminal loss nor shows substantial separation in teacher\-target probability, and it uses662\.8662\.8k teacher tokens versus roughly220220k for each control\. Accordingly, the curves establish stable optimization and expose its cost, not an equal\-budget advantage\.

![Refer to caption](https://arxiv.org/html/2608.04408v1/x1.png)Figure 3:Recorded training dynamics over 300 updates\. Panels show \(a\) online task success, \(b\) training loss, \(c\) teacher–student agreement, \(d\) teacher\-target probability, \(e\) mean distillation weight, and \(f\) cumulative teacher tokens\. Lines are smoothed only for visualization; shaded bands summarize the available training\-seed variation\.
### Recoverability Reveals Opposite Decisions

The matched\-budget branch contrast is the central result\. Among 99 non\-ambiguous recorded AIME states, continuation is favored in the 65 recoverable states \(meanΔa​c​t​i​o​n=pc​o​n​t−pr​o​l​l=0\.185\\Delta\_\{action\}=p\_\{cont\}\-p\_\{roll\}=0\.185\), whereas rollback is favored in the 34 irreversible\-but\-avoidable states \(meanΔa​c​t​i​o​n=−1\.000\\Delta\_\{action\}=\-1\.000\)\. The between\-group difference is 1\.185, with a group\-bootstrap interval of \[1\.092, 1\.277\]\. We do not infer unrecorded per\-group error bars\. Figure[4](https://arxiv.org/html/2608.04408#Sx5.F4)further shows that this sign separation remains visible across all five recorded divergence strata\. Thus a stratum of disagreement is not itself a decision rule\.

![Refer to caption](https://arxiv.org/html/2608.04408v1/x2.png)Figure 4:Stratified E2 diagnostic across five divergence bins\. The horizontal order is divergence stratum, not training time\.
### Divergence\-Only is a Weak Recorded Diagnostic

Table 5:E3 predictor diagnostic on 99 states in 18 task groups\.The proxy\-minus\-divergence AUC gap is 0\.608 \(group\-bootstrap interval \[0\.424, 0\.817\]\)\. This is diagnostic evidence from teacher branch probes, not a learned online estimator\. The divergence\-only AUC is below the no\-skill reference in this small diagnostic, while the budget\-only reference is exactly 0\.500\. The result should not be interpreted as a benchmark leaderboard: the recoverability feature itself is assembled from teacher branch probes\. Its value is to demonstrate that the targeted counterfactual contains information not captured by the recorded divergence score\.

### Interpreting the Effect Size

The E2 contrast is deliberately expressed as an action\-effect difference rather than an accuracy gain\. A positive value means that continuation succeeds more often than rollback under the recorded branch sampling; a value of−1\-1means that the recorded rollback branch succeeds while continuation does not\. The non\-overlapping action preferences and the interval excluding zero therefore support the diagnostic claim that the two labels encode opposite interventions under this protocol\.

It does not establish that every high\-divergence state is irreversible, that every recoverable state should be trained on identically, or that the threshold labels are universal\. The label rule uses finite branch samples and one teacher\. The appropriate interpretation is therefore conditional: given this candidate action space, replay implementation, branch budget, and AIME state distribution, a future\-outcome probe reveals a decision variable that is not recoverable from the reported divergence\-only baseline\.

### Oracle Feasibility is Auditable, Not a Performance Win

The oracle runs completed 300 updates for seeds 0–2 with no budget mismatch\. Seeds 0/1 teacher\-token inflation was 1\.54×\\timesand 1\.82×\\times, below the configured 2×\\timesceiling\. This supports feasibility under the recorded audit, not superiority over matched baseline outcomes\.

Table 6:Recorded oracle labeler audit\.The audit table retains the raw cost and integrity quantities rather than combining heterogeneous checks into one composite score\.

### Frozen Formal Evaluation Shows Consistent Gains on the Recorded Protocols

The expanded E5 suite replaces the earlier single\-checkpoint stress test with frozen evaluation across both AIME and science reasoning\. Table[3](https://arxiv.org/html/2608.04408#Sx4.T3)reports mean±\\pmsample standard deviation over recorded training seeds \(E0 and oracle:n=3n=3\)\. On the 30\-task AIME2025 multi\-step set, the oracle reaches0\.578±0\.0190\.578\\pm 0\.019, exceeding the strongest E0 mean, SOD at0\.517±0\.0170\.517\\pm 0\.017\. The same ordering remains on the larger 60\-task multi\-step set \(oracle0\.583±0\.0170\.583\\pm 0\.017; best E00\.492±0\.0080\.492\\pm 0\.008; reported in the text because the main table prioritizes cross\-benchmark metrics\)\. Importantly, the result is not restricted to the candidate\-action evaluation: on free\-text AIME2024–2025, the oracle improves average@32 from the strongest E0 value of 0\.2656 to 0\.3125 and pass@32 from 0\.3917 to 0\.4556\. On GPQA\-Diamond, it improves average@32 from 0\.2702 to 0\.3070 and pass@32 from 0\.3939 to 0\.4461\.

Table 7:3\-seed\-mean component ablations after 300 updates\. All entries are task\-success rates\.![Refer to caption](https://arxiv.org/html/2608.04408v1/x3.png)Figure 5:Frozen E5 scores \(left\) and 3\-seed\-mean component ablations \(right\)\. Whiskers are training\-seed standard deviations where available\. The corresponding values and protocol boundaries appear in Tables[3](https://arxiv.org/html/2608.04408#Sx4.T3)and[7](https://arxiv.org/html/2608.04408#Sx5.T7)\.The disjoint AIME2025 set and the free\-text AIME/GPQA protocols reduce the chance that the oracle ordering is only a training\-manifest or candidate\-interface artifact\. Yet these frozen adapters still come from unequal\-cost runs, and their standard deviations summarize few seeds on fixed task sets\. The result therefore establishes cross\-protocol persistence, not deployment readiness, equal\-cost superiority, or sample efficiency\.

### Ablations Identify Retention as the Largest Component Contribution

Figure[5](https://arxiv.org/html/2608.04408#Sx5.F5)and Table[7](https://arxiv.org/html/2608.04408#Sx5.T7)isolate the 3\-seed\-mean oracle controller after 300 updates\. Full control reaches 0\.578 / 0\.583 on held\-out 30 / 60; removing rollback, retention, or both reduces held\-out\-60 success to 0\.550, 0\.483, and 0\.450\. The same ordering holds online\. Although this few\-seed ablation is descriptive, its monotonic pattern identifies retention as the larger contribution and rollback as an additional benefit\.

### Ambiguity and Cost Bound the Oracle Result

Ambiguity is a measured outcome: 101 of 200 E2 states lack enough branch evidence to prefer continuation or rollback, so a learned controller should be able to abstain\. The E4 labeler processed 2,454 and 2,525 states while teacher\-token and environment\-step inflation remained below the configured twofold ceiling \(Table[6](https://arxiv.org/html/2608.04408#Sx5.T6)\)\. This verifies traceable execution, not efficiency, because E0 controls do not receive matched branches\. A deployable comparison requires a state\-only estimator and matched tokens, steps, seeds, and stopping rules\.

## Conclusion

We replace a divergence\-only post\-error heuristic with an executable question: under the same remaining budget, is continuation more successful than rollback? Matched replay reveals opposite intervention preferences for recoverable and irreversible\-but\-avoidable states, while the branch\-derived proxy separates these preferences more reliably than recorded divergence\. The oracle reaches higher online success despite lower teacher–student agreement, remains strongest on frozen AIME and GPQA\-Diamond protocols, and benefits most from retaining teacher\-correctable prefixes\.

The signals are complementary: divergence locates policy disagreement, whereas recoverability asks whether preserving the realized prefix still permits successful correction under the remaining budget\. It converts a candidate intervention into an outcome\-grounded decision\.

The ablations sharpen this interpretation: higher task success need not maximize agreement, and retaining teacher\-correctable prefixes contributes more than rollback alone\. Recoverability therefore informs selective supervision\.

These results are a diagnostic upper bound\. Evidence comes from one teacher, fixed manifests, and few seeds; the oracle uses about three times the teacher tokens, and 101 of 200 states are ambiguous\. A deployable state\-only estimator must abstain and be tested under matched budgets across models, seeds, and real tool environments\.

## References

- Agarwal et al\. \(2024\)Agarwal, R\.; Vieillard, N\.; Zhou, Y\.; Stanczyk, P\.; Ramos, S\.; Geist, M\.; and Bachem, O\. 2024\.On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.In*International Conference on Learning Representations*\.
- Amitai, Septon, and Amir \(2024\)Amitai, Y\.; Septon, Y\.; and Amir, O\. 2024\.Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes\.*Proceedings of AAAI*\.
- Bengio et al\. \(2015\)Bengio, S\.; Vinyals, O\.; Jaitly, N\.; and Shazeer, N\. 2015\.Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks\.In*Advances in Neural Information Processing Systems*, volume 28\.
- Gou et al\. \(2024\)Gou, Z\.; Shao, Z\.; Gong, Y\.; Shen, Y\.; Yang, Y\.; Duan, N\.; and Chen, W\. 2024\.CRITIC: Large Language Models Can Self\-Correct with Tool\-Interactive Critiquing\.In*International Conference on Learning Representations*\.
- Hinton, Vinyals, and Dean \(2015\)Hinton, G\.; Vinyals, O\.; and Dean, J\. 2015\.Distilling the Knowledge in a Neural Network\.*arXiv preprint arXiv:1503\.02531*\.
- Hsieh et al\. \(2023\)Hsieh, C\.\-Y\.; Li, C\.\-L\.; Yeh, C\.\-k\.; Nakhost, H\.; Fujii, Y\.; Ratner, A\.; Krishna, R\.; Lee, C\.\-Y\.; and Pfister, T\. 2023\.Distilling Step\-by\-Step\! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes\.In*Findings of the Association for Computational Linguistics: ACL 2023*, 8003–8017\. Association for Computational Linguistics\.
- Kim and Rush \(2016\)Kim, Y\.; and Rush, A\. M\. 2016\.Sequence\-Level Knowledge Distillation\.In*Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, 1317–1327\. Association for Computational Linguistics\.
- Li et al\. \(2026a\)Li, G\.; et al\. 2026a\.On\-Policy Distillation with Curriculum Turn\-level Guidance for Multi\-turn Agents\.*arXiv preprint arXiv:2606\.15912*\.
- Li et al\. \(2026b\)Li, Y\.; et al\. 2026b\.Rethinking On\-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe\.*arXiv preprint arXiv:2604\.13016*\.
- Lin et al\. \(2020\)Lin, A\.; Wohlwend, J\.; Chen, H\.; and Lei, T\. 2020\.Autoregressive Knowledge Distillation through Imitation Learning\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing*, 6121–6133\. Association for Computational Linguistics\.
- Madaan et al\. \(2023\)Madaan, A\.; Tandon, N\.; Gupta, P\.; Hallinan, S\.; Gao, L\.; Wiegreffe, S\.; Alon, U\.; Dziri, N\.; Prabhumoye, S\.; Yang, Y\.; Gupta, S\.; Majumder, B\. P\.; Hermann, K\.; Welleck, S\.; Yazdanbakhsh, A\.; and Clark, P\. 2023\.Self\-Refine: Iterative Refinement with Self\-Feedback\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Ouyang et al\. \(2022\)Ouyang, L\.; Wu, J\.; Jiang, X\.; Almeida, D\.; Wainwright, C\. L\.; Mishkin, P\.; Zhang, C\.; Agarwal, S\.; Slama, K\.; Ray, A\.; Schulman, J\.; Hilton, J\.; Kelton, F\.; Miller, L\.; Simens, M\.; Askell, A\.; Welinder, P\.; Christiano, P\.; Leike, J\.; and Lowe, R\. 2022\.Training Language Models to Follow Instructions with Human Feedback\.In*Advances in Neural Information Processing Systems*, volume 35\.
- Ross, Gordon, and Bagnell \(2011\)Ross, S\.; Gordon, G\.; and Bagnell, D\. 2011\.A Reduction of Imitation Learning and Structured Prediction to No\-Regret Online Learning\.In*Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics*, volume 15 of*Proceedings of Machine Learning Research*, 627–635\. PMLR\.
- Rusu et al\. \(2015\)Rusu, A\. A\.; Colmenarejo, S\. G\.; Gulcehre, C\.; Desjardins, G\.; Kirkpatrick, J\.; Pascanu, R\.; Mnih, V\.; Kavukcuoglu, K\.; and Hadsell, R\. 2015\.Policy Distillation\.*arXiv preprint arXiv:1511\.06295*\.
- Schick et al\. \(2023\)Schick, T\.; Dwivedi\-Yu, J\.; Dessi, R\.; Raileanu, R\.; Lomeli, M\.; Hambro, E\.; Zettlemoyer, L\.; Cancedda, N\.; and Scialom, T\. 2023\.Toolformer: Language Models Can Teach Themselves to Use Tools\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Shinn et al\. \(2023\)Shinn, N\.; Cassano, F\.; Gopinath, A\.; Narasimhan, K\.; and Yao, S\. 2023\.Reflexion: Language Agents with Verbal Reinforcement Learning\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Wang et al\. \(2023\)Wang, X\.; Wei, J\.; Schuurmans, D\.; Le, Q\. V\.; Chi, E\. H\.; Narang, S\.; Chowdhery, A\.; and Zhou, D\. 2023\.Self\-Consistency Improves Chain of Thought Reasoning in Language Models\.In*International Conference on Learning Representations*\.
- Wei et al\. \(2022\)Wei, J\.; Wang, X\.; Schuurmans, D\.; Bosma, M\.; Ichter, B\.; Xia, F\.; Chi, E\. H\.; Le, Q\. V\.; and Zhou, D\. 2022\.Chain\-of\-Thought Prompting Elicits Reasoning in Large Language Models\.In*Advances in Neural Information Processing Systems*, volume 35\.
- Yao et al\. \(2023\)Yao, S\.; Zhao, J\.; Yu, D\.; Du, N\.; Shafran, I\.; Narasimhan, K\.; and Cao, Y\. 2023\.ReAct: Synergizing Reasoning and Acting in Language Models\.In*International Conference on Learning Representations*\.
- Zelikman et al\. \(2022\)Zelikman, E\.; Wu, Y\.; Mu, J\.; and Goodman, N\. D\. 2022\.STaR: Bootstrapping Reasoning With Reasoning\.In*Advances in Neural Information Processing Systems*, volume 35\.
- Zhong et al\. \(2026\)Zhong, Q\.; et al\. 2026\.SOD: Step\-wise On\-policy Distillation for Small Language Model Agents\.*arXiv preprint arXiv:2605\.07725*\.