Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models

# Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
Source: [https://arxiv.org/html/2608.05168](https://arxiv.org/html/2608.05168)
Dayu Wang1,2Jiaye Yang1Weikang Li1Jiahui Liang1Yang Li1Deguo Xia1Jizhou Huang111footnotemark:1 1Baidu Inc\.2Nanyang Technological University dayu001@e\.ntu\.edu\.sgyamseyoung@gmail\.com wavejkd@pku\.edu\.cn\{liangjiahui03, liyang164, xiadeguo, huangjizhou01\}@baidu\.com

###### Abstract

Large language models often fail on reasoning tasks despite possessing the capability to solve them\. We argue that many such failures arise from localized reasoning bugs in intermediate steps rather than from global incompetence\. We show that these bugs are frequently repairable: inserting a short patch generated by a weak probe model after the same strong\-model reasoning prefix can redirect the trajectory toward a correct solution\.

However, this corrective effect is not reliably internalized by directly fine\-tuning on weak patches or repaired trajectories, suggesting that the useful signal lies not in the intervention text itself, but in how it reshapes the model’s future reasoning distribution\. We therefore proposeWoodpecker Distillation, a weak\-to\-strong training framework that learns from contrastive local interventions\. Our method contrasts successful and unsuccessful weak\-model patches at the same prefix, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model\.

Experiments on mathematical reasoning benchmarks show that Woodpecker Distillation consistently improves strong\-model performance and outperforms direct imitation baselines\. Code and datasets are available at[https://anonymous\.4open\.science/r/Woodpecker\-E6BD](https://anonymous.4open.science/r/Woodpecker-E6BD)\.

Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models

![Refer to caption](https://arxiv.org/html/2608.05168v1/x1.png)Figure 1:Woodpecker Distillation uses a weak probe model as a woodpecker to identify and repair local reasoning bugs in strong\-model reasoning\.## 1Introduction

Recent progress in language model reasoning has largely been driven by scaling, stronger supervision, and more effective inference\-time searchKaplanet al\.\([2020](https://arxiv.org/html/2608.05168#bib.bib8)\); Ouyanget al\.\([2022](https://arxiv.org/html/2608.05168#bib.bib9)\); Chunget al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib10)\); Yaoet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib11)\)\. Yet even strong models still make brittle errors on problems they appear capable of solvingHendryckset al\.\([2021](https://arxiv.org/html/2608.05168#bib.bib12)\); Cobbeet al\.\([2021](https://arxiv.org/html/2608.05168#bib.bib13)\)\. In this work, we focus on a simple empirical phenomenon: for some failed reasoning traces, inserting a short local guidance at an intermediate prefix can make the model’s subsequent continuation arrive at a correct solution\. We use the term*reasoning bug*to describe a recoverable failure mode in which a model’s current reasoning trajectory leads to an incorrect answer, but a short local guidance inserted at an intermediate prefix can redirect its subsequent continuation toward a correct solution\. This definition is operational: it does not require identifying a single erroneous token or step, but only tests whether the failure can be repaired by a local intervention\.

In this paper, we ask whether a weak probe model can help expose and repair such reasoning bugs in a stronger modelBurnset al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib19)\); Kentonet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib20)\)\. Given an intermediate reasoning prefix from a strong model, we insert a short reasoning patch generated by a weak probe model and then let the strong model continue\. We find that such patches can frequently turn otherwise incorrect continuations into correct solutionsPanet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib21)\); Huanget al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib22)\)\. This observation suggests that weak models may provide useful local corrective signals even when they are not stronger end\-to\-end problem solversZhouet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib24)\)\. In this view, the weak model is not used as a teacher that supplies complete solutions, but as a diagnostic probe that reveals which local interventions can improve the strong model’s continuation\.

However, directly training on these repaired trajectories is not straightforwardRosset al\.\([2011](https://arxiv.org/html/2608.05168#bib.bib25)\); Bengioet al\.\([2015](https://arxiv.org/html/2608.05168#bib.bib26)\)\. A natural approach is to fine\-tune the strong model to imitate the probe patch itself, or to imitate the full trajectory obtained after inserting the patchZelikmanet al\.\([2022](https://arxiv.org/html/2608.05168#bib.bib27)\); Hsiehet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib28)\); Kim and Rush \([2016](https://arxiv.org/html/2608.05168#bib.bib29)\)\. We find that these imitation\-based strategies are substantially less effective than the intervention that produced the repair\. This gap indicates that the useful signal is not simply the surface text of the patch, but how the patch changes the strong model’s future reasoning distribution after the same prefixHintonet al\.\([2015](https://arxiv.org/html/2608.05168#bib.bib30)\); Rusuet al\.\([2015](https://arxiv.org/html/2608.05168#bib.bib31)\); Lambet al\.\([2016](https://arxiv.org/html/2608.05168#bib.bib32)\)\. Internalizing this effect requires a training signal that captures the downstream consequence of a local intervention, rather than merely copying the intervention textRosset al\.\([2011](https://arxiv.org/html/2608.05168#bib.bib25)\); De Haanet al\.\([2019](https://arxiv.org/html/2608.05168#bib.bib33)\); Changet al\.\([2015](https://arxiv.org/html/2608.05168#bib.bib34)\)\.

We therefore proposeWoodpecker Distillation, a weak\-to\-strong training framework that converts local weak probe model interventions into corrective supervision\. For a given strong\-model prefix, we sample multiple weak probe model reasoning patches, evaluate whether each patch helps or harms the final outcome, and divide them into positive and negative intervention setsChristianoet al\.\([2017](https://arxiv.org/html/2608.05168#bib.bib35)\)\. We then compare the future token distributions induced by these two sets and construct a contrastive teacher distribution that emphasizes continuations associated with successful repair\. The strong model is trained to match this teacher, thereby learning the corrective effect of the intervention rather than directly imitating its wording\.

Our work makes three contributions\. First, we identify an empirical phenomenon in weak\-to\-strong reasoning: some strong\-model failures are recoverable reasoning bugs that can be exposed by short weak probe model interventions\. Second, we show that these bugs can often be repaired by local patches, but that direct imitation of patches or patched trajectories does not reliably internalize the repair\. Third, we introduce Woodpecker Distillation, a contrastive distillation method that turns successful and unsuccessful local interventions into token\-level corrective supervision and improves strong\-model reasoning\.

## 2Related Work

#### Weak\-to\-strong generalization and supervision\.

A growing literature studies how weaker supervision sources can improve stronger models, including weak\-to\-strong generalization, bootstrapping, and scalable oversightLanget al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib36)\); Somerstepet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib37)\); Huanget al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib38)\); Bowmanet al\.\([2022](https://arxiv.org/html/2608.05168#bib.bib39)\)\. Most prior work focuses on transferring labels, preferences, or global solution traces from weaker sources to stronger modelsRatneret al\.\([2016](https://arxiv.org/html/2608.05168#bib.bib40)\); Baiet al\.\([2022](https://arxiv.org/html/2608.05168#bib.bib41)\); Hoet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib42)\); Fuet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib43)\)\. In contrast, our setting is neither standard weak\-label supervision nor direct trace imitation: we use a weak probe model as a*diagnostic probe*that perturbs a strong model’s intermediate reasoning and reveals local failure modesMenget al\.\([2022](https://arxiv.org/html/2608.05168#bib.bib44)\); Ghandehariounet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib45)\)\.

#### Process supervision and reasoning improvement\.

Process supervision methods improve reasoning by supervising intermediate steps, reward models, or verifier\-guided searchLaiet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib46)\); Setluret al\.\([2025](https://arxiv.org/html/2608.05168#bib.bib47)\); Zhanget al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib48)\)\. These approaches typically assume access to gold intermediate annotations, strong verifiers, or explicit stepwise correctness signalsZhenget al\.\([2025](https://arxiv.org/html/2608.05168#bib.bib49)\)\. Our method instead uses*contrastive interventions*: we do not supervise which intermediate step is correct in isolation, but infer its usefulness from whether a local patch redirects the downstream reasoning trajectory toward a correct answer\.

#### Self\-distillation and preference\-based training\.

Self\-distillation and preference\-style optimization train models from improved targets or pairwise comparisonsFurlanelloet al\.\([2018](https://arxiv.org/html/2608.05168#bib.bib50)\); Guet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib51)\); Yuanet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib52)\)\. Our method is related in spirit, but differs in granularity and signal construction\. Rather than distilling from a stronger teacher or optimizing over full trajectories, Woodpecker Distillation builds a*localized distributional teacher*by contrasting successful and unsuccessful patches at the same reasoning prefixYuanet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib53)\); Zhaoet al\.\([2023](https://arxiv.org/html/2608.05168#bib.bib54)\)\. This lets us train on the future impact of local reasoning moves rather than on entire outputs\.

## 3Motivation: Local Bugs in Strong\-Model Reasoning

We first examine whether a small local guidance can change a failed continuation\. Given a problemxx, a strong modelAAgenerates a reasoning trajectoryyy, and we take an intermediate prefixc=y<tc=y\_\{<t\}\. We instantiateAAas Qwen3\-4B\-Instruct\-2507Team \([2025b](https://arxiv.org/html/2608.05168#bib.bib55)\)andBBas Gemma\-3\-4B\-ITTeam \([2025a](https://arxiv.org/html/2608.05168#bib.bib56)\), and evaluate on AIME 2024, AIME 2025, MATH\-500, Olympiad, and Omni\-Hard, where Omni\-Hard contains Omni\-MATH problems with difficulty greater than 7Zhang and Math\-AI \([2024](https://arxiv.org/html/2608.05168#bib.bib59),[2025](https://arxiv.org/html/2608.05168#bib.bib60)\); Hendryckset al\.\([2021](https://arxiv.org/html/2608.05168#bib.bib12)\); Heet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib61)\); Gaoet al\.\([2024](https://arxiv.org/html/2608.05168#bib.bib62)\)\. We only evaluate problems on which modelAAfails under greedy decoding\. We compare two continuations from the same prefix: direct continuation from\(x,c\)\(x,c\), and patched continuation from\(x,c,b\)\(x,c,b\), wherebbis a short guidance patch generated by the weak probe modelBB\. Table[1](https://arxiv.org/html/2608.05168#S3.T1)shows that this A–B–A intervention improves Pass@16, with larger gains on harder benchmarks such as AIME 2025 and Omni\-Hard\. We call this recoverable failure mode a*reasoning bug*: the original continuation fails, but a short local guidance can redirect it toward a correct answer\.

Table 1:Comparison between direct continuation and A–B–A patched continuation under the same sampling budget\. All numbers are percentages, andΔ\\Deltadenotes absolute percentage\-point improvement\.This phenomenon suggests that weak probe model patches can reveal useful local repair signals\. The weak probe model need not solve the full problem; it only needs to provide a short intervention that changes the strong model’s subsequent continuation\.

However, directly imitating these repairs is unreliable\. Fine\-tuning on the weak\-model patch learns only what the weak probe model says, while fine\-tuning on the full patched trajectory may dilute the local repair signal among many downstream continuation tokens\. Direct likelihood training can therefore overfit to the surface form of repaired examples or perturb the strong model’s original reasoning distribution, rather than internalize why the local intervention was helpful\.

These observations motivate Woodpecker Distillation: instead of copying weak\-model patches or repaired trajectories, we train the strong model on how successful and unsuccessful local interventions change its future token distribution after the same prefix\.

![Refer to caption](https://arxiv.org/html/2608.05168v1/x2.png)Figure 2:Overview of Woodpecker Distillation\.
## 4Method

### 4\.1Overview

Given a strong modelAA, we maintain two copies: a trainable modelAθA\_\{\\theta\}and a frozen reference modelArefA\_\{\\mathrm\{ref\}\}used to construct soft training targets\. We also use a weak probe modelBB\. Woodpecker Distillation is a weak\-guided self\-distillation framework:BBproposes local patches,ArefA\_\{\\mathrm\{ref\}\}compares their induced future token distributions, andAθA\_\{\\theta\}learns the resulting corrective distribution on the original, unpatched context\.

In this way, the weak probe model is not a teacher to imitate\. It only provides interventions that reveal how the strong model’s continuation changes\. The supervision is produced by contrasting successful and unsuccessful patch\-induced distributions underArefA\_\{\\mathrm\{ref\}\}, and the trainable model internalizes this effect without relying on weak\-model patches during training or inference\. Figure[2](https://arxiv.org/html/2608.05168#S3.F2)provides an overview of the Woodpecker Distillation pipeline\.

### 4\.2Step 1: Insert Weak Patches

For each problemxx, the strong model first samples a direct reasoning trajectory

y=\(y1,…,yT\)\.y=\(y\_\{1\},\\ldots,y\_\{T\}\)\.We select one or more candidate insertion positionsssin this trajectory and denote the corresponding prefix by

Conditioned on the same problem and prefix, the weak probe model generatesKKshort candidate patches:

\{bi\}i=1K∼B\(⋅∣x,cs\)\.\\\{b\_\{i\}\\\}\_\{i=1\}^\{K\}\\sim B\(\\cdot\\mid x,c\_\{s\}\)\.Each patch is treated as optional local guidance rather than a target to imitate\.

### 4\.3Step 2: Repair Test and Split

We test each candidate patch by inserting it after the prefixcsc\_\{s\}and letting the strong model continue from the patched context\. The completed solution is judged by an external verifier\. According to the final correctness, patches at positionssare divided into successful and unsuccessful sets:

ℬs\+=\{bi:Verify​\(A​\(x,cs,bi\)\)=1\},\\mathcal\{B\}^\{\+\}\_\{s\}=\\\{b\_\{i\}:\\mathrm\{Verify\}\(A\(x,c\_\{s\},b\_\{i\}\)\)=1\\\},ℬs−=\{bi:Verify​\(A​\(x,cs,bi\)\)=0\}\.\\mathcal\{B\}^\{\-\}\_\{s\}=\\\{b\_\{i\}:\\mathrm\{Verify\}\(A\(x,c\_\{s\},b\_\{i\}\)\)=0\\\}\.We keep an insertion position for training only when bothℬs\+\\mathcal\{B\}^\{\+\}\_\{s\}andℬs−\\mathcal\{B\}^\{\-\}\_\{s\}are non\-empty\. Positions where all patches succeed or all patches fail are discarded, since they do not provide a contrastive intervention signal\.

### 4\.4Step 3: Build Teacher and Train

For each retained positionss, we construct supervision on a short future window of the direct trajectory:

Ws=\{s,…,s\+L−1\}\.W\_\{s\}=\\\{s,\\ldots,s\+L\-1\\\}\.The direct trajectory is used only to anchor local token positions; the target is not to imitate the original tokenyτy\_\{\\tau\}\. Instead, we compare how successful and unsuccessful patches change the reference model’s next\-token distribution\.

Leth​\(x,cs,b\)h\(x,c\_\{s\},b\)denote the patched context that presentsbbas local guidance after prefixcsc\_\{s\}\. The prompt is designed to avoid forcing the strong model to imitate the weak\-model patch: the model is explicitly instructed to use the candidate only when it is mathematically helpful\.

Patched Prompt TemplateProblem\{x\}Current reasoning prefix\{c\_s\}Candidate local reasoning turn\{b\}*Use the candidate only if it is mathematically helpful\.*Continue solving the problem step by step\.

For each patchbband positionτ∈Ws\\tau\\in W\_\{s\}, we evaluate the frozen reference model:

pτb\(⋅\)=pref\(⋅∣h\(x,cs,b\),ys:τ\)\.p^\{b\}\_\{\\tau\}\(\\cdot\)=p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid h\(x,c\_\{s\},b\),y\_\{s:\\tau\}\)\.We then aggregate the patch\-induced distributions over successful and unsuccessful patches:

pτ\+=1\|ℬs\+\|​∑b∈ℬs\+pτb,pτ−=1\|ℬs−\|​∑b∈ℬs−pτb\.p^\{\+\}\_\{\\tau\}=\\frac\{1\}\{\|\\mathcal\{B\}^\{\+\}\_\{s\}\|\}\\sum\_\{b\\in\\mathcal\{B\}^\{\+\}\_\{s\}\}p^\{b\}\_\{\\tau\},\\qquad p^\{\-\}\_\{\\tau\}=\\frac\{1\}\{\|\\mathcal\{B\}^\{\-\}\_\{s\}\|\}\\sum\_\{b\\in\\mathcal\{B\}^\{\-\}\_\{s\}\}p^\{b\}\_\{\\tau\}\.Herepτ\+p^\{\+\}\_\{\\tau\}captures future token predictions associated with helpful interventions, whilepτ−p^\{\-\}\_\{\\tau\}captures those associated with unhelpful interventions\.

We construct a corrective soft teacher relative to the reference model’s original prediction on the unpatched trajectory:

pτ0=pref\(⋅∣x,y<τ\)\.p^\{0\}\_\{\\tau\}=p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x,y\_\{<\\tau\}\)\.For each vocabulary tokenvv, define

Δτ​\(v\)=clip​\(log⁡pτ\+​\(v\)−log⁡pτ−​\(v\),−δ,δ\),\\Delta\_\{\\tau\}\(v\)=\\mathrm\{clip\}\\left\(\\log p^\{\+\}\_\{\\tau\}\(v\)\-\\log p^\{\-\}\_\{\\tau\}\(v\),\-\\delta,\\delta\\right\),and

qτ​\(v\)=pτ0​\(v\)​exp⁡\(η​Δτ​\(v\)\)∑v′pτ0​\(v′\)​exp⁡\(η​Δτ​\(v′\)\)\.q\_\{\\tau\}\(v\)=\\frac\{p^\{0\}\_\{\\tau\}\(v\)\\exp\\left\(\\eta\\Delta\_\{\\tau\}\(v\)\\right\)\}\{\\sum\_\{v^\{\\prime\}\}p^\{0\}\_\{\\tau\}\(v^\{\\prime\}\)\\exp\\left\(\\eta\\Delta\_\{\\tau\}\(v^\{\\prime\}\)\\right\)\}\.The parameterη\\etacontrols the correction strength, andδ\\deltaprevents extreme log\-ratio shifts\.

Table 2:Greedy decoding accuracy on mathematical reasoning benchmarks\. The average is weighted by the number of problems in each dataset\.To focus training on informative positions, we use a Jensen–Shannon gate:

mτ=clip​\(JS​\(pτ\+,pτ−\)τjs,0,1\)\.m\_\{\\tau\}=\\mathrm\{clip\}\\left\(\\frac\{\\mathrm\{JS\}\(p^\{\+\}\_\{\\tau\},p^\{\-\}\_\{\\tau\}\)\}\{\\tau\_\{\\mathrm\{js\}\}\},0,1\\right\)\.When successful and unsuccessful patches induce similar distributions, the position receives little weight\.

Finally, the trainable model is optimized on the original, unpatched context:

ℒWD=∑s∑τ∈WsmτCE\(qτ,πθ\(⋅∣x,y<τ\)\)\.\\mathcal\{L\}\_\{\\mathrm\{WD\}\}=\\sum\_\{s\}\\sum\_\{\\tau\\in W\_\{s\}\}m\_\{\\tau\}\\,\\mathrm\{CE\}\\left\(q\_\{\\tau\},\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<\\tau\}\)\\right\)\.Thus,AθA\_\{\\theta\}never observes the weak\-model patch during optimization\. The patch is used only to construct the soft teacher, while the student learns to reproduce the corrective distributional effect from the original context alone\.

## 5Experiments

### 5\.1Experimental Setup

#### Models\.

We instantiate the strong modelAAwith Qwen3\-4B\-Instruct\-2507 and the weak probe modelBBwith Gemma\-3\-4B\-IT\. The trainable copyAθA\_\{\\theta\}is initialized from Qwen3\-4B\-Instruct\-2507, whileArefA\_\{\\mathrm\{ref\}\}is kept frozen to construct soft training targets\.

#### Datasets\.

We evaluate all models on five mathematical reasoning benchmarks: MATH\-500, Olympiad\-test, Omni\-Hard, AIME 2024, and AIME 2025\. For each generated response, we extract the final answer using regular\-expression rules and verify correctness with themath\-verifytoolkit\. To ensure fair comparison, all methods are evaluated with the same system prompt, conversation template, and context\-length setting\.

For training, we sample problems from Skywork\-OR1\-RL\-Data\(Heet al\.,[2025a](https://arxiv.org/html/2608.05168#bib.bib57),[b](https://arxiv.org/html/2608.05168#bib.bib58)\)\. We remove all examples that are identical to, or substantially overlap with, the evaluation benchmarks before training\. Additional details on training and experimental settings are provided in Appendix[A](https://arxiv.org/html/2608.05168#A1)\.

### 5\.2Baselines

We compare Woodpecker Distillation with the following baselines\.

#### Base strong model\.

This is the original strong modelAAwithout any additional training\. It measures the starting reasoning ability of the target model\.

#### Weak probe model\.

This is the original Weak probe modelBB\.

#### Self\-rejection sampling fine\-tuning\.

Using the same training dataset and sampling budget as our method, this baseline filters verifier\-correct trajectories from strong\-model self\-sampling and fine\-tunes on the resulting successful solutions\.

#### Woodpecker Distillation\.

Our method does not imitate the weak\-model patch or the patched trajectory\. Instead, it contrasts successful and unsuccessful interventions and distills their induced future token distributions into the strong model\.

### 5\.3Main Results: Greedy Evaluation

Table[2](https://arxiv.org/html/2608.05168#S4.T2)reports greedy decoding accuracy across five mathematical reasoning benchmarks\. Woodpecker Distillation improves the base strong model on all datasets, increasing the average accuracy from 53\.4 to 54\.8\. The gains are most pronounced on AIME 2024 and AIME 2025, where accuracy improves by 10\.0 and 13\.3 percentage points, respectively\.

Woodpecker Distillation also outperforms self\-rejection SFT under the same training data and sampling budget\. This comparison suggests that the improvement is not simply due to filtering verifier\-correct trajectories from the strong model\. Instead, using verifier outcomes to contrast successful and unsuccessful local interventions provides a more targeted supervision signal than training on complete self\-generated correct solutions\.

### 5\.4Ablation Studies

Table 3:Ablation results on mathematical reasoning benchmarks under greedy decoding\.Table 4:Robustness of Woodpecker Distillation to the choice of probe model\. The strong model is fixed as Qwen3\-4B\-Instruct\-2507, while the auxiliary probe model is varied between Gemma\-3\-4B\-IT and Mistral\-7B\-Instruct\-v0\.3\. All results are greedy decoding pass@1 accuracy in percentage points over the same 1,763\-problem evaluation suite\.We conduct ablations to evaluate the main design choices of Woodpecker Distillation\. Table[3](https://arxiv.org/html/2608.05168#S5.T3)shows that each component contributes to the final performance\.

#### Same\-model probe\.

We first replace the weak probe modelBBwith the strong model itself, i\.e\.,A=BA=B\. This variant removes the asymmetric weak\-to\-strong probing setup while keeping the rest of the method unchanged\. Its average accuracy drops from 54\.8 to 53\.4, suggesting that weak\-model patches provide useful intervention diversity beyond self\-generated perturbations from the strong model\.

#### Removing negative interventions\.

We next remove the negative intervention distribution and construct the teacher only from successful patches\. This variant collapses substantially, with average accuracy dropping to 22\.1\. Without the negative distribution, the teacher lacks a reference direction for suppressing harmful intervention\-induced shifts, which can over\-amplify patch\-specific artifacts\.

#### Removing JS gating\.

Finally, we remove the Jensen–Shannon gate by settingmτ=1m\_\{\\tau\}=1for all positions\. Performance decreases from 54\.8 to 53\.0, showing that not all token positions provide equally reliable repair signals\. Weighting positions by the divergence between positive and negative intervention distributions helps focus training on locations where the contrast is informative\.

Overall, these ablations support the three core ingredients of Woodpecker Distillation: weak\-model probing provides useful local diversity, positive\-negative contrast defines the corrective direction, and JS gating filters noisy or uninformative positions\.

### 5\.5Robustness Analysis

Table[4](https://arxiv.org/html/2608.05168#S5.T4)evaluates the robustness of Woodpecker Distillation under different random seeds and probe models\. The results remain stable across two Gemma\-3\-4B\-IT runs and when replacing the probe with Mistral\-7B\-Instruct\-v0\.3, providing initial evidence that the method is not tied to a single weak probe model instance\.

### 5\.6Cost Analysis

![Refer to caption](https://arxiv.org/html/2608.05168v1/x3.png)Figure 3:Average per\-step runtime breakdown of Woodpecker Distillation, measured on H800 GPUs\.We analyze the per\-step runtime of Woodpecker Distillation in Figure[3](https://arxiv.org/html/2608.05168#S5.F3)\. Each training step consists of four components: generating the strong\-model prefix, generating weak\-model branches, verifying patched continuations, and applying gradient updates\. The runtime is dominated by verification, which takes 1182 seconds on average and accounts for 86\.4% of the total per\-step time\. In contrast, weak\-branch generation takes only 8 seconds, contributing less than 1% of the total cost\.

This breakdown shows that the main overhead of Woodpecker Distillation does not come from the weak probe model itself\. Since the weak probe model only generates short local patches, its cost is small compared with evaluating completed continuations\. Training takes 125 seconds per step, indicating that optimization is also not the primary bottleneck\. Therefore, the efficiency of the method is mainly determined by the cost of verifier\-based evaluation, rather than by weak\-model patch generation\.

## 6Analysis

Woodpecker Distillation constructs a contrastive soft teacher from local interventions\. In this section, we provide a conservative interpretation of this teacher\. Our goal is not to prove that the resulting token updates causally guarantee better final answers\. Instead, we show that the teacher can be viewed as a KL\-regularized distributional update toward tokens that are more associated with successful local interventions than unsuccessful ones\.

### 6\.1Patch\-Conditioned Intervention Distributions

Fix a problemxxand a strong\-model prefixcc\. A weak probe model samples a local patch

b∼B\(⋅∣x,c\)\.b\\sim B\(\\cdot\\mid x,c\)\.After insertingbbafter the prefix, the strong model continues from the patched context\. Let

S​\(b\)∈\{0,1\}S\(b\)\\in\\\{0,1\\\}denote the verifier outcome of this patched continuation, whereS​\(b\)=1S\(b\)=1means that the patched continuation reaches a correct final answer\. This outcome is used only to partition patches; it is not treated as a token\-level label\.

The successful and unsuccessful patch distributions are defined as

μ\+​\(b∣x,c\)∝B​\(b∣x,c\)​𝟏​\{S​\(b\)=1\},\\mu^\{\+\}\(b\\mid x,c\)\\propto B\(b\\mid x,c\)\\mathbf\{1\}\\\{S\(b\)=1\\\},and

μ−​\(b∣x,c\)∝B​\(b∣x,c\)​𝟏​\{S​\(b\)=0\}\.\\mu^\{\-\}\(b\\mid x,c\)\\propto B\(b\\mid x,c\)\\mathbf\{1\}\\\{S\(b\)=0\\\}\.In practice, these distributions are estimated by averaging over the finite sets of successful and unsuccessful sampled patches\.

For a future token positionτ\\tau, we evaluate the frozen reference model under each patched context and aggregate the resulting next\-token distributions:

pτ\+​\(v\)=𝔼b∼μ\+​\[pref​\(v∣h​\(x,c,b\),ys:τ\)\],p^\{\+\}\_\{\\tau\}\(v\)=\\mathbb\{E\}\_\{b\\sim\\mu^\{\+\}\}\\left\[p\_\{\\mathrm\{ref\}\}\(v\\mid h\(x,c,b\),y\_\{s:\\tau\}\)\\right\],pτ−​\(v\)=𝔼b∼μ−​\[pref​\(v∣h​\(x,c,b\),ys:τ\)\]\.p^\{\-\}\_\{\\tau\}\(v\)=\\mathbb\{E\}\_\{b\\sim\\mu^\{\-\}\}\\left\[p\_\{\\mathrm\{ref\}\}\(v\\mid h\(x,c,b\),y\_\{s:\\tau\}\)\\right\]\.Hereh​\(x,c,b\)h\(x,c,b\)denotes the context obtained by inserting patchbbafter prefixcc, andys:τy\_\{s:\\tau\}is the original trajectory segment used as a common teacher\-forced alignment path\. This alignment avoids comparing next\-token distributions at mismatched free\-running continuations, but it should be viewed as a local approximation rather than a full trajectory\-level estimate\.

We define the contrastive repair score

rτ​\(v\)=log⁡pτ\+​\(v\)−log⁡pτ−​\(v\)\.r\_\{\\tau\}\(v\)=\\log p^\{\+\}\_\{\\tau\}\(v\)\-\\log p^\{\-\}\_\{\\tau\}\(v\)\.This score should be interpreted as a contrastive proxy for local repair value: it measures how strongly tokenvvis associated with distributions induced by successful interventions relative to unsuccessful ones under the same prefix\.

###### Proposition 1\(Positive\-negative contrast\)\.

Ifrτ​\(v\)\>0r\_\{\\tau\}\(v\)\>0, then tokenvvreceives higher probability under the successful\-intervention distribution than under the unsuccessful\-intervention distribution\. Ifrτ​\(v\)<0r\_\{\\tau\}\(v\)<0, then tokenvvis relatively more associated with unsuccessful interventions\.

This proposition is only an associational statement\. It does not claim that increasing the probability ofvvalone will necessarily make the final answer correct\. Rather, it motivates usingrτ​\(v\)r\_\{\\tau\}\(v\)as a local directional signal for aligning the model’s prediction with the distributional effect of successful interventions\.

### 6\.2KL\-Regularized Contrastive Teacher

Letpτ0p^\{0\}\_\{\\tau\}denote the frozen reference model’s original prediction on the unpatched context:

pτ0​\(v\)=pref​\(v∣x,y<τ\)\.p^\{0\}\_\{\\tau\}\(v\)=p\_\{\\mathrm\{ref\}\}\(v\\mid x,y\_\{<\\tau\}\)\.We clip the contrastive score to reduce the effect of unstable or extreme log\-ratios:

r~τ​\(v\)=clip​\(log⁡pτ\+​\(v\)−log⁡pτ−​\(v\),−δ,δ\)\.\\tilde\{r\}\_\{\\tau\}\(v\)=\\mathrm\{clip\}\\left\(\\log p^\{\+\}\_\{\\tau\}\(v\)\-\\log p^\{\-\}\_\{\\tau\}\(v\),\-\\delta,\\delta\\right\)\.
For each token positionτ\\tau, consider the following optimization over distributionsqq:

qτ∗=arg⁡maxq⁡\[𝔼v∼q​\[r~τ​\(v\)\]−1η​KL​\(q∥pτ0\)\],q^\{\*\}\_\{\\tau\}=\\arg\\max\_\{q\}\\left\[\\mathbb\{E\}\_\{v\\sim q\}\\left\[\\tilde\{r\}\_\{\\tau\}\(v\)\\right\]\-\\frac\{1\}\{\\eta\}\\mathrm\{KL\}\(q\\\|p^\{0\}\_\{\\tau\}\)\\right\],whereη\>0\\eta\>0controls the strength of the update\. The first term encourages probability mass on tokens associated with successful interventions, while the KL term keeps the teacher close to the original strong\-model prediction\.

###### Proposition 2\(KL\-regularized tilted teacher\)\.

The solution to the above optimization is

qτ∗​\(v\)=pτ0​\(v\)​exp⁡\(η​r~τ​\(v\)\)∑v′pτ0​\(v′\)​exp⁡\(η​r~τ​\(v′\)\)\.q^\{\*\}\_\{\\tau\}\(v\)=\\frac\{p^\{0\}\_\{\\tau\}\(v\)\\exp\\left\(\\eta\\tilde\{r\}\_\{\\tau\}\(v\)\\right\)\}\{\\sum\_\{v^\{\\prime\}\}p^\{0\}\_\{\\tau\}\(v^\{\\prime\}\)\\exp\\left\(\\eta\\tilde\{r\}\_\{\\tau\}\(v^\{\\prime\}\)\\right\)\}\.

###### Proof\.

The Lagrangian for the constrained optimization over probability distributions is

𝒥​\(q,λ\)=\\displaystyle\\mathcal\{J\}\(q,\\lambda\)=∑vq​\(v\)​r~τ​\(v\)−1η​∑vq​\(v\)​log⁡q​\(v\)pτ0​\(v\)\\displaystyle\\sum\_\{v\}q\(v\)\\tilde\{r\}\_\{\\tau\}\(v\)\-\\frac\{1\}\{\\eta\}\\sum\_\{v\}q\(v\)\\log\\frac\{q\(v\)\}\{p^\{0\}\_\{\\tau\}\(v\)\}\+λ​\(∑vq​\(v\)−1\)\.\\displaystyle\\quad\+\\lambda\\left\(\\sum\_\{v\}q\(v\)\-1\\right\)\.Taking the derivative with respect toq​\(v\)q\(v\)and setting it to zero gives

r~τ​\(v\)−1η​\(log⁡q​\(v\)pτ0​\(v\)\+1\)\+λ=0\.\\tilde\{r\}\_\{\\tau\}\(v\)\-\\frac\{1\}\{\\eta\}\\left\(\\log\\frac\{q\(v\)\}\{p^\{0\}\_\{\\tau\}\(v\)\}\+1\\right\)\+\\lambda=0\.Therefore,

q​\(v\)∝pτ0​\(v\)​exp⁡\(η​r~τ​\(v\)\)\.q\(v\)\\propto p^\{0\}\_\{\\tau\}\(v\)\\exp\\left\(\\eta\\tilde\{r\}\_\{\\tau\}\(v\)\\right\)\.Normalizing over the vocabulary gives the stated form\. ∎

This proposition shows that the teacher is an exponentially tilted version of the original reference distribution\. The tilt is determined by the clipped positive\-negative contrast, while the KL term prevents the target from moving arbitrarily far from the model’s original prediction\. Ifpτ\+=pτ−p^\{\+\}\_\{\\tau\}=p^\{\-\}\_\{\\tau\}, thenr~τ​\(v\)=0\\tilde\{r\}\_\{\\tau\}\(v\)=0for allvvandqτ∗=pτ0q^\{\*\}\_\{\\tau\}=p^\{0\}\_\{\\tau\}, so the teacher reduces to the original reference distribution\.

### 6\.3Contrast Weighting with Jensen\-Shannon Divergence

The contrastive score is useful only when successful and unsuccessful patches induce meaningfully different future\-token distributions\. Ifpτ\+p^\{\+\}\_\{\\tau\}andpτ−p^\{\-\}\_\{\\tau\}are nearly identical, their log\-ratio is likely to provide little useful training signal\. We therefore weight each position by

mτ=clip​\(JS​\(pτ\+,pτ−\)τjs,0,1\),m\_\{\\tau\}=\\mathrm\{clip\}\\left\(\\frac\{\\mathrm\{JS\}\(p^\{\+\}\_\{\\tau\},p^\{\-\}\_\{\\tau\}\)\}\{\\tau\_\{\\mathrm\{js\}\}\},0,1\\right\),whereJS​\(⋅,⋅\)\\mathrm\{JS\}\(\\cdot,\\cdot\)is the Jensen\-Shannon divergence\.

###### Proposition 3\(Contrast\-based weighting\)\.

The weightmτm\_\{\\tau\}is larger when the successful and unsuccessful intervention distributions are more separated, and becomes small when the two distributions are similar\.

We use this weight as a heuristic measure of contrast strength, not as a formal statistical confidence estimate\. Sincepτ\+p^\{\+\}\_\{\\tau\}andpτ−p^\{\-\}\_\{\\tau\}are finite\-sample estimates, large divergence can still be noisy when the number of sampled patches is small\. The clipping ofr~τ\\tilde\{r\}\_\{\\tau\}and the weighting bymτm\_\{\\tau\}are therefore practical stabilization mechanisms rather than guarantees of causal correctness\.

The final training objective is

ℒWD=∑τmτCE\(qτ∗,πθ\(⋅∣x,y<τ\)\)\.\\mathcal\{L\}\_\{\\mathrm\{WD\}\}=\\sum\_\{\\tau\}m\_\{\\tau\}\\,\\mathrm\{CE\}\\left\(q^\{\*\}\_\{\\tau\},\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<\\tau\}\)\\right\)\.The trainable model is optimized on the original unpatched context\. Thus, weak\-model patches are used only to construct the contrastive soft teacher; they are not directly imitated during training\.

## 7Conclusion

We introduced Woodpecker Distillation, a weak\-to\-strong training framework that uses weak\-model patches as diagnostic probes for localized reasoning failures in strong language models\. By contrasting the future token distributions induced by successful and unsuccessful local interventions, our method constructs a corrective soft teacher and distills it into the strong model on the original unpatched context\. Experiments on mathematical reasoning benchmarks show that this contrastive local supervision improves greedy reasoning accuracy and outperforms direct self\-rejection fine\-tuning, suggesting that weak probe models can provide useful corrective signals even when they are not stronger end\-to\-end reasoners\.

## 8Limitations

Our work has several limitations\. First, the current evaluation focuses on mathematical reasoning benchmarks, so further experiments are needed to test whether the same form of local repair transfers to broader reasoning domains, such as code generation, factual reasoning, or multi\-step planning\. Second, Woodpecker Distillation depends on verifier\-based feedback to separate successful and unsuccessful interventions; this makes the method most directly applicable to domains where reliable automatic verification is available\. Third, the main computational bottleneck is evaluating patched continuations, rather than generating weak\-model patches or performing training\. Reducing this verification cost is an important direction for making the method more scalable\. Finally, our method targets recoverable local reasoning bugs\. It may be less effective when the strong model lacks the underlying capability to solve the problem, or when failures arise from global planning errors rather than locally repairable reasoning steps\.

## References

- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. Shazeer \(2015\)Scheduled sampling for sequence prediction with recurrent neural networks\.Advances in neural information processing systems28\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- S\. R\. Bowman, J\. Hyun, E\. Perez, E\. Chen, C\. Pettit, S\. Heiner, K\. Lukošiūtė, A\. Askell, A\. Jones, A\. Chen,et al\.\(2022\)Measuring progress on scalable oversight for large language models\.arXiv preprint arXiv:2211\.03540\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike,et al\.\(2023\)Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.arXiv preprint arXiv:2312\.09390\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p2.1)\.
- K\. Chang, A\. Krishnamurthy, A\. Agarwal, H\. Daumé III, and J\. Langford \(2015\)Learning to search better than your teacher\.InInternational Conference on Machine Learning,pp\. 2058–2066\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p4.1)\.
- H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.\(2024\)Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1)\.
- P\. De Haan, D\. Jayaraman, and S\. Levine \(2019\)Causal confusion in imitation learning\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- Y\. Fu, H\. Peng, L\. Ou, A\. Sabharwal, and T\. Khot \(2023\)Specializing smaller language models towards multi\-step reasoning\.InInternational Conference on Machine Learning,pp\. 10421–10430\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Furlanello, Z\. Lipton, M\. Tschannen, L\. Itti, and A\. Anandkumar \(2018\)Born again neural networks\.InInternational conference on machine learning,pp\. 1607–1616\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Gao, F\. Song, Z\. Yang, Z\. Cai, Y\. Miao, Q\. Dong, L\. Li, C\. Ma, L\. Chen, R\. Xu, Z\. Tang, B\. Wang, D\. Zan, S\. Quan, G\. Zhang, L\. Sha, Y\. Zhang, X\. Ren, T\. Liu, and B\. Chang \(2024\)Omni\-math: a universal olympiad level mathematic benchmark for large language models\.External Links:2410\.07985,[Link](https://arxiv.org/abs/2410.07985)Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- A\. Ghandeharioun, A\. Caciularu, A\. Pearce, L\. Dixon, and M\. Geva \(2024\)Patchscopes: a unifying framework for inspecting hidden representations of language models\.arXiv preprint arXiv:2401\.06102\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)Minillm: knowledge distillation of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 32694–32717\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px3.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen, S\. Li, L\. Zeng, T\. Wei, C\. Cheng, B\. An, Y\. Liu, and Y\. Zhou \(2025a\)Skywork open reasoner 1 technical report\.arXiv preprint arXiv:2505\.22312\.Cited by:[§5\.1](https://arxiv.org/html/2608.05168#S5.SS1.SSS0.Px2.p2.1)\.
- J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen, S\. Li, L\. Zeng, T\. Wei, C\. Cheng, Y\. Liu, and Y\. Zhou \(2025b\)Skywork open reasoner series\.Note:[https://capricious\-hydrogen\-41c\.notion\.site/Skywork\-Open\-Reaonser\-Series\-1d0bc9ae823a80459b46c149e4f51680](https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaonser-Series-1d0bc9ae823a80459b46c149e4f51680)Notion BlogCited by:[§5\.1](https://arxiv.org/html/2608.05168#S5.SS1.SSS0.Px2.p2.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1),[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- N\. Ho, L\. Schmid, and S\. Yun \(2023\)Large language models are reasoning teachers\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 14852–14882\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 8003–8017\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- J\. Huang, S\. Gu, L\. Hou, Y\. Wu, X\. Wang, H\. Yu, and J\. Han \(2023\)Large language models can self\-improve\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 1051–1068\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. Yu, X\. Song, and D\. Zhou \(2024\)Large language models cannot self\-correct reasoning yet\.InInternational conference on learning representations,Vol\.2024,pp\. 32808–32824\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p2.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1)\.
- Z\. Kenton, N\. Y\. Siegel, J\. Kramár, J\. Brown\-Cohen, S\. Albanie, J\. Bulian, R\. Agarwal, D\. Lindner, Y\. Tang, N\. D\. Goodman,et al\.\(2024\)On scalable oversight with weak llms judging strong llms\.Advances in Neural Information Processing Systems37,pp\. 75229–75276\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p2.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- X\. Lai, Z\. Tian, Y\. Chen, S\. Yang, X\. Peng, and J\. Jia \(2024\)Step\-dpo: step\-wise preference optimization for long\-chain reasoning of llms\.arXiv preprint arXiv:2406\.18629\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px2.p1.1)\.
- A\. M\. Lamb, A\. G\. ALIAS PARTH GOYAL, Y\. Zhang, S\. Zhang, A\. C\. Courville, and Y\. Bengio \(2016\)Professor forcing: a new algorithm for training recurrent networks\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- H\. Lang, D\. Sontag, and A\. Vijayaraghavan \(2024\)Theoretical analysis of weak\-to\-strong generalization\.Advances in neural information processing systems37,pp\. 46837–46880\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.Advances in neural information processing systems35,pp\. 17359–17372\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1)\.
- L\. Pan, M\. Saxon, W\. Xu, D\. Nathani, X\. Wang, and W\. Y\. Wang \(2023\)Automatically correcting large language models: surveying the landscape of diverse self\-correction strategies\.arXiv preprint arXiv:2308\.03188\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p2.1)\.
- A\. J\. Ratner, C\. M\. De Sa, S\. Wu, D\. Selsam, and C\. Ré \(2016\)Data programming: creating large training sets, quickly\.Advances in neural information processing systems29\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ross, G\. Gordon, and D\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- A\. A\. Rusu, S\. G\. Colmenarejo, C\. Gulcehre, G\. Desjardins, J\. Kirkpatrick, R\. Pascanu, V\. Mnih, K\. Kavukcuoglu, and R\. Hadsell \(2015\)Policy distillation\.arXiv preprint arXiv:1511\.06295\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. Kumar \(2025\)Rewarding progress: scaling automated process verifiers for llm reasoning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 60808–60838\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Somerstep, F\. M\. Polo, M\. Banerjee, Y\. Ritov, M\. Yurochkin, and Y\. Sun \(2024\)A statistical framework for weak\-to\-strong generalization\.InICML 2024 Next Generation of AI Safety Workshop,Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Team \(2025a\)Gemma 3\.External Links:[Link](https://goo.gle/Gemma3Report)Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- Q\. Team \(2025b\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.Advances in neural information processing systems36,pp\. 11809–11822\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. Weston \(2024\)Self\-rewarding language models\.arXiv preprint arXiv:2401\.10020\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Tan, W\. Wang, S\. Huang, and F\. Huang \(2023\)Rrhf: rank responses to align language models with human feedback without tears\.arXiv preprint arXiv:2304\.05302\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px3.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. Goodman \(2022\)Star: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p3.1)\.
- D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. Tang \(2024\)Rest\-mcts\*: llm self\-training via process reward guided tree search\.Advances in Neural Information Processing Systems37,pp\. 64735–64772\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- Y\. Zhang and T\. Math\-AI \(2025\)American invitational mathematics examination \(aime\) 2025\.Cited by:[§3](https://arxiv.org/html/2608.05168#S3.p1.11)\.
- Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu \(2023\)Slic\-hf: sequence likelihood calibration with human feedback\.arXiv preprint arXiv:2305\.10425\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Zheng, Z\. Zhang, B\. Zhang, R\. Lin, K\. Lu, B\. Yu, D\. Liu, J\. Zhou, and J\. Lin \(2025\)Processbench: identifying process errors in mathematical reasoning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1009–1024\.Cited by:[§2](https://arxiv.org/html/2608.05168#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhou, Z\. Liu, J\. Liu, Z\. Dong, C\. Yang, and Y\. Qiao \(2024\)Weak\-to\-strong search: align large language models via searching over small language models\.Advances in Neural Information Processing Systems37,pp\. 4819–4851\.Cited by:[§1](https://arxiv.org/html/2608.05168#S1.p2.1)\.

## Appendix AExperimental Details

#### Training data processing\.

We construct the training set from the mathematical training split of Skywork\-OR1\-RL\-Data\. To avoid benchmark contamination, we first remove problems that are identical or highly similar to examples in the evaluation benchmarks\. From the remaining pool, we select problems whose difficulty scores under DeepSeek\-R1\-Distill\-Qwen\-32B fall between 12 and 15\. We then evaluate Qwen3\-4B\-Instruct\-2507 with eight sampled responses per problem and retain 2,956 problems with partial success, i\.e\., problems whose Pass@8 count is greater than 0 and smaller than 8\. To further include challenging examples that the actor cannot solve under its current sampling distribution, we additionally select 800 problems with the shortest labels from the subset with Pass@8 count equal to 0\. This yields a final training set of 3,756 mathematical reasoning problems\.

#### Implementation hyperparameters\.

Unless otherwise specified, we use Qwen3\-4B\-Instruct\-2507 as the strong modelAA, including the trainable modelAθA\_\{\\theta\}and the frozen reference modelArefA\_\{\\mathrm\{ref\}\}, and Gemma\-3\-4B\-IT as the weak probe modelBB\. For each problem, the strong model first generates a direct trajectory with greedy decoding\. We use candidate insertion positionss∈\{100,200,400\}s\\in\\\{100,200,400\\\}and sampleK=16K=16weak\-model patches at each position\. Each patch is generated with temperature1\.01\.0and top\-p=0\.95p=0\.95, with a maximum length of 32 tokens, and is truncated to 16 tokens under the strong\-model tokenizer before insertion\. Patched continuations use a maximum generation length of 8192 tokens and are judged by final\-answer extraction followed bymath\-verify\. We retain an insertion position only when it contains at least one successful and one unsuccessful patch\. For the contrastive teacher, we set the correction strength toη=2\.0\\eta=2\.0, the log\-ratio clipping threshold toδ=4\.0\\delta=4\.0, and the Jensen–Shannon gating threshold toτjs=0\.002\\tau\_\{\\mathrm\{js\}\}=0\.002\. All evaluation runs use the same system prompt, conversation template, and context length across methods\.

#### Artifact licenses and intended use\.

We use publicly released models, datasets, and evaluation tools, including Qwen3\-4B\-Instruct\-2507, Gemma\-3\-4B\-IT, Skywork\-OR1\-RL\-Data, mathematical reasoning benchmarks, and math\-verify\. We use these artifacts only for research purposes and cite their original creators\. Our released code and processed data are intended for research use only, subject to the licenses and terms of the corresponding original artifacts\.