Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

# Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
Source: [https://arxiv.org/html/2607.15736](https://arxiv.org/html/2607.15736)
Leichao Dong1Dongxu Zhang1Yiding Sun1Qirui Wang1Yuhan Wang2Lin Chen1Jihua Zhu1

###### Abstract

Large reasoning models often solve problems through long chain\-of\-thought \(CoT\) traces, yet much of this computation is spent on redundant derivations, repeated self\-verification, and detours that do not improve the final answer\. Existing on\-policy self\-distillation methods reduce this cost by matching a student model to a concise copy of itself on prefixes sampled from the student’s own rollouts\. We show that this objective has an initialization bottleneck\. Since supervision is applied only to visited prefixes, training from a verbose base model places the KL loss on contexts that are often noisy, redundant, or already off track\. In such regions, a concise teacher can provide only local corrections, while the student continues to explore trajectories that an efficient reasoner should avoid\. In this paper, we proposeBIRD\(BootstrappedIterative Self\-ReasoningDistillation\), a two\-stage self\-reasoning distillation method that improves the rollout distribution before on\-policy training\. BIRD first samples concise solutions from the base model under a brevity instruction, keeps only answer\-correct traces, and performs a lightweight prompt\-switch SFT step\. The traces are generated with the brevity instruction but learned under the original task prompt, turning instruction\-induced conciseness into a default reasoning behavior\. Starting from this warm model, BIRD then applies on\-policy reverse\-KL distillation with a concise self\-teacher, now on cleaner and more informative prefixes\. Across Qwen3 series models, BIRD achieves a stronger accuracy\-efficiency trade\-off than prompting and cold\-start on\-policy distillation on MATH\-500 and AIME benchmarks\. On Qwen3\-8B, it improves MATH\-500 accuracy from 86\.2% to 92\.0% while reducing the average response length from 3,099 to 1,115 tokens\. These results highlight prefix support as a central factor in efficient reasoning distillation\.

Keywords:Large Language Models, Chain\-of\-Thought, On\-Policy Self\-Distillation

![Refer to caption](https://arxiv.org/html/2607.15736v1/x1.png)Figure 1:Prefix support determines where on\-policy KL supervision is applied\. Cold\-start OPSD and warm\-start OPSD use the same concise\-teacher construction, but they induce different student rollouts\. Starting from the base model exposes the KL loss to noisy or off\-track prefixes, where the next\-token target can be diffuse and less useful for compression\.BIRDfirst applies SFT bootstrapping and shifts the visited prefixes toward cleaner reasoning contexts\. The same prefix\-local KL objective is then evaluated on sharper teacher targets and reaches a higher compression frontier\.## 1Introduction

Chain\-of\-thought \(CoT\) reasoning has become a central capability of large language models, enabling complex problems to be solved through explicit intermediate steps\[wei2022chain,huang2026cfms\]\. Yet this capability also makes inference substantially more expensive\. Reasoning traces often span thousands of tokens and contain substantial redundancy, including repeated derivations, revisited conclusions, and excessive self\-verification\[peng2025revisiting,li2026making,chiang2024over,zhang2026llms,wei2025stop\]\. In many cases, this additional computation contributes little to the final answer, making reasoning length a major obstacle to efficient deployment\. Reasoning compression addresses this problem by shortening generated traces while preserving, and ideally improving, task accuracy\.

Recent work has explored several routes toward this goal\. Training\-free methods control the reasoning process at inference time through concise prompting, early stopping, or self\-correction\[renze2024benefits,yang2025dynamic,wu2026intern\]\. RL\-based methods introduce length\-sensitive rewards to discourage unnecessary tokens\[aggarwal2025l1,wan2026mitigating,li2026stepwise,li2026leash\]\. SFT\-based methods imitate compressed traces collected from static datasets, often using external models as trace generators\[xia2025tokenskip,zhang2026chain,ma2025cot,fan2026ctrlcot\]\. From a distribution\-matching perspective, these methods can be viewed as different ways of shifting the model toward a more concise reasoning distribution\[zhang2026llms,wei2025stop,zhang2026pointcot\]\. On\-policy self\-distillation \(OPSD\) provides a self\-contained alternative, in which the model itself acts as a dynamic concise teacher when conditioned on a conciseness instruction\[zhao2026self,sang2026crisp\]\. This removes the need for an external teacher during distillation and suggests that concise reasoning behavior is already latent in the base model\.

A key limitation, however, lies in the support on which OPSD applies its supervision\. The objective is prefix\-local: at each update, the student first samples a rollout, and the KL loss is evaluated only on prefixesy<ty\_\{<t\}visited by that rollout\[agarwal2024policy,zhang2026opsdl\]\. The concise teacher is therefore not queried over an abstract space of high\-quality solutions, but on the concrete partial trajectories produced by the current student\. When training starts from a verbose base model, these early trajectories are often noisy, redundant, and unreliable\[sui2025stop,zhao2026rosd\]\. Once a prefix already contains a detour, an unnecessary self\-check, or an incorrect intermediate step, the teacher must condition on that degraded context\. The resulting signal can correct the next token locally, but it remains anchored to trajectory regions that an efficient reasoner should avoid\.

This creates a cold\-start prefix\-support bottleneck\. Cold\-start OPSD and warm\-start OPSD may use the same concise\-teacher construction, but they expose the teacher to different prefix distributions\. As illustrated in Figure[1](https://arxiv.org/html/2607.15736#S0.F1), cold\-start training places the KL objective on prefixes inherited from the base model’s verbose rollouts\. A better initialization changes the student’s rollout support before online distillation begins, allowing the same KL objective to operate on shorter, more reliable, and more informative reasoning contexts\. The central issue is therefore not only how to construct a concise teacher, but also how to make the student visit prefixes where that teacher can provide actionable compression signal\.BIRD\(Bootstrapped Iterative Self\-Reasoning Distillation\) is designed around this observation\. The method keeps the concise self\-teacher unchanged and instead reshapes the rollout distribution on which the teacher is queried\. It first performs a lightweight LoRA\-based offline bootstrap on correctness\-filtered self\-generated concise traces\[hu2022lora\]\. These traces are sampled from the base model under a conciseness instruction, but learned under the original task prompt\. This prompt switch converts instruction\-induced conciseness into a default reasoning behavior rather than an inference\-time artifact\. Starting from this warm model, OPSD concentrates prefix\-local KL updates on cleaner reasoning contexts instead of noisy or off\-track prefixes produced by cold\-start rollouts\.

Experiments on Qwen3 series models\[yang2025qwen3\]and DeepSeek\-R1\-Distill\-Llama\[guo2025deepseek\]across MATH\-500, AIME 2024, and AIME 2025 show consistent improvements in the accuracy\-efficiency trade\-off\. Compared with prompting and cold\-start OPSD,BIRDproduces shorter traces while preserving or improving accuracy\. Training dynamics further show that cold\-start OPSD converges to a lower compression frontier and does not close the gap with longer optimization\. The gains become stronger with model scale and transfer across model families, highlighting prefix support as a central factor in on\-policy reasoning compression\. The contributions are summarized as follows:

- •To the best of our knowledge, this work is the first to identify the cold\-start prefix\-support bottleneck as a key failure mode in on\-policy reasoning compression, where prefix\-local KL supervision is applied to verbose, redundant, or off\-track student trajectories\.
- •We formulate a support\-placement principle for OPSD: the effectiveness of a concise self\-teacher depends not only on the target distribution it defines, but also on the rollout prefixes where this target is queried\.
- •We introduceBIRD, a bootstrapped iterative reasoning distillation method that uses correctness\-filtered concise traces to warm\-start the student, then performs OPSD on cleaner and more informative prefix support\.
- •Extensive experiments on Qwen3 and DeepSeek\-R1\-Distill\-Llama across MATH\-500, AIME 2024, and AIME 2025 show thatBIRDconsistently improves the accuracy\-efficiency frontier over prompting and cold\-start OPSD\.

## 2Related Work

### 2\.1RL with length penalties\.

RL\-based methods encourage compression by reshaping the reward landscape\. L1\[aggarwal2025l1\]uses GRPO to condition reasoning models on target lengths, enabling controllable reasoning depth\. DiPO\[wan2026mitigating\]derives difficulty signals from self\-generated reasoning statistics and uses them to modulate length penalties\. SwAP\[li2026stepwise\]applies step\-level adaptive penalization based on on\-policy log\-probability improvement, concentrating penalties on low\-importance reasoning steps\. Leash\[li2026leash\]adaptively adjusts length penalties so that large reasoning models can shorten outputs while preserving accuracy\. These methods directly optimize the accuracy\-efficiency trade\-off, but their effectiveness depends on reward design and often requires careful tuning of penalty strength to balance correctness and brevity\.

### 2\.2SFT on compressed chains\-of\-thought\.

SFT\-based methods train on compressed reasoning traces collected\[tang2026mitigatinghallucinationsinterlayerconsistency,tang2026seememitigatinghallucinationslarge,guo2026fademitigatinghallucinationsreducing,wang2026pointrft\]or constructed offline\[zhang2026pointcot,zhang2025not,zhang2026not\]\. TokenSkip\[xia2025tokenskip\]learns to identify and skip low\-importance reasoning tokens\. V\-Skip\[zhang2026chain\]uses a visual\-anchored information bottleneck for selective token pruning\. CoT\-Valve\[ma2025cot\]identifies a parameter\-space direction that controls reasoning length, and CtrlCoT\[fan2026ctrlcot\]combines semantic abstraction with logic\-preserving pruning\. The main advantage of this family is its simplicity, as the desired compressed behavior is explicitly specified by the training data\. Its limitation is potential distribution mismatch: the student is trained on static traces that may differ from the trajectories it later generates at inference time\.

### 2\.3On\-policy self\-distillation\.

OPSD\[zhao2026self\]conditions the teacher on ground\-truth answers and improves sample efficiency over RL\. TIP\[xu2026tip\]further studies on\-policy distillation from a token\-importance perspective, assigning greater weight to more informative tokens during training\. FiRe\-OPD\[li2026filter\]rethinks the granularity of OPD optimization by first filtering low\-quality trajectories and then softly reweighting tokens within the retained trajectories according to their learning value\. CRISP\[sang2026crisp\]shows that a conciseness instruction alone can serve as the teacher context, removing the need for ground\-truth answers during distillation\. However, existing OPSD methods mainly focus on teacher construction or token weighting, while paying less attention to the prefix support on which distillation is actually applied\.

## 3Method

### 3\.1Preliminaries and Distribution\-Matching View

A language modelπθ\\pi\_\{\\theta\}with parametersθ\\thetadefines a conditional distribution over token sequences\. Given an input promptxx, the model generates an output sequencey=\(y1,y2,…,yT\)y=\(y\_\{1\},y\_\{2\},\\dots,y\_\{T\}\)autoregressively\. At decoding steptt, the next\-token distribution is conditioned on both the original prompt and the previously generated prefixy<ty\_\{<t\}, which represents the current partial reasoning state\. The probability of the complete response factorizes as:

πθ​\(y∣x\)=∏t=1Tπθ​\(yt∣x,y<t\),\\pi\_\{\\theta\}\(y\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\),\(1\)wherey<t=\(y1,…,yt−1\)y\_\{<t\}=\(y\_\{1\},\\dots,y\_\{t\-1\}\)denotes the prefix generated up to positiont−1t\{\-\}1\. This prefix\-conditioned view is particularly important for reasoning models with extended generation traces\. Intermediate tokens are not merely surface realizations, but determine the context from which subsequent reasoning steps and the final answer are generated\.

In reasoning tasks,yytypically consists of a reasoning tracerrfollowed by a final answeraa\. Reasoning compression can therefore be viewed as modifyingπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)to favor shorter and more direct trajectories while preserving correctness\. We use distribution matching as a unifying formal lens\. Letπtarget\\pi\_\{\\mathrm\{target\}\}denote a concise target distribution induced by inference\-time controls, reward shaping, offline traces, or a dynamic self\-teacher\. Compression can be written as:

minθ⁡D​\(πθ,πtarget\),\\min\_\{\\theta\}\\;D\\bigl\(\\pi\_\{\\theta\},\\pi\_\{\\mathrm\{target\}\}\\bigr\),\(2\)where methods differ not only in howπtarget\\pi\_\{\\mathrm\{target\}\}is constructed, but also in where the matching loss is evaluated\.

![Refer to caption](https://arxiv.org/html/2607.15736v1/x2.png)Figure 2:Overview ofBIRD\. Stage 1 constructs a static target from one self\-generated concise trace per problem, filters it with ground\-truth verification, and applies prompt\-switch SFT\. The traces are generated with the conciseness instructionx\+cx\+cbut learned under the original promptxx\. This offline bootstrap moves the student from cold prefix support, which may contain detours and self\-checks, to warmer support that is more concise, direct, and on\-track\. Stage 2 then performs prefix\-local reverse\-KL self\-distillation from this warm start, using the concise teacher only as a dynamic target and no ground\-truth answers\.Algorithm 1BIRD: Bootstrapped Iterative Self\-Reasoning DistillationInput: Base modelπθ0\\pi\_\{\\theta\_\{0\}\}, data𝒟\\mathcal\{D\}, instructioncc

Output: Compressed modelπθ∗\\pi\_\{\\theta^\{\*\}\}

1:Stage 1: bootstrap concise prefix support

2:

𝒟filtered←∅\\mathcal\{D\}\_\{\\mathrm\{filtered\}\}\\leftarrow\\emptyset
3:foreach

\(xi,GTi\)∈𝒟\(x\_\{i\},\\mathrm\{GT\}\_\{i\}\)\\in\\mathcal\{D\}do

4:Sample

yi∼πθ0\(⋅∣xi,c\)y\_\{i\}\\sim\\pi\_\{\\theta\_\{0\}\}\(\\cdot\\mid x\_\{i\},c\)
5:Extract

a^i=answer​\(yi\)\\hat\{a\}\_\{i\}=\\mathrm\{answer\}\(y\_\{i\}\)
6:if

a^i=GTi\\hat\{a\}\_\{i\}=\\mathrm\{GT\}\_\{i\}then

7:

y~i←Truncate​\(yi,TSFT\)\\tilde\{y\}\_\{i\}\\leftarrow\\mathrm\{Truncate\}\(y\_\{i\},T\_\{\\mathrm\{SFT\}\}\)
8:

𝒟filtered←𝒟filtered∪\{\(xi,y~i\)\}\\mathcal\{D\}\_\{\\mathrm\{filtered\}\}\\leftarrow\\mathcal\{D\}\_\{\\mathrm\{filtered\}\}\\cup\\\{\(x\_\{i\},\\tilde\{y\}\_\{i\}\)\\\}
9:endif

10:endfor

11:

θSFT←LoRA​\-​SFT​\(θ0,𝒟filtered;\(xi\+c\)→xi\)\\theta\_\{\\mathrm\{SFT\}\}\\leftarrow\\mathrm\{LoRA\\text\{\-\}SFT\}\(\\theta\_\{0\},\\mathcal\{D\}\_\{\\mathrm\{filtered\}\};\\,\(x\_\{i\}\+c\)\\rightarrow x\_\{i\}\)
12:Set

θ←θSFT\\theta\\leftarrow\\theta\_\{\\mathrm\{SFT\}\}
13:Stage 2: warm\-start on\-policy self\-distillation

14:

θ¯←StopGrad​\(θ\)\\bar\{\\theta\}\\leftarrow\\mathrm\{StopGrad\}\(\\theta\)
15:fortraining step

k=1,2,…,Kk=1,2,\\dots,Kdo

16:if

kmodM=0k\\bmod M=0then

17:

θ¯←StopGrad​\(θ\)\\bar\{\\theta\}\\leftarrow\\mathrm\{StopGrad\}\(\\theta\)
18:endif

19:Sample a batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}
20:foreach

xi∈ℬx\_\{i\}\\in\\mathcal\{B\}do

21:Sample student rollout

yi∼πθ\(⋅∣xi\)y\_\{i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{i\}\)
22:

Ti←min⁡\(\|yi\|,TKL\)T\_\{i\}\\leftarrow\\min\(\|y\_\{i\}\|,T\_\{\\mathrm\{KL\}\}\)
23:

ℓi,t←DKL\(πθ\(⋅∣xi,yi,<t\)∥πθ¯\(⋅∣xi,c,yi,<t\)\)\\ell\_\{i,t\}\\leftarrow D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{i\},y\_\{i,<t\}\)\\,\\\|\\,\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x\_\{i\},c,y\_\{i,<t\}\)\\right\)
24:

ℒi←∑t=1Tiℓi,t\\mathcal\{L\}\_\{i\}\\leftarrow\\sum\_\{t=1\}^\{T\_\{i\}\}\\ell\_\{i,t\}
25:endfor

26:

ℒℬ←\|ℬ\|−1​∑xi∈ℬℒi\\mathcal\{L\}\_\{\\mathcal\{B\}\}\\leftarrow\|\\mathcal\{B\}\|^\{\-1\}\\sum\_\{x\_\{i\}\\in\\mathcal\{B\}\}\\mathcal\{L\}\_\{i\}
27:Update

θ\\thetawith

∇θℒℬ\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{B\}\}
28:endfor

29:

θ∗←θ\\theta^\{\*\}\\leftarrow\\thetareturn

πθ∗\\pi\_\{\\theta^\{\*\}\}

### 3\.2The Cold\-Start Problem in OPSD

OPSD trains on prefixes generated by the current student rather than on a fixed offline dataset\. At each training step, the student samples a rollout from the original task prompt, and the training prefixes are taken from this rollout\. The same prefixes are then evaluated by both the student and a teacher\. The teacher is a stop\-gradient copy of the model conditioned on an additional conciseness instruction\. It does not receive the ground\-truth answer or the student’s final answer as input, and provides next\-token distributions only at prefixes already produced by the student\. The loss compares the student and teacher distributions at each prefix, while gradients are applied only to the student\.

In this setup, the student policyπθ\\pi\_\{\\theta\}is initialized from the base modelπθ0\\pi\_\{\\theta\_\{0\}\}and optimized with a reverse\-KL objective\. Letqt=πθ\(⋅∣x,y<t\)q\_\{t\}=\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)andpt=πθ¯\(⋅∣x,c,y<t\)p\_\{t\}=\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x,c,y\_\{<t\}\), whereθ¯\\bar\{\\theta\}denotes the teacher parameters implemented as a periodically updated stop\-gradient copy ofθ\\theta\. The training objective is:

ℒ​\(θ\)=𝐄x∼𝒟,y∼πθ\(⋅∣x\)​\[∑tDKL​\(qt∥pt\)\]\.\\mathcal\{L\}\(\\theta\)=\\mathbf\{E\}\_\{x\\sim\\mathcal\{D\},\\;y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\Bigl\[\\sum\_\{t\}D\_\{\\mathrm\{KL\}\}\(q\_\{t\}\\\|p\_\{t\}\)\\Bigr\]\.\(3\)
Bothqtq\_\{t\}andptp\_\{t\}are defined at the same student\-sampled prefixy<ty\_\{<t\}, and the rolloutyyitself is sampled from the current policy\. Thus, the support on which KL supervision is applied is determined by the prefixes that the student already visits\. When training starts from the base model, many visited prefixes are verbose, redundant, or off\-track\. The KL objective then matches the concise teacher in trajectory regions that a compressed reasoner should ideally avoid\. The issue is therefore not simply how to construct a concise teacher, but how to place the on\-policy matching objective on informative prefix support\. Poor initialization can make this support inefficient for compression, slowing convergence and limiting the attainable compression frontier\.

### 3\.3BIRD: Two\-Stage Self\-Distillation

The key idea behindBIRD, summarized in Algorithm[1](https://arxiv.org/html/2607.15736#alg1)and Figure[2](https://arxiv.org/html/2607.15736#S3.F2), is to decompose distribution matching into an offline bootstrap followed by on\-policy refinement\. The offline stage first shifts the student’s rollout\-induced prefix support, while the online stage then applies the same prefix\-local OPSD objective on the resulting warm\-started support\. Thus, the two stages differ less in the form of the optimization objective than in the rollout distribution on which the matching signal is actually evaluated\.

The bootstrap stage uses the base model itself to construct a simple correctness\-filtered static target\. For each training problemxx, we sample one concise rollouty∼πθ0\(⋅∣x,c\)y\\sim\\pi\_\{\\theta\_\{0\}\}\(\\cdot\\mid x,c\)under the conciseness instructioncc, and retain the rollout only when its final answer matches the ground truth:

𝒟filtered=\{\(x,y\)∣answer​\(y\)=GT​\(x\)\}\.\\mathcal\{D\}\_\{\\text\{filtered\}\}=\\\{\(x,y\)\\mid\\mathrm\{answer\}\(y\)=\\mathrm\{GT\}\(x\)\\\}\.\(4\)
Ground truth is used only for this binary verification step and is never provided as part of the model input during training\. We then perform one epoch of LoRA SFT on𝒟filtered\\mathcal\{D\}\_\{\\text\{filtered\}\}by maximizing the likelihood of the retained concise traces under the original task prompt\. A central design choice is the prompt switch: traces are generated with the teacher promptx\+cx\+c, but SFT training uses the student promptxxwithout the conciseness instruction\. The resulting modelπθSFT\\pi\_\{\\theta\_\{\\mathrm\{SFT\}\}\}is encouraged to internalize instruction\-induced conciseness as a default reasoning behavior rather than relying on an inference\-time prompt\. We keep the bootstrap deliberately lightweight, using a single rollout per problem, a 2,048\-token cutoff, one training epoch, and no external teacher model\. These choices limit dependence on a fixed offline trace set while still moving the student toward concise and on\-track prefixes before on\-policy training begins\.

Starting fromπθSFT\\pi\_\{\\theta\_\{\\mathrm\{SFT\}\}\},BIRDperforms on\-policy reverse\-KL self\-distillation using Eq\.[3](https://arxiv.org/html/2607.15736#S3.E3), again with LoRA\. The teacher parametersθ¯\\bar\{\\theta\}are periodically refreshed from the student, and the KL loss is computed on the first 1,024 tokens of each rollout\. The objective is the same as in conventional OPSD\. The critical difference is the initialization\. Optimization begins fromπθSFT\\pi\_\{\\theta\_\{\\mathrm\{SFT\}\}\}rather thanπθ0\\pi\_\{\\theta\_\{0\}\}\. Since the warm\-started policy already visits prefixes from shorter and more reliable trajectories, the prefix\-local KL terms are evaluated on cleaner contexts\. This allows on\-policy optimization to refine useful reasoning behavior instead of spending much of its signal in verbose or off\-track regions inherited from the base model\.

Table 1:Main results across three model scales and three benchmarks\. TE = accuracy /ln\\ln\(response length\)\. MATH\-500:n=1n\{=\}1\(pass@1\); AIME:n=8n\{=\}8\(mean@8\)\. CRISP and BIRD values are taken at the checkpoint with the highest TE on each benchmark\.

## 4Experiments

### 4\.1Experimental Setup

#### Datasets\.

All training methods are trained on DAPO\-Math\-17k\-dedup\[yu2026dapo\]\. We evaluate on MATH\-500, AIME 2024, and AIME 2025\. MATH\-500 is evaluated with pass@1 \(n=1n=1\), whereas the AIME benchmarks are evaluated with mean@8 \(n=8n=8\)\[chen2021evaluating\]\.

#### Models and Training\.

We evaluate four model configurations spanning the Qwen3 and DeepSeek\-R1\-Distill\-Llama families, including Qwen3\-1\.7B, Qwen3\-4B, Qwen3\-8B\[yang2025qwen3\], and DeepSeek\-R1\-Distill\-Llama\-8B\[guo2025deepseek\]\. All training\-based methods use LoRA adaptation\. For fair comparison, CRISP andBIRDare trained for one epoch, with checkpoints saved every 25 steps\. All experiments are conducted using 8 NVIDIA A800 GPUs\.

#### Metrics and Baselines\.

During inference, we set the temperature to 0\.6, top\-ppto 0\.95, top\-kkto 20, and the maximum generation length to 8,192 tokens\. Our primary metric is Token Efficiency \(TE\), defined as:

TE=Accuracyln⁡\(Length\),\\mathrm\{TE\}=\\frac\{\\mathrm\{Accuracy\}\}\{\\ln\(\\mathrm\{Length\}\)\},\(5\)where Accuracy denotes the average fraction of correct answers across all sampled solutions, and Length denotes the average response length in tokens, excluding truncated outputs\. The logarithmic denominator captures diminishing returns from reducing output length as responses become shorter, thereby rewarding relative rather than absolute compression\. Unless otherwise specified, we report the checkpoint that achieves the highest TE on each benchmark\. The baselines areBase, the original model without any conciseness instruction,Concise, which applies the conciseness instruction only at inference time without additional training, andCRISP, a cold\-start on\-policy self\-distillation baseline corresponding to our reimplementation of CRISP\.

### 4\.2Main Results and Scaling

Table[1](https://arxiv.org/html/2607.15736#S3.T1)summarizes the performance ofBIRDand competing methods across three Qwen3 model scales and three benchmarks\. Overall,BIRDachieves the highest Token Efficiency \(TE\) in 8 out of 9 model\-benchmark pairs, and its advantage over cold\-start OPSD becomes more pronounced as model scale increases\. At 1\.7B, the gains are driven mainly by stronger compression rather than accuracy improvement\. For example, compared with CRISP on MATH\-500,BIRDreduces response length by 28% while incurring only a 1\.8\-point accuracy drop, yielding a net TE gain\. At 4B,BIRDstarts to improve accuracy and efficiency jointly\. The effect is strongest at 8B, whereBIRDconsistently achieves higher accuracy with shorter responses than CRISP across all three benchmarks\. For example, on AIME 2024, it improves accuracy by 5\.0 points while using 18% fewer tokens\.

This scaling trend suggests that larger models benefit more from the proposed two\-stage procedure\. A plausible explanation is that larger models follow the conciseness instruction more reliably during Stage 1, producing cleaner initial prefix support for OPSD, while their greater representational capacity leaves more room for Stage 2 to compress reasoning traces without sacrificing correctness\. Notably, applying a conciseness instruction only at inference time, without any additional training, already improves over the original base model, suggesting that concise reasoning behavior is latent in pretrained reasoning models\.BIRDturns this instruction\-induced behavior into a persistent default and achieves substantially stronger compression than prompting alone\.

Table 2:Cross\-model generalization on DeepSeek\-R1\-Distill\-Llama\-8B, a Llama\-based reasoning model whose base responses are already relatively compact\.BIRDretains its accuracy\-efficiency advantage beyond the Qwen3 family\.![Refer to caption](https://arxiv.org/html/2607.15736v1/x3.png)Figure 3:Training dynamics of CRISP andBIRDon Qwen3\-8B evaluated on MATH\-500\. Top: average response length\. Bottom: Token Efficiency \(TE\)\.BIRDstarts from the SFT\-warmed checkpoint before OPSD updates and maintains lower response length and higher TE throughout training\.
### 4\.3Cross\-Model Generalization

To examine whetherBIRDgeneralizes beyond the Qwen family, we apply the same training recipe to DeepSeek\-R1\-Distill\-Llama\-8B, whose base responses are already much shorter than those of Qwen3, leaving less apparent redundancy to remove\. Table[2](https://arxiv.org/html/2607.15736#S4.T2)summarizes the results\. Despite this compact starting point,BIRDimproves MATH\-500 accuracy by 8\.0 percentage points while simultaneously reducing response length by 33%\. It also improves the accuracy\-efficiency trade\-off on both AIME benchmarks\. CRISP improves TE over the base model, indicating that cold\-start OPSD can still remove some redundancy\. However, its gains are smaller than those ofBIRD\. In particular, on MATH\-500, CRISP achieves compression at the cost of a slight accuracy drop, whereasBIRDimproves both accuracy and TE\. These results suggest that SFT\-based prefix\-support warm starting is not specific to Qwen3 and can transfer to a reasoning model whose initial outputs are already relatively compact\.

### 4\.4Training Dynamics

Figure[3](https://arxiv.org/html/2607.15736#S4.F3)shows the evolution of response length and Token Efficiency \(TE\) during OPSD training for Qwen3\-8B on MATH\-500, which we use because it is the largest benchmark and has the lowest evaluation variance\. Before any OPSD update,BIRDalready starts from a more favorable prefix distribution\. The SFT\-warmed model reduces the average response length from 3,099 to 2,057 tokens, corresponding to a 34% reduction relative to the base model\. On this benchmark, its initial TE already matches the peak TE of CRISP, indicating that even lightweight SFT moves the student away from many verbose prefixes on relatively simple problems\. This advantage persists throughout training\. After only 25 optimization steps,BIRDreaches a TE of 11\.92, already exceeding the best value attained by CRISP\. In other words, the performance that CRISP reaches after nearly a full epoch is achieved byBIRDalmost immediately\. Although both methods plateau before the end of training, they converge to markedly different levels\.BIRDstabilizes at a TE of 13\.02–13\.11, whereas CRISP remains in the range of 11\.46–11\.71\. The resulting gap of approximately 1\.4 TE points persists after convergence, suggesting that prolonged cold\-start OPSD alone does not close the initialization\-induced gap\.

![Refer to caption](https://arxiv.org/html/2607.15736v1/x4.png)Figure 4:Qualitative trace analysis on a representative MATH\-500 counting problem using Qwen3\-1\.7B\. Blue segments denote useful reasoning steps, while red segments denote self\-doubt or detours\. The Base model produces a long trace with many self\-checks and ends incorrectly\. CRISP removes some redundancy but retains enough hedging to remain off\-track\.BIRDfollows a direct combinatorial path, reduces self\-checks from88to11, and reaches the correct answer\.
### 4\.5Qualitative Examples

Figure[4](https://arxiv.org/html/2607.15736#S4.F4)presents reasoning traces from the Base model, CRISP, andBIRDon a representative MATH\-500 problem using Qwen3\-1\.7B\. The task requires choosing 4 of 5 upper\-class soldiers and 8 of 10 lower\-class soldiers, whose direct solution is\(54\)​\(108\)=225\\binom\{5\}\{4\}\\binom\{10\}\{8\}=225\. The Base model generates 20 reasoning steps, including 8 redundant self\-doubt interruptions, and ultimately produces the incorrect answer 1\. CRISP shortens the trace to 13 steps and reduces such interruptions from 8 to 4, but the remaining hedging still derails the solution\. In contrast,BIRDrequires only 9 steps, contains a single brief self\-check, and directly follows the combinatorial counting path to the correct answer\. This example shows that excessive self\-verification is not only token\-inefficient, but can also perturb reasoning by reopening settled subproblems\.

## 5Ablation Studies

#### Role of each stage\.

Table[3](https://arxiv.org/html/2607.15736#S5.T3)compares SFT\-only, cold\-start OPSD \(CRISP\), and the fullBIRDpipeline on Qwen3\-8B\. SFT\-only already gives a competitive TE, showing that the bootstrap stage can move the model toward shorter and more useful rollouts before any on\-policy updates\. CRISP also improves over the base model, but it remains below the full pipeline on all three benchmarks\. The best results come from applying OPSD after the SFT warm start, suggesting that the two stages are most effective when used together\.

Table 3:Stage necessity ablation on Qwen3\-8B\. We compare the SFT bootstrap alone, cold\-start OPSD \(CRISP\), and the fullBIRDpipeline\. The full SFT→\\toOPSD procedure obtains the highest accuracy and TE, as well as the shortest responses\.
#### Importance of stage order\.

Table[4](https://arxiv.org/html/2607.15736#S5.T4)compares the proposed ordering \(SFT→\\rightarrowOPSD\) with the reversed ordering \(OPSD→\\rightarrowSFT\) on Qwen3\-4B\. We first train an OPSD model from the base model, select the checkpoint with the highest average TE across benchmarks, generate concise trajectories from this checkpoint, and then perform SFT\. Although the reversed ordering produces shorter responses, it leads to substantial accuracy degradation, particularly on more challenging benchmarks\. For example, accuracy on AIME 2025 drops from 40\.83% to 28\.33%\. This observation suggests that applying SFT after OPSD may partially overwrite the compression policy learned through on\-policy optimization with a fixed offline target, whereas the proposed ordering preserves the adaptability introduced during the second stage\.

Table 4:Stage order ablation on Qwen3\-4B\. We compare applying SFT before OPSD with applying OPSD before SFT\. The reversed order produces shorter responses, while the proposed SFT→\\toOPSD order achieves higher accuracy\.

## 6Conclusion

We presentedBIRD, a two\-stage distribution\-matching method for compressing reasoning traces\. The central observation is that on\-policy self\-distillation is governed not only by the concise teacher, but also by the prefix support on which the teacher is queried\. When training starts from a verbose base model, prefix\-local KL supervision can be applied to redundant or off\-track trajectories, limiting the usefulness of the distillation signal\.BIRDaddresses this bottleneck by first using a lightweight correctness\-filtered SFT bootstrap to shift the student toward shorter and more reliable rollouts, and then applying standard on\-policy self\-distillation from this warmer initialization\. Across model scales, model families, and math reasoning benchmarks,BIRDimproves the accuracy\-efficiency trade\-off over prompting and cold\-start OPSD\. These results suggest that offline bootstrapping and on\-policy refinement provide complementary benefits, and that the placement of supervision over rollout prefixes is a key design factor for efficient reasoning distillation\.

## References