Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation

# Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation
Source: [https://arxiv.org/html/2607.10805](https://arxiv.org/html/2607.10805)
Keqin Peng1, Chen Li1, Yuanxin Ouyang1,Yancheng Yuan2∗,Liang Ding3 1Beihang University2Hong Kong Polytechnic University3Alibaba Group keqin\.peng@buaa\.edu\.cnliangding\.liam@gmail\.com

###### Abstract

On\-Policy Self\-Distillation \(OPSD\) has emerged as a crucial paradigm for enhancing and aligning Large Language Models \(LLMs\)\. However, in complex reasoning tasks, OPSD paradoxically degrades downstream performance\. In this paper, we systematically investigate this pathology and identify a severe optimization trap we define asThinking Collapse—a sharp decline in the model’s native intermediate reasoning behavior, measured by epistemic\-token density \(ET per 1k\)\. Through entropy\-based gradient masking and token\-level target analysis, we show that this collapse is triggered by aggressive teacher gradients at high\-student\-entropy decision forks, where student epistemic tokens are frequently suppressed into teacher non\-epistemic targets and are highly concentrated in high pointwise student\-teacher divergence regions\. To resolve this optimization pathology, we proposeAdaptive Dual\-Perspective OPSD \(AD\-OPSD\), a robust control framework that dynamically moderates the self\-distillation objective\. AD\-OPSD selectively anchors high\-suppression\-risk sandboxed tokens to a reference prior derived from the frozen base model via an asymmetrical pointwise divergence gate, preserving native thinking capacity while retaining OPSD’s error\-correcting power\. Extensive experiments across competitive mathematical benchmarks show that AD\-OPSD improves over standard OPSD by up to\+4\.1%absolute average accuracy across diverse model scales and datasets\. Further analysis demonstrates that AD\-OPSD mitigates thinking collapse and generalizes robustly to different post\-training paradigms\.

Diagnosing and Mitigating Thinking Collapse in On\-Policy Self\-Distillation

Keqin Peng1, Chen Li1, Yuanxin Ouyang1, Yancheng Yuan2∗, Liang Ding3††thanks:Corresponding Authors\.1Beihang University2Hong Kong Polytechnic University3Alibaba Groupkeqin\.peng@buaa\.edu\.cnliangding\.liam@gmail\.com

## 1Introduction

Recently, on\-policy self\-distillation \(OPSD\) has emerged as a crucial paradigm for enhancing and aligning Large Language Models\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib6); Shenfeldet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib12); Hübotteret al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib13)\)\. By distilling ground\-truth\-conditioned, on\-policy target distributions of a teacher model into the student policy over its own sampled rollouts, OPSD avoids the exposure bias of off\-policy imitation and the sparse\-reward bottleneck of reinforcement learning\. This dense, token\-level supervision has yielded remarkable success in general alignment and preference\-tuning tasks\.

However, when applied to complex reasoning tasks, OPSD paradoxically degrades downstream performance\(Kimet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib1); Kauret al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib11)\)\. Prior studies attribute this degradation to the suppression of epistemic verbalizations\(Kimet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib1)\)or related “fork suppression” at high\-entropy decision points\(Kauret al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib11)\)\. Yet, these works primarily document empirical symptoms; the underlying mechanics ofwhereandwhythis suppression occurs remain unclear, leaving how to resolve this optimization trap without losing OPSD’s error\-correcting power an open challenge\.

In this paper, we systematically investigate this pathology, identifying a severe optimization trap we define asThinking Collapse—a precipitous decline in the model’s native intermediate reasoning steps, measured as epistemic tokens per 1K generated tokens \(ET per 1k\)\. To understand its mechanics, inspired by the findings ofWanget al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib3)\)andXuet al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib14)\)that reasoning updates and critical decision forks are heavily concentrated on small sets of high\-entropy tokens, we propose an entropy\-based gradient masking diagnostic experiment to locate the spatial boundary of thinking collapse\. We find that masking gradients on high\-student\-entropy tokens substantially recovers thinking density, but also exposes a correction trade\-off\. Specifically, masking the self\-distillation gradients of only the top 20% student entropy tokens recovers the model’s native thinking density from OPSD’s collapsed7\.9to9\.8 ET per 1k, while still failing to recover downstream accuracy\.

Furthermore, we conduct a microscopic token\-level discrepancy transition analysis at initialization, discovering a substantial top\-1 target discrepancy where the student’s top\-1 predicted epistemic token is forcefully suppressed by a teacher non\-epistemic token\. Lastly, mapping the student’s epistemic space under active suppression \(log⁡Ps\>log⁡Pt\\log P\_\{s\}\>\\log P\_\{t\}\) reveals that native thinking steps are densely concentrated in high pointwise divergence regions, supporting the use of pointwise student\-teacher divergence as a robust indicator of suppression risk\.

To resolve this optimization pathology, we proposeAdaptive Dual\-Perspective OPSD \(AD\-OPSD\), a robust control framework that dynamically moderates the self\-distillation objective\. Instead of binary token masking, AD\-OPSD employs a continuous, asymmetrical soft\-gating mechanism\. Within a localized high\-entropy sandbox, it selectively anchors high\-suppression\-risk tokens to a reference prior derived from the frozen base model using a sigmoid pointwise KL gate\. This dynamically shields the model’s native exploratory reasoning paths under active suppression while allowing standard teacher corrective gradients on factual tokens to proceed at full strength\.

Extensive experiments across competitive benchmarks demonstrate that AD\-OPSD consistently recovers thinking density and delivers up to a\+4\.1%absolute average accuracy boost over standard OPSD across diverse model scales and datasets\. Further discussions confirm that AD\-OPSD mitigates thinking collapse and generalizes robustly across different post\-training alignment configurations, such as the student Non\-Think setup\.

In summary, ourcontributionsare threefold:

- •We diagnoseThinking Collapsein reasoning OPSD through a multi\-scale evidence chain: entropy\-based gradient masking localizes the failure to high\-student\-entropy decision forks; token\-level target analysis reveals severe Student\-ET→\\toTeacher\-Non\-ET discrepancies at initialization; and pointwise divergence mapping identifies high\-risk suppression regions for real\-time intervention\.
- •We proposeAD\-OPSD, a student\-side, token\-wise post\-training framework that dynamically anchors high\-suppression\-risk tokens to a frozen base\-model reference prior via an asymmetrical pointwise divergence gate, preserving native exploratory reasoning while retaining teacher corrective supervision\.
- •Extensive evaluations across competitive mathematical benchmarks and diverse model families show that AD\-OPSD mitigates thinking collapse, recovers thinking density, and improves over standard OPSD by up to a\+4\.1%\+4\.1\\%absolute average accuracy gain, with robust generalization to non\-thinking alignment configurations\.

## 2Thinking Collapse in OPSD

While On\-Policy Self\-Distillation \(OPSD\) has proven highly effective for standard alignment tasks, its application to reasoning models reveals a paradoxical performance degradation\. Prior studies\(Kimet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib1); Kauret al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib11)\)trace the degradation to the suppression of epistemic verbalizations or fork tokens\. Inspired by these pioneering findings, we introduce the quantitative metric ofthinking densityto represent the model’s exploratorythinking intensityand investigate how this behavior changes across different self\-distillation configurations\.

Specifically, while prior literature often measures the absolute count of epistemic tokens \(ET\) within a generated rollout \(E​\(y\)=∑tcount​\(t,y\)E\(y\)=\\sum\_\{t\}\\text\{count\}\(t,y\)\), this raw count is highly sensitive to variations in sequence length\. To establish a robust, length\-invariant proxy for the model’s underlying thinking intensity, we formally definethinking densityas the frequency of epistemic tokens per 1,000 generated tokens:

Thinking Density=∑t∈𝒯count​\(t,y\)L​\(y\)×1000\\text\{Thinking Density\}=\\frac\{\\sum\_\{t\\in\\mathcal\{T\}\}\\text\{count\}\(t,y\)\}\{L\(y\)\}\\times 1000\(1\)whereL​\(y\)L\(y\)is the total sequence length, and𝒯\\mathcal\{T\}represents the set of 10 epistemic markers acting as indicators of planning, uncertainty externalization, and backtracking\(Kimet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib1)\):

𝒯=\{\\displaystyle\\mathcal\{T\}=\\\{wait,hmm,perhaps,maybe,actually,\\displaystyle\\text\{wait\},\\text\{hmm\},\\text\{perhaps\},\\text\{maybe\},\\text\{actually\},\(2\)alternatively,seems,might,likely,check\}\\displaystyle\\text\{alternatively\},\\text\{seems\},\\text\{might\},\\text\{likely\},\\text\{check\}\\\}To empirically verify the impact of self\-distillation on thinking density, we train Qwen3\-1\.7B across distinct OPSD configurations on the AIME25 benchmark, including Think Student / Think Teacher \(T/T\) and Non\-Think Student / Think Teacher \(NT/T\) setups\. The empirical results, illustrated in Figure[1](https://arxiv.org/html/2607.10805#S2.F1), reveal a severe optimization pathology\. While standard OPSD \(T/T\) severely suppresses the student’s thinking density—collapsing it precipitously from the base baseline of 10\.3 down to7\.9 ET per 1k—it simultaneously causes a downstream accuracy decay from 38\.1% to33\.9%\(a \-4\.2% degradation\)\. Conversely, disabling student thinking during training \(NT/T\) preserves a healthy thinking density of11\.1during evaluation, allowing accuracy to climb to42\.2%\(\+4\.1% absolute boost over Base\)\. This tight correlation strongly suggests that OPSD’s reasoning degradation is closely linked to the pathology ofThinking Collapse, which suppresses the model’s exploratory, intermediate reasoning steps\.

![Refer to caption](https://arxiv.org/html/2607.10805v1/x1.png)Figure 1:AIME25 Accuracy vs\. Thinking Density \(ET per 1k\) Across OPSD Distillation Modes on Qwen3\-1\.7B\. On\-policy self\-distillation under the student thinking mode \(T/T\) severely impairs reasoning performance, which is highly correlated with the sharp collapse of its thinking density\.Although NT/T achieves the strongest accuracy in Figure[1](https://arxiv.org/html/2607.10805#S2.F1), it introduces a cross\-mode distillation factor: a non\-thinking student learns from a thinking teacher\. This makes it difficult to determine whether the observed gain comes from OPSD itself or from mode\-level transfer\. In contrast, T/T keeps the student and teacher in the same thinking mode, thereby providing a cleaner setting to diagnose how OPSD directly affects the model’s own reasoning trajectories\. We therefore focus our subsequent diagnostic analysis on T/T, as it reveals the core failure mode in which OPSD suppresses the model’s intrinsic exploratory and backtracking behavior\.

## 3Diagnosing the Causes of Thinking Collapse

While Section[2](https://arxiv.org/html/2607.10805#S2)establishes the empirical existence of Thinking Collapse in reasoning OPSD, the underlying mechanics ofwhereandwhythis suppression occurs remain unclear\. To unpack the root causes of this pathology, we conduct a systematic, multi\-scale diagnostic investigation across three analytical phases:

- •Entropy\-Based Gradient Masking \(Section[3\.1](https://arxiv.org/html/2607.10805#S3.SS1)\):We perform targeted token masking based on student entropy during backpropagation to spatially isolate and localize the main region of thinking collapse\.
- •Token\-Level Discrepancy Analysis \(Section[3\.2](https://arxiv.org/html/2607.10805#S3.SS2)\):We analyze static target distributions at initialization to investigate the microscopic target discrepancies over epistemic tokens at critical decision forks\.
- •Pointwise Divergence Mapping \(Section[3\.3](https://arxiv.org/html/2607.10805#S3.SS3)\):We investigate the statistical properties of pointwise KL divergence to examine its correlation with thinking collapse and validate its potential as a dynamic guidance indicator\.

This diagnostic roadmap directly exposes the underlying optimization pathology of Thinking Collapse, providing the empirical foundation for our proposed Adaptive Dual\-Perspective OPSD \(AD\-OPSD\) framework\.

![Refer to caption](https://arxiv.org/html/2607.10805v1/x2.png)Figure 2:Consolidated OPSD Diagnosticsunder Token\-Level Masking and Microscopic Target Discrepancy\. Subfigures \(a\) and \(b\) illustrate OPSD performance under entropy\-based masking \(LHS: Thinking Density, RHS: Accuracy %\)\. Subfigure \(c\) depicts the cumulative student ET count \(LHS, blue curve\) and cumulative ET\-to\-Non\-ET target discrepancy transitions \(RHS, orange curve\) across student entropy percentiles sorted in descending order at initialization \(step 0\)\. Both masking plots include Base and standard OPSD \(Unmasked\) as reference baselines\.### 3\.1Localizing Thinking Collapse via Entropy\-Based Gradient Masking

To investigate where and how thinking collapse occurs in the generation sequence, we analyze how OPSD performance and thinking density respond to targeted token masking\. In reasoning models, only a small fraction of tokens exhibit high entropy, acting as critical decision forks that steer the model toward diverse reasoning pathways\(Wanget al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib3)\)\. Inspired by this, we study OPSD dynamics from the perspective of student entropyH​\(yi∣y<i\)H\(y\_\{i\}\\mid y\_\{<i\}\)\. Using the standard OPSD training settings\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib6)\), we partition the student’s predictive token space dynamically during backpropagation and selectively mask out the self\-distillation gradients under two polar configurations: \(1\)Top\-p%p\\%Student Entropy \(SE\) Mask\(wherep∈\{20,30,50\}p\\in\\\{20,30,50\\\}\), which shields the student’s most uncertain decision points, and \(2\)Bottom\-50%50\\%Student Entropy \(SE\) Mask, which restricts distillation exclusively to these high\-uncertainty regions\. The experimental results, illustrated in Figures[2](https://arxiv.org/html/2607.10805#S3.F2)\(a\) and[2](https://arxiv.org/html/2607.10805#S3.F2)\(b\), provide a systematic, multi\-scale localization of thinking collapse, while exposing a critical learning trade\-off\.

#### High\-Entropy Supervision as the Primary Source of Collapse\.

The macro\-level contrast of high\-entropy versus low\-entropy masking in Figure[2](https://arxiv.org/html/2607.10805#S3.F2)\(a\) highlights the primary driver of this pathology\. When applying the Bottom\-50% SE mask \(distilling exclusively on high\-entropy tokens\), the evaluation thinking density collapses heavily to7\.6 ET per 1k—even lower than the standard unmasked OPSD baseline of 7\.9\. Conversely, completely shielding the high\-entropy region via the Top\-50% SE mask fully restores thinking density to10\.7 ET per 1k, exceeding the pre\-trained base baseline \(10\.3\)\. This stark contrast strongly indicates that thinking collapse is primarily driven by the teacher’s aggressive supervision over the student’s highest\-entropy states\.

#### Suppressive Pressure Concentrated at Critical Decision Forks\.

A closer examination of the high\-entropy region’s progression—increasing the masked fraction from Top 20% to Top 50% in Figure[2](https://arxiv.org/html/2607.10805#S3.F2)\(b\)—reveals the localized concentration of this suppressive force\. Shielding only the Top\-20% SE tokens is highly sufficient to recover thinking density to9\.8 ET per 1k, representing a massive rebound from standard OPSD \(7\.9\)\. However, further widening the mask to Top\-30% and Top\-50% yields heavily diminishing returns, restoring density to10\.2and10\.7respectively\. This decelerating growth curve demonstrates that suppressive pressure is highly concentrated within the very tip of the highest\-entropy states \(the top 20% critical decision forks\)\. Crucially, as we gradually recover more of the model’s native thinking density \(from 9\.8 to 10\.7 ET per 1k\), downstream accuracy exhibits a corresponding climb from31\.4%to33\.9%, indicating a clear positive association between preserving native intermediate thinking steps and downstream performance\.

#### Optimization Deadlock of Binary Gradient Masking\.

Despite these large differences in thinking density, a striking aspect of the results is that standard OPSD \(33\.9%\), Mask Top\-50% \(33\.9%\), and Mask Bottom\-50% \(31\.9%\) achieve relatively similar and degraded downstream accuracies\. This performance deadlock arises from two distinct and opposite failure modes\. While standard OPSD receives the teacher’s full corrective signals but suffers from severe thinking collapse \(Thinking Density = 7\.9\), the Mask Top\-50% setup preserves the model’s native thinking capacity \(Thinking Density = 10\.7\) but is deprived of the teacher’s vital corrective feedback at critical decision forks\. Meanwhile, the Mask Bottom\-50% configuration suffers from collapsed thinking density \(7\.6\) while also being deprived of teacher gradients on low\-entropy factual tokens\. This detailed diagnosis suggests that simple binary gradient masking is caught in an optimization deadlock: we cannot resolve thinking collapse without sacrificing teacher correction, and vice\-versa, highlighting the need for a non\-binary, adaptive soft\-gating formulation to balance thinking preservation and factual correction\.

### 3\.2Dissecting Thinking Collapse via Microscopic Transition Analysis

To gain a deep, token\-level understanding of how thinking collapse is triggered under standard OPSD, we conduct a microscopic transition analysis\. Our diagnostic is astatic target discrepancy analysisconducted at the very beginning of training \(using the initial checkpoints before any gradient updates\), which isolates the raw, unmodified supervisory signals that dictate the initial gradient directions\. We randomly sample 200 tasks from the OpenThought dataset and generate 4 independent rollouts under Qwen3\-1\.7B’s thinking mode, yielding a total of 800 rollouts \(3\.9M tokens\) scored by a ground\-truth\-conditioned teacher model at initialization\.

The initial static distributions, illustrated in Figure[2](https://arxiv.org/html/2607.10805#S3.F2)\(c\), reveal two crucial findings explaining the root cause of thinking collapse:

- •High\-Entropy Concentration of Epistemic Tokens:The student model’s native epistemic tokens \(ETs\) are heavily concentrated within high student entropy regions\. Specifically, the top 10% highest\-entropy tokens contain17,021ETs \(34\.03%of all generated ETs\), and the top 50% highest\-entropy tokens contain47,687ETs \(95\.35%of all ETs\)\. This suggests that the model’s native reasoning and intermediate thinking steps are structurally localized at high\-entropy decision "forks" where the student actively explores\.
- •Severe Top\-1 Discrepancy \(Student ET→\\toTeacher Non\-ET\):Within these identical high\-entropy regions, there is a substantial target discrepancy where the student’s top\-1 predicted token is an ET, but the teacher’s corresponding top\-1 predicted token is a Non\-ET\. Out of 9,807 total ET\-to\-Non\-ET discrepancy events across the entire 3\.9M token corpus,6,501 \(66\.29%\)are concentrated in the top 10% highest\-entropy interval, and9,803 \(99\.96%\)occur within the top 50% highest\-entropy region\.

These initial static discrepancies generate strong negative gradients on the student’s exploratory paths at step 0\. This optimization force aggressively suppresses the student’s ET probabilities, explaining why standard OPSD rapidly triggers thinking collapse\.

### 3\.3Mapping the Epistemic Space of Thinking Collapse via Pointwise Divergence

To resolve the thinking collapse trap, we need a robust, real\-time guidance indicator to identify where intermediate thinking steps are under high risk of being suppressed during on\-policy self\-distillation\. On exploratory tokens \(e\.g\., intermediate verification steps like “Wait”\), the student’s probability is typically much larger than the teacher’s, yielding a substantial log probability ratio of\(log⁡Ps​\(yi\)−log⁡Pt​\(yi\)\)≫0\(\\log P\_\{s\}\(y\_\{i\}\)\-\\log P\_\{t\}\(y\_\{i\}\)\)\\gg 0\(Shenet al\.,[2026a](https://arxiv.org/html/2607.10805#bib.bib8); Kimet al\.,[2026a](https://arxiv.org/html/2607.10805#bib.bib9); Yanget al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib26)\)\. To prevent false positives where the student probability is near\-zero but the teacher’s is even smaller, we weight this log ratio by the student’s probability, resulting in the pointwise KL divergence contribution:

Ci=Ps​\(yi∣y<i\)​\(log⁡Ps​\(yi∣y<i\)−log⁡Pt​\(yi∣y<i\)\)C\_\{i\}=P\_\{s\}\(y\_\{i\}\\mid y\_\{<i\}\)\\left\(\\log P\_\{s\}\(y\_\{i\}\\mid y\_\{<i\}\)\-\\log P\_\{t\}\(y\_\{i\}\\mid y\_\{<i\}\)\\right\)\(3\)whereyiy\_\{i\}is the actually sampled token in the rollout sequence\.

![Refer to caption](https://arxiv.org/html/2607.10805v1/x3.png)Figure 3:Distribution and Concentration of Epistemic Tokensacross Pointwise KL Percentiles within High\-Entropy Regions \(Top 20% Student Entropy\) under Active Suppression \(Ps\>PtP\_\{s\}\>P\_\{t\}\)\. Pointwise KL serves as a highly robust guidance indicator of suppression risk, allowing AD\-OPSD to protect vital thinking steps by shielding regions of high pointwise divergence\.To validate whether the pointwise KL divergence contributionCiC\_\{i\}can serve as a robust proxy for locating these high\-risk suppression regions, we focus exclusively on the active suppression region \(Ps\>PtP\_\{s\}\>P\_\{t\}, i\.e\.,log⁡Ps−log⁡Pt\>0\\log P\_\{s\}\-\\log P\_\{t\}\>0\) within the top 20% student entropy\. We sort these tokens by their pointwise KLCiC\_\{i\}in descending order and partition them into standard 10% interval slices from 10% to 100%\. The resulting local ET density and cumulative ET coverage are illustrated in Figure[3](https://arxiv.org/html/2607.10805#S3.F3)\.

The empirical distributions reveal that epistemic tokens \(ETs\) are exceptionally densely concentrated within regions of high pointwise divergence, which mathematically validatesCiC\_\{i\}as a powerful real\-time indicator of suppression risk:

- •Dense ET Concentration in High Divergence Intervals:As shown in Figure[3](https://arxiv.org/html/2607.10805#S3.F3), the local ET density is monotonically and sharply higher in the top pointwise KL intervals, representing a heavily suppressed thinking zone\. Specifically, the local ET density peaks at7\.9%in the top 10% highest interval, and steadily decreases to6\.9%\(10%–20% interval\),6\.1%\(20%–30% interval\), and down to under3\.5%at the lowest intervals\. This stark contrast demonstrates that high pointwise divergence regions naturally cluster a high density of epistemic tokens\.
- •High\-Recall Cumulative Coverage:Concurrently, the cumulative coverage curve exhibits a steep, near\-linear rise at the beginning\. Specifically, the top 10% pointwise KL interval alone captures15\.6%of all generated ETs within this active suppression zone, the top 20% highest intervals cover29\.2%, and the top 50% highest intervals capture a substantial61\.0%of the total epistemic tokens, before eventually reaching100\.0%at the 100% percentile\.

This quantitative behavior provides a strong design justification for our adaptive framework as highlighted in our abstract and introduction\. Since epistemic tokens are densely concentrated within high pointwise divergence regions under active suppression, local pointwise KL can serve as an exceptionally robust real\-time indicator of suppression risk\. By modulating the distillation objective using a soft\-gating mechanism triggered by this pointwise divergence, we can dynamically assign larger weights to the reference prior in high suppression\-risk regions\. This allows us to successfully protect the model’s native thinking steps while retaining OPSD’s error\-correcting power in other regions\. This directly guides our AD\-OPSD framework detailed in Section[4](https://arxiv.org/html/2607.10805#S4)\.

## 4Adaptive Dual\-Perspective On\-Policy Self\-Distillation

To mitigate the thinking collapse, we proposeAdaptive Dual\-Perspective On\-Policy Self\-Distillation \(AD\-OPSD\), a robust control framework that dynamically moderates the self\-distillation objective\. Instead of binary token masking, which deprives the model of corrective teacher signals, AD\-OPSD treats the pointwise divergence as a continuous indicator of thinking suppression risk\. It dynamically moderates the teacher’s influence and shifts trust to the frozen base model, preserving native reasoning capacity without losing OPSD’s error\-correcting power\. AD\-OPSD consists of two core components: \(1\) a dual\-perspective target formulation, and \(2\) an adaptive gating mechanism\.

### 4\.1Dual\-Perspective Target Formulation

In standard OPSD, the student policyπθ​\(yi∣y<i,X\)\\pi\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\},X\)is optimized directly toward the ground\-truth\-conditioned teacher distributionπt​\(yi∣y<i,X,GT\)\\pi\_\{t\}\(y\_\{i\}\\mid y\_\{<i\},X,\\text\{GT\}\)via Forward Kullback\-Leibler \(KL\) divergence\. To protect the model’s exploratory reasoning capacity from aggressive teacher gradients, we reformulate the target distribution for each tokenyiy\_\{i\}into a dynamically interpolated targetπi∗\\pi^\{\*\}\_\{i\}:

πi∗​\(v\)=\\displaystyle\\pi^\{\*\}\_\{i\}\(v\)=\{\}Ui​πbase​\(v∣y<i,X\)\\displaystyle U\_\{i\}\\pi\_\{\\text\{base\}\}\(v\\mid y\_\{<i\},X\)\(4\)\+\(1−Ui\)​πt​\(v∣y<i,X,GT\)\\displaystyle\+\(1\-U\_\{i\}\)\\pi\_\{t\}\(v\\mid y\_\{<i\},X,\\text\{GT\}\)whereπbase\\pi\_\{\\text\{base\}\}is the frozen base model acting as anepistemic anchor, andUi∈\[0,1\]U\_\{i\}\\in\[0,1\]represents theTeacher Unreliability Index\. Scoring student rollouts with the frozen base modelπbase\\pi\_\{\\text\{base\}\}anchors the target to the model’s unsuppressed native reasoning prior, preventing policy degradation\.

### 4\.2Adaptive Gating Mechanism

The unreliability indexUiU\_\{i\}dynamically moderates the balance between the base prior and the corrective teacher\. To safeguard exploration without compromising critical corrective signals, we propose a two\-stageAdaptive Gating Mechanism:

#### Spatial Sandbox Localization\.

Grounded in our findings in Section[3\.1](https://arxiv.org/html/2607.10805#S3.SS1), we restrict the gating intervention strictly to the student’s high\-entropy region𝒱High\-​H\\mathcal\{V\}\_\{\\text\{High\-\}H\}, defined as the top\-p%p\\%student entropy\. For any token outside this high\-entropy sandbox \(yj∉𝒱High\-​Hy\_\{j\}\\notin\\mathcal\{V\}\_\{\\text\{High\-\}H\}\), the teacher’s feedback is trusted completely, and we statically setUj=0U\_\{j\}=0, leaving standard factual error correction untouched\.

#### Pointwise Divergence Gating\.

Within the high\-entropy sandbox \(yi∈𝒱High\-​Hy\_\{i\}\\in\\mathcal\{V\}\_\{\\text\{High\-\}H\}\), we compute the unreliability indexUiU\_\{i\}using an asymmetrical soft pointwise divergence gating mechanism\. To selectively protect exploratory paths under suppression risk while preserving constructive teacher feedback, the gating is active exclusively in regions of positive pointwise log\-ratio:

Ui=\{σ​\(Pθ​\(yi∣⋅\)​\(log⁡Pθ​\(yi∣⋅\)−log⁡Pt​\(yi∣⋅\)\)τ\)if​ui\>00otherwiseU\_\{i\}=\\begin\{cases\}\\sigma\\left\(\\frac\{P\_\{\\theta\}\(y\_\{i\}\\mid\\cdot\)\\left\(\\log P\_\{\\theta\}\(y\_\{i\}\\mid\\cdot\)\-\\log P\_\{t\}\(y\_\{i\}\\mid\\cdot\)\\right\)\}\{\\tau\}\\right\)&\\text\{if \}u\_\{i\}\>0\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(5\)whereσ​\(⋅\)\\sigma\(\\cdot\)is the standard sigmoid,Pθ​\(yi∣⋅\)P\_\{\\theta\}\(y\_\{i\}\\mid\\cdot\)is shorthand forPθ​\(yi∣y<i,X\)P\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\},X\),uiu\_\{i\}islog⁡Pθ​\(yi∣⋅\)Pt​\(yi∣⋅\)\\log\\frac\{P\_\{\\theta\}\(y\_\{i\}\\mid\\cdot\)\}\{P\_\{t\}\(y\_\{i\}\\mid\\cdot\)\}, andτ\\tauis a temperature parameter \(set to1\.01\.0as default\)\. In implementation,UiU\_\{i\}is computed from stop\-gradient student probabilities and is used only to construct the token\-level target distribution\.

This asymmetrical formulation ensures that the epistemic anchorπbase\\pi\_\{\\text\{base\}\}is blended in smoothly via the pointwise KL contribution only when the student probability on a sampled token exceeds the teacher’s \(Pθ\>PtP\_\{\\theta\}\>P\_\{t\}\), representing potential thinking suppression\. Otherwise, when the teacher endorses or corrects the student’s output \(Pθ≤PtP\_\{\\theta\}\\leq P\_\{t\}\),UiU\_\{i\}collapses to0\.00\.0, allowing corrective gradients to proceed at full strength\.

#### Static Blending Variant\.

We also propose a simpler baseline variant namedStatic Dual Perspective OPSD \(SD\-OPSD\)\. This foundational variant fixes the unreliability index inside the active suppression region \(log⁡Ps\>log⁡Pt\\log P\_\{s\}\>\\log P\_\{t\}\) of the sandbox to a constant factor,Ui=αU\_\{i\}=\\alpha\(whereα=0\.5\\alpha=0\.5as default\)\. While this static variant can mitigate thinking collapse by anchoring the target, its uniform penalty does not distinguish between different levels of suppression risk across tokens\. In contrast, our defaultDynamic Blendingformulation represents a further optimization that treats pointwise student\-teacher divergence as a continuous indicator, selectively and proportionally shielding critical reasoning steps whereUiU\_\{i\}is active \(Ui\>0U\_\{i\}\>0\), while letting standard corrections where no suppression is detected \(Ui=0U\_\{i\}=0exactly\) proceed at full strength\.

### 4\.3AD\-OPSD Loss Formulation

The final optimization objective of AD\-OPSD minimizes the KL divergence toward our reformulated targetπi∗\\pi^\{\*\}\_\{i\}inside the high\-entropy sandbox𝒱High\-​H\\mathcal\{V\}\_\{\\text\{High\-\}H\}, while applying standard teacher\-guided OPSD across the remaining tokens:

ℒAD​\-​OPSD​\(θ\)=\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{AD\\text\{\-\}OPSD\}\}\(\\theta\)=\{\}∑yi∈𝒱High​\-​HDKL\(πi∗\(⋅\)∥πθ\(⋅∣y<i,X\)\)\\displaystyle\\sum\_\{y\_\{i\}\\in\\mathcal\{V\}\_\{\\mathrm\{High\}\\text\{\-\}H\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi^\{\*\}\_\{i\}\(\\cdot\)\\middle\\\|\\pi\_\{\\theta\}\(\\cdot\\mid y\_\{<i\},X\)\\right\)\(6\)\+∑yj∉𝒱High​\-​HDKL\(πt\(⋅\)∥πθ\(⋅∣y<j,X\)\)\.\\displaystyle\+\\sum\_\{y\_\{j\}\\notin\\mathcal\{V\}\_\{\\mathrm\{High\}\\text\{\-\}H\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{t\}\(\\cdot\)\\middle\\\|\\pi\_\{\\theta\}\(\\cdot\\mid y\_\{<j\},X\)\\right\)\.
By optimizing this sequence\-split objective, AD\-OPSD mitigates the thinking collapse and the thinking\-correction dilemma, preserving the model’s native exploratory reasoning capacity while keeping factual error correction fully functional\.

## 5Experimental Setup

#### Models and Datasets\.

To verify the generalizability of our proposed framework, we evaluate AD\-OPSD across two prominent reasoning model families at multiple scales: theQwen3\(Yanget al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib4)\)family \(specifically Qwen3\-1\.7B and Qwen3\-4B\) and the widely usedDeepSeek\-R1\-Distill\-Qwen\-1\.5Bmodel\(Guoet al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib10)\)\. For training data, we employ the mathematical reasoning subset of theOpenThoughts\(Guhaet al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib5)\)dataset, sampling up to 3K high\-quality problem\-solution pairs equipped with chain\-of\-thought \(CoT\) reasoning trajectories\. To comprehensively evaluate the models’ competition\-level mathematical reasoning and Test\-Time Scaling \(TTS\) potential, we benchmark performance across four challenging mathematics benchmarks:MATH\(Lightmanet al\.,[2023](https://arxiv.org/html/2607.10805#bib.bib15)\),AIME 2024\(Zhang and Math\-AI,[2024](https://arxiv.org/html/2607.10805#bib.bib17)\),AIME 2025\(Zhang and Math\-AI,[2025](https://arxiv.org/html/2607.10805#bib.bib18)\), andHMMT 2025\(Dekonincket al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib16)\)\. The details of the datasets are provided in Appendix[A](https://arxiv.org/html/2607.10805#A1)\.

#### Baselines\.

To systematically demonstrate that our AD\-OPSD framework mitigates the epistemic suppression bottleneck and preserves reasoning capabilities, we compare our approach against two foundational external baselines:

- •Base:The original pre\-trained reasoning model evaluated under its native thinking mode, representing the model’s native exploratory prior before alignment\.
- •OPSD:The standard On\-Policy Self\-Distillation\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib6)\)baseline trained under the fully unmasked Student\-Think/Teacher\-Think paradigm, where both the active student and the ground\-truth\-guided teacher utilize their full thinking capabilities\.

Additionally, we evaluate theStatic Blend \(wi=0\.5w\_\{i\}=0\.5\)variant of our method \(formulated in Section[4](https://arxiv.org/html/2607.10805#S4)\) as a direct internal ablation study to verify the necessity of our dynamic, compression\-governed soft\-gating mechanism\.

Table 1:Mathematical Reasoning Performance\(Accuracy, %\) Across Baselines and Models on Competition\-Level Benchmarks\. We compare our proposed AD\-OPSD framework against the pre\-trained Base Model, standard OPSD, and our static baseline variant SD\-OPSD\. Bold indicates the absolute best\-performing model for each metric\.
#### Implementation Details\.

Following the training protocol established in standard on\-policy distillation frameworks\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib6)\), we fix the teacher policy as the initial, pre\-trained starting checkpoint\. This stabilized target setup acts as an implicit regularizer, preventing excessive policy deviation from the native exploratory prior\. We perform full\-vocabulary logit distillation across all training runs\. All experiments are implemented using Parameter\-Efficient Fine\-Tuning via LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.10805#bib.bib7)\)and conducted on 4 NVIDIA RTX\-5090 GPUs\. Detailed hyperparameters, templates, and optimization configurations are provided in Appendix[B](https://arxiv.org/html/2607.10805#A2)and[C](https://arxiv.org/html/2607.10805#A3)\.

## 6Main Results

We evaluate the mathematical reasoning capabilities of our proposedAD\-OPSDframework alongside our baselines across multiple competitive mathematics benchmarks\. The main experimental results are summarized in Table[1](https://arxiv.org/html/2607.10805#S5.T1)\. We compare our proposed adaptive framework \(AD\-OPSD\) against the Base Model, standard OPSD, and our static baseline variant \(SD\-OPSD\)\.

#### Consistent Performance Gains across Different Models and Datasets\.

Across all evaluated model scales and datasets,AD\-OPSDconsistently and systematically outperforms standard OPSD\. While standard OPSD degrades reasoning performance compared to the Base baseline \(e\.g\., plunging from 38\.1% to 33\.9% on AIME 2025 for Qwen\-1\.7B, and from 69\.2% to 67\.0% average for Qwen\-4B\), AD\-OPSD completely mitigates this optimization pathology, delivering a substantial performance surge \(e\.g\., achieving \+4\.1% absolute improvement over standard OPSD on Qwen\-1\.7B average\)\. Crucially, this performance gap is even more pronounced onharder reasoning tasks\. For example, on the challenging AIME 2025 benchmark, AD\-OPSD achieves massive absolute boosts of\+6\.7%\(40\.6% vs 33\.9%\) on Qwen\-1\.7B and\+3\.1%\(66\.4% vs 63\.3%\) on Qwen\-4B over standard OPSD\. This pattern indicates that preserving the student’s exploratory reasoning steps is especially vital when dealing with complex, multi\-step logical deductions where standard distillation gradients induce catastrophic thinking collapse\.

#### Effectiveness of Static Anchoring and Superiority of Dynamic Gating\.

From the results in Table[1](https://arxiv.org/html/2607.10805#S5.T1), we observe that even our proposed simpler static variant,SD\-OPSD, is highly effective compared to standard OPSD\. By anchoring high\-entropy suppressed states to the unsuppressed base prior, SD\-OPSD systematically outperforms OPSD across all backbones, delivering average accuracy boosts of up to\+3\.2%absolutely over standard OPSD\. This confirms that static anchoring within the sandbox is highly capable of safeguarding exploratory reasoning trajectories\. However, compared to this static baseline, our default dynamicAD\-OPSDframework achieves further superior performance on the Qwen models, reaching70\.2%vs68\.8%on Qwen\-4B and51\.8%vs50\.9%on Qwen\-1\.7B\. This result proves that, compared to a uniform blend, dynamically adjusting the unreliability index based on local pointwise KL is more effective, validating the necessity of our pointwise divergence soft\-gating\.

## 7Discussion and Ablation Studies

In this section, we conduct a series of empirical investigations to discuss and resolve the following four core questions regarding the performance, robustness, and parameters of AD\-OPSD:

- •Q1 \(Thinking Density\):Does our proposed AD\-OPSD successfully preserve and recover the student’s thinking density under on\-policy self\-distillation?
- •Q2 \(Generality\):Can our robust control framework generalize to non\-thinking alignment paradigms \(specifically the standard NT/NT distillation\)?
- •Q3 \(Context Horizon\):How does the maximum sequence length \(LmaxL\_\{\\text\{max\}\}\) used during training influence our method’s behavior?
- •Q4 \(Boundary and Weight\):What are the optimal sandbox spatial boundaries and gating weight functions to balance epistemic preservation and corrective feedback?

### 7\.1Preservation of Thinking Density

To verify whether our method can recover the thinking density under OPSD, we evaluate our proposed AD\-OPSD framework against the Base model and standard OPSD \(T/T\) on Qwen3\-1\.7B\. We track downstream reasoning accuracy and the corresponding thinking density \(ET per 1k\) across the challenging AIME 2024 and AIME 2025 benchmarks\.

Table 2:Reasoning Performance \(Accuracy, %\) and Thinking Density \(TD\)on Qwen3\-1\.7B across AIME benchmarks\.The experimental results, summarized in Table[2](https://arxiv.org/html/2607.10805#S7.T2), demonstrate several critical empirical advantages\. First, standard OPSD triggers a severe collapse in thinking density, which plunges by24\.2%\(from 10\.3 to 7\.8\) on AIME 2024 and25\.5%\(from 10\.6 to 7\.9\) on AIME 2025\. In contrast, AD\-OPSD partially recovers the thinking density to8\.4and8\.6respectively\. This recovery confirms that our asymmetrical unreliability gating shields the student’s exploratory paths from destructive distillation gradients\. Second, by preserving reasoning pathways while enforcing corrective teacher feedback, AD\-OPSD improves accuracy over standard OPSD by\+5\.3%on AIME 2024 and\+6\.7%on AIME 2025\. This validation shows that our dynamic target formulation can break the thinking\-correction dilemma, enabling the model to retain more of its Test\-Time Scaling potential while improving performance\.

Table 3:Performance Comparison\(Accuracy, %\) of Qwen3\-1\.7B Trained Under NT/NT Alignment, Evaluated across NonThinking and Thinking Inference Modes\.
### 7\.2Generalization to Non\-Thinking Alignment Paradigms

To demonstrate the generalization of our method, we apply our robust control mechanism to the Non\-Think Student / Non\-Think Teacher \(NT/NT\) alignment configuration, where explicit intermediate reasoning tags are completely stripped during distillation\. We train Qwen3\-1\.7B with standard OPSD and AD\-OPSD strictly under the NT/NT setup, and evaluate the aligned models under both NonThinking and Thinking inference modes\.

The empirical findings in Table[3](https://arxiv.org/html/2607.10805#S7.T3)show that even in the absence of explicit intermediate reasoning chains, AD\-OPSD mitigates phrasing suppression under NonThinking inference, raising the average accuracy from standard OPSD’s 12\.0% to13\.5%\(outperforming the Base baseline of 12\.4%\)\. Furthermore, when evaluated under Thinking inference mode \(activating test\-time chain\-of\-thought\), the performance boost is substantial\. While standard OPSD achieves an average accuracy of only 43\.2%, AD\-OPSD climbs to48\.1%\(a\+4\.9% absolute improvementover OPSD and\+4\.7%over Base\)\. This phenomenon suggests that standard OPSD can corrupt latent reasoning structures even under stripped NT/NT post\-training, while AD\-OPSD’s protective anchoring helps safeguard these latent reasoning circuits and preserve test\-time scaling capability\.

### 7\.3Influence of Maximum Training Context Length

To analyze the influence of AD\-OPSD to the maximum sequence length used during training \(LmaxL\_\{\\text\{max\}\}\), we train Qwen3\-1\.7B under three distinct context horizons:512,1024, and2048tokens, and plot the comparative accuracy and relative gains in Figures[4](https://arxiv.org/html/2607.10805#S7.F4)\(a\) and[4](https://arxiv.org/html/2607.10805#S7.F4)\(b\)\.

![Refer to caption](https://arxiv.org/html/2607.10805v1/x4.png)Figure 4:Consolidated discussion and hyperparameter studyfor Qwen3\-1\.7B on AIME benchmarks\. Subfigures \(a\) and \(b\) present the downstream reasoning accuracy of Standard OPSD and our proposed AD\-OPSD across varying maximum context lengths \(Lmax∈\{512,1024,2048\}L\_\{\\text\{max\}\}\\in\\\{512,1024,2048\\\}\) with absolute gains \(Δ\\Delta\) clearly annotated\. Subfigure \(c\) presents the hyperparameter study on the Entropy Sandbox percentile ratio \(p%∈\{10%,20%,30%,40%,100%\}p\\%\\in\\\{10\\%,20\\%,30\\%,40\\%,100\\%\\\}\) against standard OPSD \(0% Sandbox\) represented as horizontal baselines\.The empirical findings, visualized in Figures[4](https://arxiv.org/html/2607.10805#S7.F4)\(a\) and[4](https://arxiv.org/html/2607.10805#S7.F4)\(b\), reveal two key properties: AD\-OPSD consistently and systematically outperforms standard OPSD across all context horizons\. However, the margin of improvement \(Δ\\Delta\) exhibits a clear tapering trend on longer horizons\. Specifically, the absolute gain on AIME 2024 narrows from\+7\.5%atLmax=512L\_\{\\text\{max\}\}=512down to\+5\.3%at10241024, and further down to\+1\.3%at20482048tokens\. A similar trend is observed on AIME 2025, where the margin tapers from\+5\.5%\(512\) and\+6\.7%\(1024\) down to\+2\.2%\(2048\)\.

This diminishing margin of improvement on longer context horizons is explained by a natural optimization convergence of target distributions:

limi→∞∥\\displaystyle\\lim\_\{i\\to\\infty\}\\Big\\\|πt​\(yi∣y<i,X,GT\)\\displaystyle\\pi\_\{t\}\(y\_\{i\}\\mid y\_\{<i\},X,\\text\{GT\}\)\(7\)−πbase\(yi∣y<i,X\)∥→0\\displaystyle\-\\pi\_\{\\text\{base\}\}\(y\_\{i\}\\mid y\_\{<i\},X\)\\Big\\\|\\to 0This behavior is strongly supported by recent literature\. For instance,Zhanget al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib30)\)demonstrate that distillation signals are naturally concentrated within early reasoning prefixes, whereasZihenget al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib29)\)identify that teacher supervision quality degrades on late tokens due to cumulative off\-policy drift \(termedOff\-Policy Teacher Decay\)\.

These insights align perfectly with our dual\-perspective target formulation in Equation[4](https://arxiv.org/html/2607.10805#S4.E4)\. During the early\-to\-mid stages of generation \(short prefixes\), prompt\-response alignment constraints are highly active, and standard OPSD imposes aggressive information compression, making our protective anchoring highly effective\. As the horizon extends, autoregressive uncertainty and off\-policy drift cause the teacher’s distribution to converge toward the base prior\. Consequently, the active optimization conflict dissipates in long sequences, naturally narrowing the performance gap\.

### 7\.4Sensitivity and Ablation Analysis of the Sandbox Configuration

To deeply investigate our design space, we perform two empirical analyses evaluating \(1\) spatial sensitivity to the sandbox percentilep%p\\%, and \(2\) ablation of alternative gating weight functions\. All experiments are conducted on Qwen3\-1\.7B across the competitive AIME benchmarks\.

#### Spatial Sensitivity on Sandbox Percentile\.

We perform a parameter sweep overp%∈\{10%,20%,30%,40%,100%\}p\\%\\in\\\{10\\%,20\\%,30\\%,40\\%,100\\%\\\}to calibrate the localized boundary of the high\-entropy sandbox, establishing standard OPSD \(0%0\\%Sandbox\) as a reference baseline in Figure[4](https://arxiv.org/html/2607.10805#S7.F4)\(c\)\. The empirical results reveal that AD\-OPSD consistently outperforms the standard OPSD baseline across all protection percentiles on both benchmarks\. Notably, even under the extreme100%100\\%full\-protection sandbox, AD\-OPSD achieves51\.9%on AIME 2024 and35\.3%on AIME 2025 \(exceeding standard OPSD’s 45\.3% and 33\.9%, respectively\), highlighting the effectiveness of our dual\-perspective anchoring\. Furthermore, downstream accuracy exhibits a distinct bell\-curve trend as the sandbox sizep%p\\%increases, supporting our spatial diagnostics in Section[3\.1](https://arxiv.org/html/2607.10805#S3.SS1)\. Restricting the anchoring shield to highly localized decision forks \(the top 20%–30% predictive entropy\) is sufficient to restore reasoning pathways, as AIME 2024 peaks at a wider sandbox of30%\(54\.7%\) while the more challenging AIME 2025 peaks at a more compact sandbox of20%\(40\.6%\)\. Beyond these thresholds, over\-expanding the sandbox introduces a correction dilution problem that weakens vital teacher corrective gradients\. Thus, localizing the protection to the top 20%–30% highest\-entropy region strikes the optimal balance, and we selectp%=20%p\\%=20\\%as default\.

#### Ablation on Gating Weight Functions\.

Within the active suppression region \(log⁡Ps\>log⁡Pt\\log P\_\{s\}\>\\log P\_\{t\}\) of the top\-20% Entropy Sandbox, we ablate our default thresholded pointwise KL sigmoid gate against three alternative weighting functions: \(1\) a pure log\-ratio gate \(Pure Log\-Ratio,σ​\(log⁡Ps−log⁡Pt\)\\sigma\(\\log P\_\{s\}\-\\log P\_\{t\}\)\), \(2\) a global vocabulary\-level uncertainty difference gate \(Global Entropy,σ​\(Hs−Ht\)\\sigma\(H\_\{s\}\-H\_\{t\}\)\), and \(3\) a hyperbolic tangent mapping of the pointwise KL \(Tanh Pointwise,tanh⁡\(Ps​\(log⁡Ps−log⁡Pt\)\)\\tanh\(P\_\{s\}\(\\log P\_\{s\}\-\\log P\_\{t\}\)\)\)\.

Table 4:Ablation Studyon Alternative Gating Formulations on Qwen3\-1\.7B\.The empirical results in Table[4](https://arxiv.org/html/2607.10805#S7.T4)demonstrate that our proposedSigmoid Pointwiseformulation achieves the highest average reasoning accuracy \(45\.6%\) and peaks at40\.6%on the challenging AIME 2025 benchmark, outperforming the next\-best alternative by\+3\.9% absolute\. In contrast, while thePure Log\-Ratiogate excels on the simpler AIME 2024, its performance sharply decays to35\.3%on AIME 2025, revealing that lacking student\-confidence dampening \(PsP\_\{s\}\) leaves it vulnerable to false\-positive protection from near\-zero probability noise\. Furthermore, theGlobal Entropygate yields stable but sub\-optimal performance \(43\.5%average\) due to its lack of token\-specific granularity, while theTanh Pointwisegate \(43\.2%average\) suffers from steep gradient transitions that destabilize early optimization\. Ultimately, this ablation justifies our thresholded pointwise KL sigmoid gating as the most robust and precise mechanism for protecting exploratory reasoning pathways\.

## 8Related Work

Our work is closely related to three lines of research: reasoning post\-training and test\-time scaling, on\-policy distillation and its pathologies, and token\-level intervention for preserving reasoning behavior\.

### 8\.1Reasoning Post\-Training and Test\-Time Scaling

Exposing and sampling intermediate reasoning paths \(e\.g\., chain\-of\-thought\) lies at the core of unleashing LLM reasoning capabilities\(Weiet al\.,[2022](https://arxiv.org/html/2607.10805#bib.bib20); Wanget al\.,[2023](https://arxiv.org/html/2607.10805#bib.bib21)\)\. While outcome or step\-level supervision provides dense feedback to guide these paths\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.10805#bib.bib22); Lightmanet al\.,[2023](https://arxiv.org/html/2607.10805#bib.bib15)\), process\-level alignment studies further suggest that optimizing only final outcomes can damage reasoning integrity even when final accuracy improves\(Wuet al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib42)\)\. Recent advances in reasoning models demonstrate that enabling longer rollouts and test\-time scaling compute is crucial to high performance on complex reasoning tasks\(Snellet al\.,[2024](https://arxiv.org/html/2607.10805#bib.bib23); Guoet al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib10)\)\. At the same time, recent analyses caution that reasoning quality is not determined by length alone: redundant self\-doubt can induce overthinking, self\-improvement training can suffer from collapse on complex reasoning tasks, and efficiency\-oriented objectives can favor denser reasoning traces without sacrificing accuracy\(Penget al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib39); Zhonget al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib41); Penget al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib40)\)\. These works primarily study how to elicit, allocate, or compress reasoning compute\. However, they do not address why standard post\-training optimization objectives might inadvertently erase these vital intermediate paths\. This optimization pathology motivates us to investigate the underlying causes of thinking collapse under on\-policy self\-distillation, shifting the focus from increasing generation length to protecting native thinking density\.

### 8\.2On\-Policy Distillation and its Pathologies

To bridge the train\-test mismatch of standard sequence\-level distillation, on\-policy distillation \(OPD\) optimizes the student policy over its own sampled trajectories\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.10805#bib.bib31)\)\. Although OPD avoids exposure bias, recent studies highlight its fragility\. For instance, successful alignment requires compatible student\-teacher thinking patterns\(Liet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib32)\), failing which can lead to imbalanced token\-level supervision, prefix\-drift, or tokenizer mismatch\(Fuet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib37); Zhuet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib33)\)\. To leverage privileged teacher information while avoiding these pathologies, on\-policy self\-distillation \(OPSD\) has emerged to condition the policy on gold solutions or detailed feedback\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib6); Heet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib35)\)\. Different from general OPD/OPSD studies, we show that the core failure mode of OPSD on reasoning tasks is not global mismatch, but rather a localized collapse of high\-student\-entropy thinking regions, which we resolve through selective reference anchoring\.

### 8\.3Token\-Level Intervention and Fork Suppression

Our diagnosis of thinking collapse is closely related to prior investigations on self\-distillation risks\(Kimet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib1); Kauret al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib11)\)\. Specifically,Kimet al\.\([2026b](https://arxiv.org/html/2607.10805#bib.bib1)\)show that distillation suppresses epistemic verbalization, whileKauret al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib11)\)identify that privileged\-context targets suppress high\-entropy decision forks\. Most closely concurrent with our work, Purified OPSD\(Shenet al\.,[2026b](https://arxiv.org/html/2607.10805#bib.bib36)\)addresses this suppression pathology by decomposing the teacher update using reference\-only prompting to filter reference\-specific shortcuts\. While highly relevant, our approach differs fundamentally in mechanism: whereas Purified OPSD relies on teacher\-side target reconstruction across all tokens, AD\-OPSD operates directly on the student side\. By restricting intervention to a localized high\-entropy sandbox and scaling the unreliability index via pointwise KL, our method provides a dynamic, uncertainty\-aware control gate to protect critical reasoning paths\. Our formulation is also related in spirit to token\-level self\-evolution training, which dynamically regularizes uncertain learning states at a finer granularity than sequence\-level objectives\(Penget al\.,[2023](https://arxiv.org/html/2607.10805#bib.bib38)\)\. This selective student\-side intervention aligns perfectly with findings that reasoning updates concentrate heavily on high\-entropy decision tokens\(Wanget al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib3); Xuet al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib14)\), offering a complementary and highly precise control framework\.

## 9Conclusion

In this paper, we diagnosed and mitigatedThinking Collapse, an optimization pathology in reasoning\-oriented On\-Policy Self\-Distillation \(OPSD\) where the model’s native intermediate reasoning behavior sharply declines\. Through entropy\-based gradient masking, token\-level discrepancy analysis, and pointwise divergence mapping, we showed that this collapse is concentrated at high\-student\-entropy decision forks, where teacher gradients frequently suppress student epistemic tokens into non\-epistemic targets\.

To address this pathology, we proposedAdaptive Dual\-Perspective OPSD \(AD\-OPSD\), a robust post\-training framework that dynamically interpolates the teacher distribution with a frozen base\-model reference prior\. By applying an asymmetrical pointwise divergence gate within a localized high\-entropy sandbox, AD\-OPSD preserves native thinking behavior while retaining OPSD’s corrective supervision\. Experiments across competitive mathematical reasoning benchmarks demonstrate that AD\-OPSD consistently improves over standard OPSD, yielding up to a\+4\.1%absolute average accuracy gain, recovering thinking density, and generalizing to non\-thinking alignment configurations\. These findings suggest that stable reasoning post\-training requires not only stronger supervision, but also selective protection of the model’s native exploratory reasoning paths\.

## Limitations

Despite the consistent empirical gains and robust theoretical justifications of our proposed framework, several limitations warrant further research:

- •Model Scale and Parameter\-Efficient Training:Due to computational resource constraints, our empirical evaluation is primarily restricted to compact model scales trained using Parameter\-Efficient Fine\-Tuning via LoRA\. Although our results demonstrate clear statistical significance and systematically mitigate thinking collapse at these scales, examining the scaling laws and generalizability of AD\-OPSD on ultra\-large models \(e\.g\., 32B, 70B, or larger frontier models\) and under full\-parameter fine\-tuning is a vital direction for industrial\-scale deployment\.
- •Domain Specialization in Mathematics:Our diagnostics and main evaluations are strictly situated within competition\-level math reasoning benchmarks\. Mathematics is selected because symbolic deduction naturally exhibits dense, long\-horizon chain\-of\-thought trajectories where thinking collapse is exceptionally pronounced\. However, the generalizability of our dual\-perspective anchoring framework to non\-mathematical complex reasoning domains \(such as symbolic logic, code generation, or multi\-hop reading comprehension\) remains to be extensively explored\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InICLR,External Links:[Link](https://arxiv.org/abs/2306.13649)Cited by:[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InICLR,External Links:[Link](https://openreview.net/pdf?id=mZn2Xyh9Ec)Cited by:[Appendix B](https://arxiv.org/html/2607.10805#A2.p3.1)\.
- J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvaldsson, I\. Petrov, C\. Sun, and M\. Vechev \(2026\)Beyond benchmarks: matharena as an evaluation platform for mathematics with llms\.External Links:2605\.00674,[Link](https://arxiv.org/abs/2605.00674)Cited by:[3rd item](https://arxiv.org/html/2607.10805#A1.I1.i3.p1.1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Fu, H\. Huang, K\. Jiang, Y\. Zhu, and D\. Zhao \(2026\)Revisiting on\-policy distillation: empirical failure modes and simple fixes\.arXiv preprint arXiv:2603\.25562\.External Links:[Link](https://arxiv.org/abs/2603.25562)Cited by:[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- E\. K\. Guha, R\. Marten, S\. Keh, N\. Raoof, G\. Smyrnis, H\. Bansal, M\. Nezhurina, J\. Mercat, T\. Vu, Z\. R\. Sprague,et al\.\(2025\)OpenThoughts: data recipes for reasoning models\.InFirst Workshop on Foundations of Reasoning in Language Models,External Links:[Link](https://openreview.net/pdf?id=7xjoTuaNmN)Cited by:[§A\.1](https://arxiv.org/html/2607.10805#A1.SS1.p1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.External Links:[Link](https://arxiv.org/pdf/2501.12948)Cited by:[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1),[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- Y\. He, S\. Kaur, A\. Bhaskar, Y\. Yang, J\. Liu, N\. Ri, L\. Fowl, A\. Panigrahi, D\. Chen, and S\. Arora \(2026\)Self\-distillation zero: self\-revision turns binary rewards into dense supervision\.arXiv preprint arXiv:2604\.12002\.External Links:[Link](https://arxiv.org/abs/2604.12002)Cited by:[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- E\. J\. Hu, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)LoRA: low\-rank adaptation of large language models\.InICLR,External Links:[Link](https://openreview.net/pdf?id=nZeVKeeFYf9)Cited by:[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.External Links:[Link](https://arxiv.org/pdf/2601.20802)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p1.1)\.
- S\. Kaur, N\. Ri, Y\. He, L\. H\. Fowl, and S\. Arora \(2026\)Rethinking on\-policy self\-distillation for thinking models\.InICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning,External Links:[Link](https://openreview.net/pdf?id=VhCJItwQHn)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p2.1),[§2](https://arxiv.org/html/2607.10805#S2.p1.1),[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- J\. Kim, J\. Jeon, D\. Li, and Y\. Yang \(2026a\)Rebellious student: reversing teacher signals for reasoning exploration with self\-distilled rlvr\.arXiv preprint arXiv:2605\.10781\.External Links:[Link](https://arxiv.org/pdf/2605.10781)Cited by:[§3\.3](https://arxiv.org/html/2607.10805#S3.SS3.p1.1)\.
- J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. Yang \(2026b\)Why does self\-distillation \(sometimes\) degrade the reasoning capability of llms?\.arXiv preprint arXiv:2603\.24472\.External Links:[Link](https://arxiv.org/pdf/2603.24472)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p2.1),[§2](https://arxiv.org/html/2607.10805#S2.p1.1),[§2](https://arxiv.org/html/2607.10805#S2.p2.3),[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.External Links:[Link](https://arxiv.org/abs/2604.13016)Cited by:[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InICLR,External Links:[Link](https://openreview.net/forum?id=O1gRv0T4uX)Cited by:[1st item](https://arxiv.org/html/2607.10805#A1.I1.i1.p1.1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1),[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- K\. Peng, L\. Ding, Y\. Ouyang, M\. Fang, and D\. Tao \(2025\)Revisiting overthinking in long chain\-of\-thought from the perspective of self\-doubt\.arXiv preprint arXiv:2505\.23480\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.23480),[Link](https://arxiv.org/abs/2505.23480)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- K\. Peng, L\. Ding, Q\. Zhong, Y\. Ouyang, W\. Rong, Z\. Xiong, and D\. Tao \(2023\)Token\-level self\-evolution training for sequence\-to\-sequence learning\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Toronto, Canada,pp\. 841–850\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-short.73),[Link](https://aclanthology.org/2023.acl-short.73/)Cited by:[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- K\. Peng, Y\. Ouyang, X\. Liu, Z\. Tian, R\. Han, Y\. Yuan, and L\. Ding \(2026\)Think dense, not long: dynamic decoupled conditional advantage for efficient reasoning\.arXiv preprint arXiv:2602\.02099\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.02099),[Link](https://arxiv.org/abs/2602.02099)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- G\. Shen, X\. Cheng, C\. Zhao, L\. Huang, J\. Li, D\. Zhao, and X\. Yu \(2026a\)Anti\-self\-distillation for reasoning rl via pointwise mutual information\.arXiv preprint arXiv:2605\.11609\.External Links:[Link](https://arxiv.org/pdf/2605.11609)Cited by:[§3\.3](https://arxiv.org/html/2607.10805#S3.SS3.p1.1)\.
- Z\. Shen, J\. Tong, S\. Yan, C\. Shen, H\. Chen, W\. Ye, X\. Hu, R\. Miao, H\. Wang, J\. Zhao, G\. Chen, and J\. Ye \(2026b\)Purified opsd: on\-policy self\-distillation without losing how to think\.arXiv preprint arXiv:2607\.02234\.External Links:[Link](https://arxiv.org/abs/2607.02234)Cited by:[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.External Links:[Link](https://arxiv.org/pdf/2601.19897)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p1.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.External Links:[Link](https://arxiv.org/abs/2408.03314)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang,et al\.\(2026\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.NIPS\.External Links:[Link](https://openreview.net/pdf?id=yfcpdY4gMP)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.10805#S3.SS1.p1.4),[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InICLR,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InNIPS,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- Y\. Wu, L\. Ding, L\. Shen, and D\. Tao \(2025\)Reason\-ke\+\+: aligning the process, not just the outcome, for faithful llm knowledge editing\.arXiv preprint arXiv:2511\.12661\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2511.12661),[Link](https://arxiv.org/abs/2511.12661)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard \(2026\)Tip: token importance in on\-policy distillation\.arXiv preprint arXiv:2604\.14084\.External Links:[Link](https://arxiv.org/pdf/2604.14084)Cited by:[§1](https://arxiv.org/html/2607.10805#S1.p3.1),[§8\.3](https://arxiv.org/html/2607.10805#S8.SS3.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/pdf/2505.09388)Cited by:[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1)\.
- C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan \(2026\)Self\-distilled rlvr\.arXiv preprint arXiv:2604\.03128\.External Links:[Link](https://arxiv.org/pdf/2604.03128)Cited by:[§3\.3](https://arxiv.org/html/2607.10805#S3.SS3.p1.1)\.
- D\. Zhang, Z\. Yang, S\. Janghorbani, J\. Han, A\. Ressler II, Q\. Qian, G\. D\. Lyng, S\. S\. Batra, and R\. E\. Tillman \(2026\)Fast and effective on\-policy distillation from reasoning prefixes\.InFindings of ACL,External Links:[Link](https://aclanthology.org/2026.findings-acl.1276.pdf)Cited by:[§7\.3](https://arxiv.org/html/2607.10805#S7.SS3.p3.2)\.
- Y\. Zhang and T\. Math\-AI \(2024\)American invitational mathematics examination \(aime\) 2024\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime24)Cited by:[2nd item](https://arxiv.org/html/2607.10805#A1.I1.i2.p1.1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhang and T\. Math\-AI \(2025\)American invitational mathematics examination \(aime\) 2025\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[2nd item](https://arxiv.org/html/2607.10805#A1.I1.i2.p1.1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px1.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.External Links:[Link](https://arxiv.org/pdf/2601.18734)Cited by:[Appendix B](https://arxiv.org/html/2607.10805#A2.p3.1),[§1](https://arxiv.org/html/2607.10805#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.10805#S3.SS1.p1.4),[2nd item](https://arxiv.org/html/2607.10805#S5.I1.i2.p1.1),[§5](https://arxiv.org/html/2607.10805#S5.SS0.SSS0.Px3.p1.1),[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- Q\. Zhong, L\. Ding, J\. Liu, B\. Du, L\. Rutkowski, and D\. Tao \(2026\)Better, faster: harnessing self\-improvement in large reasoning models\.arXiv preprint arXiv:2605\.24998\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.24998),[Link](https://arxiv.org/abs/2605.24998)Cited by:[§8\.1](https://arxiv.org/html/2607.10805#S8.SS1.p1.1)\.
- S\. Zhu, X\. Ye, H\. Lu, W\. Shi, and G\. Liu \(2026\)The many faces of on\-policy distillation: pitfalls, mechanisms, and fixes\.arXiv preprint arXiv:2605\.11182\.External Links:[Link](https://arxiv.org/abs/2605.11182)Cited by:[§8\.2](https://arxiv.org/html/2607.10805#S8.SS2.p1.1)\.
- Z\. Ziheng, J\. Li, H\. Tang, Y\. N\. Wu, and D\. Terzopoulos \(2026\)Less is more: early stopping rollout for on\-policy distillation\.arXiv preprint arXiv:2605\.27028\.External Links:[Link](https://arxiv.org/pdf/2605.27028)Cited by:[§7\.3](https://arxiv.org/html/2607.10805#S7.SS3.p3.2)\.

## Appendix ADataset Specifications

To support reproducibility, we detail the training corpus, downstream benchmarks, optimization settings, method\-specific hyperparameters, evaluation protocol, and prompt templates used in our experiments\.

### A\.1Training Corpus

We sample up to 3K high\-quality mathematical reasoning problem\-solution pairs from the open\-source OpenThoughts dataset\(Guhaet al\.,[2025](https://arxiv.org/html/2607.10805#bib.bib5)\)\. Each training example contains a problem statement and a reference solution used as privileged teacher context during self\-distillation\. The same sampled training subset is used across all trained OPSD variants, including standard OPSD, SD\-OPSD, and AD\-OPSD, so that observed differences arise from the distillation objective rather than from data variation\. Downstream benchmark problems are used only for evaluation\.

OpenThoughts is well suited for our study because its examples contain dense, multi\-step chain\-of\-thought \(CoT\) reasoning trajectories\. Such trajectories naturally include high\-entropy decision forks, self\-verification steps, and backtracking behavior, making the dataset an appropriate setting for diagnosing whether OPSD suppresses native exploratory reasoning\.

### A\.2Downstream Evaluation Benchmarks

We benchmark the mathematical reasoning capacity and test\-time scaling behavior of aligned models across four competition\-grade math datasets:

- •MATH / MATH500\(Lightmanet al\.,[2023](https://arxiv.org/html/2607.10805#bib.bib15)\):We evaluate on the commonly used MATH500 subset, which covers high\-school competition\-level mathematical problems across algebra, number theory, geometry, probability, combinatorics, prealgebra, and precalculus\.
- •AIME 2024\(Zhang and Math\-AI,[2024](https://arxiv.org/html/2607.10805#bib.bib17)\)and AIME 2025\(Zhang and Math\-AI,[2025](https://arxiv.org/html/2607.10805#bib.bib18)\):The American Invitational Mathematics Examination contains challenging problems with integer answers between 0 and 999\. These benchmarks require long\-horizon symbolic reasoning and extensive verification, making them central for evaluating preservation of native thinking capacity\.
- •HMMT 2025\(Dekonincket al\.,[2026](https://arxiv.org/html/2607.10805#bib.bib16)\):The Harvard\-MIT Mathematics Tournament contains difficult competition problems that require rigorous mathematical precision and creative exploratory paths, posing a challenging testbed for distilled reasoning backbones\.

## Appendix BExperimental Details

We provide the shared training configuration in Table[5](https://arxiv.org/html/2607.10805#A2.T5), the method\-specific AD\-OPSD configuration in Table[6](https://arxiv.org/html/2607.10805#A2.T6), and the evaluation configuration in Table[7](https://arxiv.org/html/2607.10805#A2.T7)\. Unless otherwise stated for ablation studies, all OPSD variants use the same training data, LoRA configuration, optimizer, batch size, number of training steps, and sampling settings\.

Table 5:Shared Training Configuration for OPSD VariantsTable 6:Default AD\-OPSD and SD\-OPSD Method Configuration\.For the sandbox sensitivity study in Section[7](https://arxiv.org/html/2607.10805#S7), we sweepp%∈\{10%,20%,30%,40%,100%\}p\\%\\in\\\{10\\%,20\\%,30\\%,40\\%,100\\%\\\}while keeping the other training settings fixed\. For the context\-length study, we vary the maximum completion length over\{512,1024,2048\}\\\{512,1024,2048\\\}tokens\. For the gating ablation, we replace only the gating weight function and keep the top\-20%20\\%entropy sandbox fixed\.

Table 7:Evaluation Parameters\.During evaluation, we sample 4 completions per prompt for MATH500 and 12 completions per prompt for AIME 2024, AIME 2025, and HMMT 2025, and report the mean accuracy over sampled completions followingZhaoet al\.\([2026](https://arxiv.org/html/2607.10805#bib.bib6)\)\. All experiments were conducted using 4 RTX5090 GPUs with gradient checkpointing and Flash Attention 2\(Dao,[2024](https://arxiv.org/html/2607.10805#bib.bib27)\)for memory efficiency\.

### B\.1Answer Extraction and Scoring

All models are prompted to put the final answer inside`\\boxed\{\}`\. We evaluate mathematical correctness by exact match after normalized answer extraction\. When a response contains one or more boxed answers, we extract the final boxed expression as the model’s answer\. If no boxed expression is present, we fall back to the final explicit answer\-like expression in the completion\. We normalize common formatting artifacts such as whitespace, surrounding punctuation, and simple LaTeX wrappers before comparison\. For AIME\-style benchmarks, answers are compared as normalized integer strings in the expected0–999999range\. For MATH500 and HMMT\-style problems, we compare the normalized extracted expression against the benchmark reference answer\.

### B\.2Thinking and Non\-Thinking Modes

Our main OPSD experiments use the thinking mode, where the student is prompted to reason step by step and output a boxed final answer\. In NonThinking configurations \(e\.g\., NT/NT analysis\), explicit intermediate reasoning instructions are removed and the model is prompted to provide the final answer directly\. For Thinking inference, we use the step\-by\-step prompt template in Section[C](https://arxiv.org/html/2607.10805#A3); for NonThinking inference, we use the same problem format but omit the step\-by\-step instruction\. This separation allows us to test whether post\-training damages only explicit reasoning traces or also the latent reasoning behavior recoverable under Thinking inference\.

## Appendix CPrompt Templates

We fill the following templates before applying each model’s chat template\. For standard OPSD training, the student receives only the problem, while the teacher receives the same problem together with the reference solution as privileged context\. The reference prior used by AD\-OPSD is the frozen base model scored on the same student prefix without ground\-truth context\.

Student Prompt TemplateProblem:\{problem\}Please reason step by step, and put your final answer within \\boxed\{\}\.

The teacher model is guided by a cognitive transition prompt to encourage independent reasoning and verification rather than direct copying of the reference solution\.

Teacher Prompt TemplateProblem:\{problem\}Here is a reference solution to this problem:=== Reference Solution Begin ===\{solution\}=== Reference Solution End ===After reading the reference solution above, make sure you truly understand the reasoning behind each step \-\-\- do not copy or paraphrase it\. Now, using your own words and independent reasoning, derive the same final answer to the problem above\. Think step by step, explore different approaches, and don’t be afraid to backtrack or reconsider if something doesn’t work out:Please reason step by step, and put your final answer within \\boxed\{\}\.

Here\{solution\}denotes the reference solution used as teacher context during distillation\.

NonThinking Prompt TemplateProblem:\{problem\}Please provide the final answer within \\boxed\{\}\.