Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

# Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
Source: [https://arxiv.org/html/2606.07000](https://arxiv.org/html/2606.07000)
Shizhe Xiang1\*†, Ke An2\*†, Wenlong Yu3, Yue Liu4†, Jian Luan5, Pei Fu5§,Qilong Wang1‡, 1Tianjin University2Beijing Institute of Technology3Singapore Management University 4University of Chinese Academy of Sciences5Xiaomi Inc

###### Abstract

Recent post\-training methods, particularly Reinforcement Learning with Verifiable Rewards \(RLVR\), have significantly enhanced the reasoning ability of Large Vision\-Language Models \(LVLMs\)\. However, the sparse nature of verifiable rewards provides little token\-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks\. Although policy distillation can offer dense guidance, external teacher based methods introduce substantial computational overhead, while answer conditioned tuning methods may expose answer\-level information and induce shortcut\-like generation behavior\. To address these limitations, we propose PTD\-PO, a Privileged Tutoring Distillation Policy Optimization framework for RLVR that provides dense guidance without exposing the answer to the student policy\. Specifically, PTD\-PO constructs structured privileged hints from spatial attention guidance and intermediate textual reasoning steps, and uses them through in\-context learning to produce step\-wise token\-distribution supervision\. The student is still optimized under the original answer\-free context, and its failed rollouts are aligned with the hint\-augmented reference model at the token\-distribution level\. To further stabilize distillation under the distribution shift between guided and unguided contexts, we introduce a Top\-K Jensen\-Shannon divergence objective that focuses alignment on informative token probabilities while reducing memory overhead\. Experiments on LVLMs ranging from 2B to 8B parameters show that PTD\-PO consistently outperforms RLVR and distillation baselines, mitigates entropy collapse, and improves complex multimodal reasoning performance\. Project page:[https://github\.com/XszNeverSleep/PTD\-PO](https://github.com/XszNeverSleep/PTD-PO)\.

11footnotetext:Equal contribution\.22footnotetext:Work done during the internship at Xiaomi\.33footnotetext:Primary corresponding author\.44footnotetext:Co\-corresponding author\.## 1Introduction

Recent advances in Large Vision\-Language Models \(LVLMs\) have enabled strong performance on complex multimodal reasoning tasks, where models must ground visual evidence, perform multi\-step inference, and produce verifiable answers\(Liuet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib1); Yueet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib2); Luet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib3); Baiet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib4); Daiet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib35); Liet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib34)\)\. Reinforcement Learning with Verifiable Rewards \(RLVR\) has become a promising post\-training paradigm for further eliciting such abilities, as it replaces costly human preference annotation with outcome\-level feedback\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.07000#bib.bib5); Shaoet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib6); Guoet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib7); Teamet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib8); Askellet al\.,[2021](https://arxiv.org/html/2606.07000#bib.bib16)\)\. However, verifiable rewards are usually assigned only to the final answer\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib9); Chenet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib37)\)\. Failed rollouts therefore provide little information about which intermediate visual grounding or reasoning steps are incorrect, making credit assignment difficult and often causing exploration failure in large reasoning spaces\(Prakash and Buvanesh,[2025](https://arxiv.org/html/2606.07000#bib.bib10); Tranet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib38); Renet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib36)\)\.

A natural way to alleviate reward sparsity is to replace or supplement outcome\-level RLVR with dense token\-level guidance\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib15); Guet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib17)\)\. Online Policy Distillation \(OPD\) provides such guidance by aligning student rollouts with a teacher policy during training\. However, it typically requires online inference from an external teacher over student\-generated trajectories, incurring substantial computational overhead, lowering training efficiency, and potentially introducing tokenizer or vocabulary mismatch\(Fuet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib18); Minixhoferet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib19)\)\. A more efficient alternative is contextual self\-distillation, which constructs the teacher from the student model itself under additional conditioning\. Existing methods often use ground\-truth answers or complete solutions as this conditioning signal to obtain stronger supervision\. Yet such answer\-revealing contexts alter the teaching behavior: rather than guiding the model to discover the reasoning path, they may induce answer\-level shortcuts, resulting in overly deterministic trajectories, sharpened token distributions, accelerated entropy collapse, and reduced exploration of alternative reasoning paths\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib20); Hübotteret al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib11); Zhanget al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib22)\)\. Together, these limitations call for a new form of dense supervision that is both efficient, avoiding costly online external teachers, and non\-revealing, providing corrective reasoning guidance without exposing answer\-level information that undermines exploration\.

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PID-GRPO_intro.png)Figure 1:Conceptual illustration of PTD\-PO\. \(a\) PTD\-PO complements sparse\-reward RLVR with answer\-free privileged hints, avoiding the overly deterministic behavior of answer\-revealing distillation\. \(b\) In a failed geometry example, the actor gives a wrong answer from the question\-only context\. The privileged hint guides the reference model to focus on area relationships and equal\-area frames, and to infer the relation amongA​B​C​DABCD,E​F​G​HEFGH, andI​J​K​LIJKLwithout exposing the final area\. The resulting hint\-augmented teacher distribution provides dense supervision for correcting the failed student trajectory\.To address these challenges, we propose Privileged Tutoring Distillation Policy Optimization \(PTD\-PO\)\. As illustrated in Figure[1](https://arxiv.org/html/2606.07000#S1.F1), PTD\-PO uses answer\-free hints as tutoring signals for self\-distillation\. Unlike answer\-revealing self\-distillation, where the teacher is conditioned on the complete solution trace, PTD\-PO lets the reference model use non\-spoiling hints to infer a corrective reasoning direction, while the student still learns under the original answer\-free context\. This design decouples the information used to generate supervision from the information available to the student policy, following the learning\-with\-privileged\-information paradigm\(Vapnik and Vashist,[2009](https://arxiv.org/html/2606.07000#bib.bib23); Sharmanskaet al\.,[2013](https://arxiv.org/html/2606.07000#bib.bib24)\)\. The hints combine visual grounding cues and high\-level reasoning guidance, which are related to explanation\-guided vision\-language learning and step\-wise reasoning distillation\(Selvarajuet al\.,[2019](https://arxiv.org/html/2606.07000#bib.bib25); Hsiehet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib26)\)\. In this way, PTD\-PO turns failed rollouts identified by verifiable rewards into dense learning signals, without encouraging answer\-conditioned shortcuts\. However, asymmetric distillation between hint\-augmented teachers and answer\-free students can be unstable and memory\-intensive\. We therefore introduce a Top\-K Jensen\-Shannon Divergence objective with tail compensation, which stabilizes context\-mismatched distribution alignment while reducing the memory cost of token\-level distillation\.

We evaluate PTD\-PO on LVLMs ranging from 2B to 8B parameters across complex multimodal reasoning benchmarks\. Experiments show that PTD\-PO consistently improves over RLVR and distillation\-based baselines\. Further analyses demonstrate that PTD\-PO maintains higher policy entropy, improves recovery from failed rollouts, and reduces the memory overhead of online distillation\. These results show that PTD\-PO can provide dense reasoning guidance while preserving exploration during RLVR post\-training\.

In summary, our contributions are as follows:

- •Privileged Tutoring for RLVR\.We identify that sparse outcome rewards and solution\-revealing self\-distillation can limit exploration in multimodal RLVR, and introduce an answer\-free privileged tutoring framework that provides dense guidance without exposing solution\-level information\.
- •PTD\-PO\.We propose PTD\-PO, a privileged tutoring distillation framework that routes dense supervision to failed trajectories by conditioning a frozen reference model on spatial and textual hints, while keeping the student policy under the original answer\-free context\. We further introduce a Top\-K JSD objective with tail compensation to enable stable and memory\-efficient asymmetric distillation\.
- •Empirical Effectiveness and Generality\.Experiments show that PTD\-PO consistently improves LVLMs across model scales and multimodal reasoning benchmarks\. We further demonstrate that the proposed PTD module is compatible with different RLVR optimizers, suggesting its general applicability\.

## 2Methodology

### 2\.1Preliminary

#### Group Relative Policy Optimization\.

Given a multimodal questionxx, a policy modelπθ\\pi\_\{\\theta\}generates a group of responses\{yi\}i=1G\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}, whereyi=\(yi,1,…,yi,Ti\)y\_\{i\}=\(y\_\{i,1\},\\ldots,y\_\{i,T\_\{i\}\}\)\. Each response is evaluated by a verifiable rewardrir\_\{i\}\. GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib6)\)estimates the advantage of each response by normalizing rewards within the sampled group:

Ai=ri−mean​\(\{rj\}j=1G\)std​\(\{rj\}j=1G\)\+ϵ\.A\_\{i\}=\\frac\{r\_\{i\}\-\\mathrm\{mean\}\(\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\)\}\{\\mathrm\{std\}\(\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\)\+\\epsilon\}\.\(1\)Letπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}denote the rollout policy and define the token\-level importance ratio as:

ρi,t​\(θ\)=πθ​\(yi,t∣x,yi,<t\)πθold​\(yi,t∣x,yi,<t\)\.\\rho\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\.\(2\)The GRPO objective can be written as:

𝒥GRPO​\(θ\)=𝔼x,\{yi\}i=1G​\[1G​∑i=1G1Ti​∑t=1Ti\(min⁡\(ρi,t​\(θ\)​Ai,clip​\(ρi,t​\(θ\),1−ϵc,1\+ϵc\)​Ai\)−β​DKL\)\],\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{x,\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\left\(\\min\\left\(\\rho\_\{i,t\}\(\\theta\)A\_\{i\},\\mathrm\{clip\}\(\\rho\_\{i,t\}\(\\theta\),1\-\\epsilon\_\{c\},1\+\\epsilon\_\{c\}\)A\_\{i\}\\right\)\-\\beta D\_\{\\mathrm\{KL\}\}\\right\)\\right\],\(3\)wheresi,t=\(x,yi,<t\)s\_\{i,t\}=\(x,y\_\{i,<t\}\)denotes the generation state\. GRPO removes the need for a learned value function by using group\-relative rewards, but its learning signal is still derived from outcome\-level verification\. Therefore, when all sampled responses fail, the policy may receive weak or even uninformative guidance for improving intermediate reasoning steps\.

#### Policy Distillation and Self\-Distillation\.

Online policy distillation\(Yeet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib21)\)provides dense token\-level supervision by aligning the student policyπθ\\pi\_\{\\theta\}with a teacher policyπT\\pi\_\{T\}\. Given an inputxxand a rolloutyiy\_\{i\}sampled from the student, we denote the generation state at stepttassi,t=\(x,yi,<t\)s\_\{i,t\}=\(x,y\_\{i,<t\}\)\. The standard online policy distillation objective is:

ℒOPD\(θ\)=𝔼x,\{yi\}\[1G∑i=1G1Ti∑t=1TiDKL\(πT\(⋅∣si,t\)∥πθ\(⋅∣si,t\)\)\]\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{x,\\\{y\_\{i\}\\\}\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}D\_\{\\mathrm\{KL\}\}\\left\(\\pi\_\{T\}\(\\cdot\\mid s\_\{i,t\}\)\\\|\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{i,t\}\)\\right\)\\right\]\.\(4\)Compared with RLVR, which only provides scalar outcome\-level rewards, this objective specifies a token\-level target distribution at each reasoning step and thus offers more informative optimization signals\. However, whenπT\\pi\_\{T\}is instantiated as a larger external model, online distillation requires additional teacher inference and may suffer from tokenizer or vocabulary mismatch\.

Self\-distillation\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib20); Agarwalet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib15); Hübotteret al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib11)\)avoids external teachers by derivingπT\\pi\_\{T\}from the actor model itself\. Existing methods mainly differ in how the self\-teacher is constructed\. For example, SDPO uses an exponential moving average of the actor parameters as the teacher, while OPSD uses the original frozen reference model as the teacher weights\. Both can be viewed as special cases of the above policy distillation objective with different self\-teacher choices\. Although such designs reduce the dependence on external teachers, their teachers are still conditioned only on the original problem context\. As a result, they may provide limited corrective supervision when the actor repeatedly fails on difficult multimodal reasoning trajectories\.

### 2\.2Motivating Analysis

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/fig2_group_reward_stats.png)Figure 2:Group\-level reward signal statistics during GRPO training\. \(a\) Composition of sampled groups categorized by the success countm=∑i=1Grim=\\sum\_\{i=1\}^\{G\}r\_\{i\}\. \(b\) Ratio curves of all\-fail\(m=0\)\(m=0\), informative\(0<m<G\)\(0<m<G\), and all\-pass\(m=G\)\(m=G\)groups over training\. Informative groups provide discriminative relative rewards, but their proportion remains limited throughout training\.#### Analysis 1: GRPO receives limited informative reward groups\.

We first examine how much useful supervision outcome\-level rewards provide during GRPO training\. For each questionxx, the policy samples a group ofGGresponses with verifiable rewards\{ri\}i=1G\\\{r\_\{i\}\\\}\_\{i=1\}^\{G\}\. We define the group success count as

m​\(x\)=∑i=1Gri\.m\(x\)=\\sum\_\{i=1\}^\{G\}r\_\{i\}\.\(5\)Based onm​\(x\)m\(x\), each group is categorized as all\-fail\(m=0\)\(m=0\), informative\(0<m<G\)\(0<m<G\), or all\-pass\(m=G\)\(m=G\)\. Among them, informative groups are the most useful for group\-relative optimization, since they contain both successful and failed responses and thus provide discriminative reward differences for advantage estimation\. In contrast, all\-fail and all\-pass groups have nearly uniform rewards, resulting in weak or uninformative relative signals\.

Figure[2](https://arxiv.org/html/2606.07000#S2.F2)shows the group\-level reward statistics during GRPO training on the 4B model\. As shown in Figure[2](https://arxiv.org/html/2606.07000#S2.F2)\(a\), all\-fail groups dominate the early training stage, indicating that sparse outcome rewards often fail to provide successful trajectories for hard problems\. Although the all\-fail ratio decreases over training, Figure[2](https://arxiv.org/html/2606.07000#S2.F2)\(b\) shows that informative groups remain limited, stabilizing at roughly one third of all sampled groups, while all\-pass groups gradually become dominant\. These results suggest that GRPO frequently operates on low\-information groups, motivating an auxiliary dense guidance signal to correct failed reasoning trajectories\.

![Refer to caption](https://arxiv.org/html/2606.07000v1/x1.png)

Figure 3:Context\-conditioned self\-teacher statistics\. \(a\) Token\-levelΔ\\DeltaNLL, \(b\) full\-distribution KL divergence, and \(c\) response length show that complete solution traces induce the largest distribution shift and shortcut\-like generation behavior, while hints provide a moderate guidance signal\.
#### Analysis 2: Solution\-revealing contexts induce distribution shift and shortcut\-like teacher behavior\.

We next examine how different types of additional information change the self\-teacher’s token distribution\. As shown in Figure[3](https://arxiv.org/html/2606.07000#S2.F3), simply providing the final GT answer has only a marginal effect on the teacher distribution, with both token\-levelΔ\\DeltaNLL and full\-distribution KL remaining close to the question\-only baseline\. In contrast, conditioning the teacher on both the complete CoT solution and the GT answer leads to a much larger distribution shift: it substantially increasesΔ\\DeltaNLL on student\-generated tokens and yields the highest KL divergence from the question\-only teacher\. Moreover, the response length under the CoT\-and\-GT condition drops sharply compared with the question\-only setting, suggesting that the teacher no longer needs to perform an autonomous reasoning process and instead tends to follow a shortcut\-like solution path\. By comparison, the hint\-conditioned teacher introduces a moderate distribution shift: it changes the teacher’s prediction enough to provide useful corrective information, but avoids the severe behavioral change caused by exposing the complete solution trace\. These results motivate using privileged hints rather than solution\-revealing trajectories as the supervision source for self\-distillation\.

### 2\.3PTD\-PO: Privileged Tutoring Distillation Policy Optimization

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PID-GRPO_Arch.png)Figure 4:Overview of PTD\-PO, which combines RLVR with privileged\-tutoring distillation from the hint\-augmented reference model on failed trajectories\.As shown in Figure[4](https://arxiv.org/html/2606.07000#S2.F4), PTD\-PO complements RLVR with privileged tutoring distillation\. During rollout, the actor still explores under the original question\-only context and is optimized by GRPO with verifiable rewards\. When a trajectory fails, PTD\-PO elicits the hint\-augmented reference model as a privileged tutor, which leverages spatial and reasoning hints to provide token\-level guidance on how the failed reasoning path should be corrected\. In this way, PTD\-PO preserves the exploration nature of RLVR while turning failed rollouts into dense supervision for learning reasoning strategies rather than answer conditioned shortcuts\.

#### Structured Privileged Hint Construction\.

For each training example, letxxdenote the original multimodal question anda⋆a^\{\\star\}denote the final verifiable answer used for reward computation\. We construct privileged hints by prompting a strong model to reverse engineer an answer\-consistent but non\-revealing guidance signal from\(x,a⋆\)\(x,a^\{\\star\}\), rather than generating a complete solution trace\. Importantly, the answer is used only for offline hint construction and is never exposed to the student policy\. To ensure that the generated hint is structured rather than arbitrary, the prompt is designed with three key constraints\. First, it enforces consistency with a valid solution direction, so that the hint provides useful tutoring guidance for the target problem\. Second, it follows a zero\-spoiler rule, forbidding the final answer, exact intermediate numerical results, or answer\-identifying object names, thereby preventing the hint from degenerating into answer\-conditioned supervision\. Third, it requires the model to filter visual distractors and common reasoning traps, which is crucial for multimodal tasks where failures often arise from irrelevant visual evidence or misleading textual cues\.

This structured construction produces two complementary types of guidance: spatial guidance, which highlights relevant regions, objects, relations, or distractors to ignore, and textual reasoning guidance, which provides high\-level intermediate reasoning directions without revealing the final answer\. These hints are used to elicit a hint\-augmented reference model through in\-context learning, allowing it to provide reasoning\-path distillation signals rather than answer imitation\. The ablation in Figure[5](https://arxiv.org/html/2606.07000#S3.F5)further supports the necessity of structured hint design\. More details of the hint construction protocol are provided in Appendix[C\.1](https://arxiv.org/html/2606.07000#A3.SS1)\.

#### Failure\-Routed Privileged Tutoring Distillation\.

Given a multimodal questionxx, the rollout policy samples a group ofGGresponsesyii=1G\{y\_\{i\}\}\_\{i=1\}^\{G\}under the original answer\-free context,

yi∼πθold\(⋅∣x\),ri=R\(x,yi\)∈0,1,y\_\{i\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid x\),\\qquad r\_\{i\}=R\(x,y\_\{i\}\)\\in\{0,1\},\(6\)whereR​\(⋅\)R\(\\cdot\)is the verifiable reward function\. We characterize the difficulty and training informativeness of the sampled group by its empirical accuracy,

r¯​\(x\)=1G​∑i=1Gri\.\\bar\{r\}\(x\)=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}r\_\{i\}\.\(7\)Whenr¯​\(x\)\\bar\{r\}\(x\)is small, most sampled trajectories fail to reach the correct answer, and the outcome\-level reward provides limited information about how their intermediate visual grounding or reasoning steps should be corrected\. Therefore, instead of applying privileged tutoring distillation uniformly to all rollouts, we route PTD supervision to the incorrect responses in the sampled group\. Specifically, with the activation thresholdτptd\\tau\_\{\\mathrm\{ptd\}\}, the set of token positions selected for PTD is:

ℱ=\{\(i,t\)\|r¯​\(x\)<τptd,ri=0,mi,t=1\},\\mathcal\{F\}=\\left\\\{\(i,t\)\\;\\middle\|\\;\\bar\{r\}\(x\)<\\tau\_\{\\mathrm\{ptd\}\},\\;r\_\{i\}=0,\\;m\_\{i,t\}=1\\right\\\},\(8\)wheremi,tm\_\{i,t\}is the response mask\. In our main setting, we useτptd=1\.0\\tau\_\{\\mathrm\{ptd\}\}=1\.0, which means that PTD is applied to all failed trajectories, while trajectories in fully correct groups are optimized only by the standard RLVR objective\.

For each questionxx, we construct an answer\-free privileged hinthhand form the hint\-augmented teacher contextxh=\[x;h\]x^\{h\}=\[x;h\]\. The privileged self\-teacher is the frozen reference model conditioned onxhx^\{h\}:

qi,th=πref\(⋅∣xh,yi,<t\),q^\{h\}\_\{i,t\}=\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x^\{h\},y\_\{i,<t\}\),\(9\)while the student policy is still evaluated under the original answer\-free context:

pi,t=πθ\(⋅∣x,yi,<t\)\.p\_\{i,t\}=\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{i,<t\}\)\.\(10\)This asymmetric construction decouples the information used to produce dense supervision from the information available to the student policy\. The teacher can use privileged spatial and textual hints to infer a corrective reasoning direction, whereas the student must learn to reproduce the improved token distribution without directly observing the hint\.

The privileged tutoring distillation loss is defined over the selected failed positions:

ℒPTD=∑\(i,t\)∈ℱDPTD​\(qi,th,pi,t\)\|ℱ\|\+ϵ\.\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}=\\frac\{\\sum\_\{\(i,t\)\\in\\mathcal\{F\}\}D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\)\}\{\|\\mathcal\{F\}\|\+\\epsilon\}\.\(11\)The final training objective combines the GRPO loss with the PTD regularizer:

ℒPTD​\-​PO=−JGRPO​\(θ\)\+λPTD​ℒPTD,\\mathcal\{L\}\_\{\\mathrm\{PTD\\text\{\-\}PO\}\}=\-J\_\{\\mathrm\{GRPO\}\}\(\\theta\)\+\\lambda\_\{\\mathrm\{PTD\}\}\\mathcal\{L\}\_\{\\mathrm\{PTD\}\},\(12\)whereλPTD\\lambda\_\{\\mathrm\{PTD\}\}controls the strength of privileged distillation\.

#### Asymmetric\-Context Top\-KKJSD with Tail Compensation\.

Since the teacher and student distributions are conditioned on different contexts, directly matching them with a directional KL divergence may produce overly strong and unstable gradients\. We therefore use a Jensen–Shannon divergence objective under an asymmetric\-context distillation setting\. For two distributionsqqandpp, the full JSD is:

DJSD​\(q,p\)=12​DKL​\(q∥m\)\+12​DKL​\(p∥m\),m=12​\(q\+p\)\.D\_\{\\mathrm\{JSD\}\}\(q,p\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(q\\\|m\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\\\|m\),\\qquad m=\\frac\{1\}\{2\}\(q\+p\)\.\(13\)However, computing the full\-vocabulary JSD requires storing distributions over the entire vocabulary\. To reduce the memory cost, we approximate it with a Top\-KKsupport and useK=100K=100throughout our experiments\. For each selected token position\(i,t\)\(i,t\), we construct the compact support as the union of the teacher and student Top\-KKtokens:

𝒮i,t=Top​\-​K⁡\(pi,t\)∪Top​\-​K⁡\(qi,th\)\.\\mathcal\{S\}\_\{i,t\}=\\operatorname\{Top\\text\{\-\}K\}\(p\_\{i,t\}\)\\cup\\operatorname\{Top\\text\{\-\}K\}\(q^\{h\}\_\{i,t\}\)\.\(14\)The probability mass outside this support is aggregated into a single tail bucket:

pi,ttail=1−∑v∈𝒮i,tpi,t​\(v\),qi,th,tail=1−∑v∈𝒮i,tqi,th​\(v\)\.p^\{\\mathrm\{tail\}\}\_\{i,t\}=1\-\\sum\_\{v\\in\\mathcal\{S\}\_\{i,t\}\}p\_\{i,t\}\(v\),\\qquad q^\{h,\\mathrm\{tail\}\}\_\{i,t\}=1\-\\sum\_\{v\\in\\mathcal\{S\}\_\{i,t\}\}q^\{h\}\_\{i,t\}\(v\)\.\(15\)This gives the compressed distributions:

p~i,t=\(pi,t​\(v\)v∈𝒮i,t,pi,ttail\),q~i,th=\(qi,th​\(v\)v∈𝒮i,t,qi,th,tail\)\.\\widetilde\{p\}\_\{i,t\}=\\left\(\{p\_\{i,t\}\(v\)\}\_\{v\\in\\mathcal\{S\}\_\{i,t\}\},p^\{\\mathrm\{tail\}\}\_\{i,t\}\\right\),\\qquad\\widetilde\{q\}^\{h\}\_\{i,t\}=\\left\(\{q^\{h\}\_\{i,t\}\(v\)\}\_\{v\\in\\mathcal\{S\}\_\{i,t\}\},q^\{h,\\mathrm\{tail\}\}\_\{i,t\}\\right\)\.\(16\)The PTD divergence is then computed as:

DPTD​\(qi,th,pi,t\)=DJSD​\(q~i,th,p~i,t\)\.D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\)=D\_\{\\mathrm\{JSD\}\}\\left\(\\widetilde\{q\}^\{h\}\_\{i,t\},\\widetilde\{p\}\_\{i,t\}\\right\)\.\(17\)The tail bucket preserves the total probability mass outside the Top\-KKsupport and avoids the probability loss caused by hard truncation\. As a result, the memory cost of token\-level distribution alignment is reduced from𝒪​\(B​T​V\)\\mathcal\{O\}\(BTV\)to𝒪​\(B​T​K\)\\mathcal\{O\}\(BTK\)withK=100K=100, while retaining the dominant distributional signal needed for stable privileged tutoring distillation\.

## 3Experiments

### 3\.1Experimental Setup

#### Training Data\.

We use ViRL39K as the training corpus for multimodal reasoning post\-training\. ViRL39K contains 38,870 verifiable vision\-language question\-answering examples and covers diverse reasoning scenarios, including mathematics, science, charts, diagrams, tables, documents, and spatial reasoning\(Luet al\.,[2023](https://arxiv.org/html/2606.07000#bib.bib3); Yueet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib2); Xiaoet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib28); Zhanget al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib29); Johnsonet al\.,[2017](https://arxiv.org/html/2606.07000#bib.bib31); Luet al\.,[2021](https://arxiv.org/html/2606.07000#bib.bib30)\)\. To support different paradigms, we further construct two auxiliary supervision sources for each example: solution trajectories with chain\-of\-thought rationales and privileged hints\. Specifically, we use Qwen3\-VL\-235B\-A22B\-Thinking and Gemini\-3\.0\-Pro as external teacher models to generate CoT\-style solutions and hint annotations\. Detailed data construction prompts, filtering rules, and quality\-control procedures are provided in Appendix[C](https://arxiv.org/html/2606.07000#A3)\.

#### Evaluation Benchmarks\.

We evaluate our method on the PAPO\(Wanget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib27)\)multimodal reasoning benchmark suite, which adapts a collection of vision\-language reasoning datasets into a unified verifiable evaluation protocol\. The benchmark covers diverse tasks such as mathematical reasoning, geometry, logic reasoning, visual question answering, and knowledge\-intensive multimodal understanding\. Following the original setup, the evaluation set is organized into two categories: general multimodal reasoning and vision\-dependent multimodal reasoning\. For datasets such as MathVista and MathVerse, free\-form answer instances are filtered out to enable rule\-based answer verification and avoid relying on LLM\-as\-a\-judge\. We report performance on each benchmark as well as the averaged results over the general, vision\-dependent, and overall splits\.

#### Implementation Details\.

For the main experiments, all model sizes are trained and evaluated with a maximum response length of 4096 tokens\. For RL\-based methods, including GRPO, self\-distillation baselines, and PTD\-PO, we use a rollout number of 8 during training\. To reduce computational cost, ablation studies are conducted under a smaller but matched setting, with rollout number 5 and maximum response length 2048 tokens\. More implementation details, hyperparameters, reward rules, and training configurations are provided in Appendix[B](https://arxiv.org/html/2606.07000#A2)\.

### 3\.2Main Results

Table[1](https://arxiv.org/html/2606.07000#S3.T1)presents the main comparison across Qwen3\-VL\-Thinking models of different scales\. PTD\-PO consistently achieves the best overall performance and clearly outperforms the standard GRPO baseline on both general multimodal reasoning and vision\-dependent reasoning tasks, showing that privileged tutoring distillation provides effective dense guidance beyond sparse outcome\-level rewards\. Compared with HDPO\(Ding,[2026](https://arxiv.org/html/2606.07000#bib.bib52)\), which relies on ground truth answer conditioned self\-distillation, PTD\-PO obtains stronger results without exposing the final answer, suggesting that answer\-free privileged hints are more suitable for guiding failed multimodal reasoning trajectories while preserving exploration\. PTD\-PO also surpasses PAPO\(Wanget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib27)\), a perception\-aware policy optimization method that improves multimodal RLVR through visual grounding signals, further demonstrating that our privileged\-information guidance offers a more general and effective way to enhance multimodal reasoning\.

Table 1:Comparison of different training methods\.
### 3\.3Ablation Study

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/ablation_recovery.png)Figure 5:Effect of different teacher contexts on Qwen3\-VL\-4B\-Thinking\. \(a\) Recovery distribution on hard questions\. \(b\) pass@nnon randomly sampled examples\.#### Effect of privileged hints\.

We examine whether privileged hints provide useful guidance for the self\-teacher without turning distillation into direct solution imitation\. Using Qwen3\-VL\-4B\-Thinking, we sample 16 rollouts for each of 384 examples under different teacher contexts\. Figure[5](https://arxiv.org/html/2606.07000#S3.F5)\(a\) reports the recovery distribution on hard questions where the question\-only teacher fails in all rollouts\. Compared with GT\-answer prompting, privileged hints recover a larger fraction of hard cases into partial success, indicating that hints provide effective corrective information beyond the original question\. Figure[5](https://arxiv.org/html/2606.07000#S3.F5)\(b\) further shows that hints consistently improve pass@nnover the question\-only setting and achieve performance close to the GT\-answer condition on the full evaluation subset\. By contrast, CoT\+GT obtains the highest pass@nnbut also produces a large saturated all\-pass ratio on hard cases, suggesting shortcut\-like behavior caused by exposing complete solution traces\. These results support our design choice of using hints as privileged guidance in PTD\-PO\.

Table 2:Ablation Study for PTD\-PO\.
#### Effect of structured hint design\.

We further examine whether PTD\-PO benefits from our structured hint construction rather than merely from adding extra answer\-free context\. Thew/o Structured Hint Designvariant removes the key visual\-grounding and trap\-filtering constraints, and only asks the teacher model to provide a general answer\-free reasoning hint\. Specifically, it does not explicitly identify relevant visual regions, key objects, spatial relations, or distractors to ignore\. As shown in Table[2](https://arxiv.org/html/2606.07000#S3.T2), removing these structured constraints yields smaller gains and can even hurt performance on the stronger 8B model\. These results indicate that the effectiveness of PTD\-PO relies on carefully constructed privileged hints that provide grounded, non\-spoiling reasoning guidance rather than arbitrary answer\-free hints\.

#### Effect of PTD activation threshold\.

We further ablate the trajectory selection strategy of PTD\-PO, whereτptd\\tau\_\{\\mathrm\{ptd\}\}denotes the group\-level activation threshold and “All Trajectories” applies PTD to all rollouts regardless of correctness\. As shown in Table[2](https://arxiv.org/html/2606.07000#S3.T2), applying PTD to all failed trajectories withτptd=1\.0\\tau\_\{\\mathrm\{ptd\}\}=1\.0consistently achieves the best overall performance across the 2B, 4B, and 8B models\. Smaller thresholds activate PTD only on more severely failed groups, which reduces the amount of corrective supervision and leads to weaker gains\. In contrast, applying PTD to all trajectories is also less effective, especially on larger models, because correct rollouts already provide reliable reward\-driven learning signals and forcing them to match the hint\-augmented teacher may over\-regularize the policy\. These results support our design choice: PTD should focus on failed trajectories, where verifiable rewards provide limited guidance and privileged tutor distillation is most beneficial\.

### 3\.4Compatibility with Other RLVR Optimizers

PTD is designed as an auxiliary distillation module and is therefore decoupled from the specific RLVR optimizer\. Beyond GRPO, it can be directly combined with other policy optimization objectives, such as DAPO, GSPO\(Yuet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib32); Zhenget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib33)\)\. Additional experiments in Appendix[E\.1](https://arxiv.org/html/2606.07000#A5.SS1)verify that the proposed privileged tutoring distillation remains effective across different RLVR methods\.

## 4Conclusion

In this paper, we presented PTD\-PO, a privileged tutoring distillation framework for RLVR based LVLM post\-training\. PTD\-PO provides dense token\-level guidance for failed rollouts by conditioning a self\-teacher on non\-answer\-revealing spatial and textual hints, while keeping the student under the original answer\-free context\. To reduce memory cost, we introduced a Top\-K JSD objective with tail compensation\. Experiments on LVLMs from 2B to 8B show that PTD\-PO consistently improves over RLVR and distillation baselines across general and vision\-dependent multimodal reasoning tasks\. Further analyses demonstrate that PTD\-PO helps recover failed trajectories, preserves exploration, and mitigates entropy collapse, suggesting that privileged\-information self\-distillation is an effective way to complement verifiable reward optimization\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.07000#S2.SS1.SSS0.Px2.p2.1)\.
- A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma,et al\.\(2021\)A general language assistant as a laboratory for alignment\.arXiv preprint arXiv:2112\.00861\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.Advances in Neural Information Processing Systems37,pp\. 27056–27087\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. Hoi \(2023\)Instructblip: towards general\-purpose vision\-language models with instruction tuning\.Advances in neural information processing systems36,pp\. 49250–49267\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- Y\. Deng, H\. Bansal, F\. Yin, N\. Peng, W\. Wang, and K\. Chang \(2025\)Openvlthinker: an early exploration to complex vision\-language reasoning via iterative self\-improvement\.arXiv e\-prints,pp\. arXiv–2503\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- K\. Ding \(2026\)HDPO: hybrid distillation policy optimization via privileged self\-distillation\.arXiv preprint arXiv:2603\.23871\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2606.07000#S3.SS2.p1.1)\.
- Y\. Fu, H\. Huang, K\. Jiang, Y\. Zhu, and D\. Zhao \(2026\)Revisiting on\-policy distillation: empirical failure modes and simple fixes\.arXiv preprint arXiv:2603\.25562\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p2.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)Minillm: knowledge distillation of large language models\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 8003–8017\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p3.1)\.
- S\. Huang, X\. Qu, Y\. Li, Y\. Luo, Z\. He, D\. Liu, and Y\. Cheng \(2025a\)Spotlight on token perception for multimodal reinforcement learning\.arXiv preprint arXiv:2510\.09285\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- W\. Huang, B\. Jia, Z\. Zhai, S\. Cao, Z\. Ye, F\. Zhao, Z\. Xu, X\. Tang, Y\. Hu, and S\. Lin \(2025b\)Vision\-r1: incentivizing reasoning capability in multimodal large language models\.arXiv preprint arXiv:2503\.06749\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.07000#S2.SS1.SSS0.Px2.p2.1)\.
- J\. Johnson, B\. Hariharan, L\. Van Der Maaten, L\. Fei\-Fei, C\. Lawrence Zitnick, and R\. Girshick \(2017\)Clevr: a diagnostic dataset for compositional language and elementary visual reasoning\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2901–2910\.Cited by:[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- G\. Li, T\. Yang, J\. Fang, M\. Song, M\. Zheng, H\. Guo, D\. Zhang, J\. Wang, and T\. Chua \(2026a\)Unifying group\-relative and self\-distillation policy optimization via sample routing\.arXiv preprint arXiv:2604\.02288\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational conference on machine learning,pp\. 19730–19742\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu,et al\.\(2026b\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- Z\. Lin, M\. Lin, Y\. Xie, and R\. C\. Ji \(2025\)Accelerating the training of group relative policy optimization\-based reasoning models\.arXiv preprint arXiv:2503\.2234237\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2023\)Mathvista: evaluating mathematical reasoning of foundation models in visual contexts\.arXiv preprint arXiv:2310\.02255\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- P\. Lu, R\. Gong, S\. Jiang, L\. Qiu, S\. Huang, X\. Liang, and S\. Zhu \(2021\)Inter\-gps: interpretable geometry problem solving with formal language and symbolic reasoning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 6774–6786\.Cited by:[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- F\. Meng, L\. Du, Z\. Liu, Z\. Zhou, Q\. Lu, D\. Fu, B\. Shi, W\. Wang, J\. He, K\. Zhang,et al\.\(2025\)Mm\-eureka: exploring visual aha moment with rule\-based large\-scale reinforcement learning\.arXiv preprint arXiv:2503\.07365\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- B\. Minixhofer, I\. Vulić, and E\. M\. Ponti \(2025\)Cross\-tokenizer distillation via approximate likelihood matching\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- J\. Prakash and A\. Buvanesh \(2025\)What can you do when you have zero rewards during rl?\.arXiv preprint arXiv:2510\.03971\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- Y\. Ren, H\. Zhang, L\. Xiao, X\. Zhang, J\. Huang, J\. Qiu, B\. Yu, Q\. Chen, and L\. Liu \(2026\)Recycling failures: salvaging exploration in rlvr via fine\-grained off\-policy guidance\.arXiv preprint arXiv:2602\.24110\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- R\. R\. Selvaraju, S\. Lee, Y\. Shen, H\. Jin, S\. Ghosh, L\. Heck, D\. Batra, and D\. Parikh \(2019\)Taking a hint: leveraging explanations to make vision and language models more grounded\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 2591–2600\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p3.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.07000#S2.SS1.SSS0.Px1.p1.5)\.
- V\. Sharmanska, N\. Quadrianto, and C\. H\. Lampert \(2013\)Learning to rank using privileged information\.InProceedings of the IEEE international conference on computer vision,pp\. 825–832\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p3.1)\.
- H\. Tan, Y\. Ji, X\. Hao, M\. Lin, P\. Wang, Z\. Wang, and S\. Zhang \(2025\)Reason\-rft: reinforcement fine\-tuning for visual reasoning\.arXiv e\-prints,pp\. arXiv–2503\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.\(2025\)Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- H\. Tran, Z\. Yao, and H\. Yu \(2025\)Exploiting tree structure for credit assignment in rl training of llms\.arXiv preprint arXiv:2509\.18314\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1)\.
- V\. Vapnik and A\. Vashist \(2009\)A new learning paradigm: learning using privileged information\.Neural networks22\(5\-6\),pp\. 544–557\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p3.1)\.
- Z\. Wang, X\. Guo, S\. Stoica, H\. Xu, H\. Wang, H\. Ha, X\. Chen, Y\. Chen, M\. Yan, F\. Huang,et al\.\(2025\)Perception\-aware policy optimization for multimodal reasoning\.arXiv preprint arXiv:2507\.06448\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2606.07000#S3.SS2.p1.1)\.
- Y\. Xiao, E\. Sun, T\. Liu, and W\. Wang \(2024\)Logicvista: multimodal llm logical reasoning benchmark in visual contexts\.arXiv preprint arXiv:2407\.04973\.Cited by:[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- S\. Yang, Y\. Niu, Y\. Liu, Y\. Ye, B\. Lin, and L\. Yuan \(2026\)Look\-back: implicit visual re\-focusing in mllm reasoning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 11694–11702\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- T\. Ye, L\. Dong, Z\. Chi, X\. Wu, S\. Huang, and F\. Wei \(2025\)Black\-box on\-policy distillation of large language models\.arXiv preprint arXiv:2511\.10643\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1)\.
- T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei \(2026\)On\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2606.07000#S2.SS1.SSS0.Px2.p1.6)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2026\)Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2606.07000#S3.SS4.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9556–9567\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- R\. Zhang, D\. Jiang, Y\. Zhang, H\. Lin, Z\. Guo, P\. Qiu, A\. Zhou, P\. Lu, K\. Chang, Y\. Qiao,et al\.\(2024\)Mathverse: does your multi\-modal llm truly see the diagrams in visual math problems?\.InEuropean Conference on Computer Vision,pp\. 169–186\.Cited by:[§3\.1](https://arxiv.org/html/2606.07000#S3.SS1.SSS0.Px1.p1.1)\.
- W\. Zhang, J\. Yan, J\. Huang, Y\. Shen, S\. Shi, P\. Chen, N\. Wang, Z\. Liu, K\. Wang, and S\. Lian \(2026\)HEAL: hindsight entropy\-assisted learning for reasoning distillation\.arXiv preprint arXiv:2603\.10359\.Cited by:[§1](https://arxiv.org/html/2606.07000#S1.p2.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.07000#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.07000#S2.SS1.SSS0.Px2.p2.1)\.
- Y\. Zhao, Y\. Liu, J\. Liu, J\. Chen, X\. Wu, Y\. Hao, T\. Lv, S\. Huang, L\. Cui, Q\. Ye,et al\.\(2025\)Geometric\-mean policy optimization\.arXiv preprint arXiv:2507\.20673\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2606.07000#S3.SS4.p1.1)\.
- \[49\]H\. Zhou, X\. Li, R\. Wang, M\. Cheng, T\. Zhou, and C\. HsiehR1\-zero’s “aha moment” in visual reasoning on a 2b non\-sft model, 2025\.URL https://arxiv\. org/abs/2503\.051329,pp\. 10–16\.Cited by:[Appendix A](https://arxiv.org/html/2606.07000#A1.SS0.SSS0.Px1.p1.1)\.

## Appendix

Appendix Contents

A Related Work[A](https://arxiv.org/html/2606.07000#A1)

B More Implementation Details[B](https://arxiv.org/html/2606.07000#A2)

C Data Construction and Quality Control[C](https://arxiv.org/html/2606.07000#A3)

C\.1 Privileged Hints Construction\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C\.1](https://arxiv.org/html/2606.07000#A3.SS1)

C\.2 Chain\-of\-Thought Answer Construction\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C\.2](https://arxiv.org/html/2606.07000#A3.SS2)

D Proofs for Theoretical Analysis[D](https://arxiv.org/html/2606.07000#A4)

D\.1 A Unified View of GRPO and Token\-Level Distillation\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.1](https://arxiv.org/html/2606.07000#A4.SS1)

D\.2 Why Jensen\-Shannon Divergence Instead of Directional KL\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.2](https://arxiv.org/html/2606.07000#A4.SS2)

D\.3 Approximation Property of Top\-K JSD with Tail Compensation\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.3](https://arxiv.org/html/2606.07000#A4.SS3)

D\.4 Optimization Rationale for Applying PTD to Failed Trajectories\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.4](https://arxiv.org/html/2606.07000#A4.SS4)

E Additional Experiments and Analysis[E](https://arxiv.org/html/2606.07000#A5)

E\.1 Compatibility With Other RLVR Optimizers\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E\.1](https://arxiv.org/html/2606.07000#A5.SS1)

E\.2 Sensitivity Analysis on the Top\-KKSupport Size\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E\.2](https://arxiv.org/html/2606.07000#A5.SS2)

F Case Study[F](https://arxiv.org/html/2606.07000#A6)

G Limitations and Future Work[G](https://arxiv.org/html/2606.07000#A7)

## Appendix ARelated Work

#### RLVR for Multimodal Reasoning\.

Reinforcement Learning with Verifiable Rewards \(RLVR\) has emerged as a scalable post\-training paradigm for reasoning models, replacing learned preference rewards in RLHF\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.07000#bib.bib39); Ouyanget al\.,[2022](https://arxiv.org/html/2606.07000#bib.bib5); Askellet al\.,[2021](https://arxiv.org/html/2606.07000#bib.bib16)\)with automatically checkable outcome signals\. In text\-only domains, DeepSeekMath, DeepSeek\-R1, Kimi k1\.5, and related rule\-based RL methods show that verifiable rewards can elicit mathematical and multi\-step reasoning abilities\(Shaoet al\.,[2024](https://arxiv.org/html/2606.07000#bib.bib6); Guoet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib7); Teamet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib8); Yuet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib32); Zhenget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib33); Zhaoet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib48)\)\. Recent studies extend this paradigm to LVLMs through multimodal cold\-start data, rule\-based rewards, and GRPO\-style optimization, including Vision\-R1, R1\-V/R1\-Zero\-style visual reasoning, R1\-VL, MM\-Eureka, OpenVLThinker, and Reason\-RFT\(Huanget al\.,[2025b](https://arxiv.org/html/2606.07000#bib.bib40);[Zhouet al\.,](https://arxiv.org/html/2606.07000#bib.bib44); Menget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib45); Denget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib46); Tanet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib47)\)\. Beyond improving final\-answer accuracy, another line of work highlights that multimodal RLVR must explicitly account for visual perception and grounding, using visual perception rewards, perception\-aware objectives, token\-level visual dependency, contrastive perception learning, or visual re\-focusing guidance\(Wanget al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib27); Huanget al\.,[2025a](https://arxiv.org/html/2606.07000#bib.bib42); Linet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib43); Yanget al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib41)\)\. These studies suggest that effective multimodal RLVR should optimize both reasoning correctness and visual evidence utilization\. Nevertheless, most existing methods still depend primarily on outcome\-level rewards, which provide weak supervision for failed trajectories and offer limited guidance on which intermediate visual grounding or reasoning steps should be corrected\.

#### On\-Policy and Self\-Distillation for Dense Supervision\.

Another closely related direction is on\-policy distillation \(OPD\), which supervises student\-generated trajectories with dense teacher distributions, combining the distributional relevance of on\-policy learning with token\-level feedback from knowledge distillation\(Baiet al\.,[2025](https://arxiv.org/html/2606.07000#bib.bib4); Yeet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib21);[2025](https://arxiv.org/html/2606.07000#bib.bib49)\)\. Compared with off\-policy distillation on fixed teacher traces, OPD can better expose the teacher to student\-side errors, but it still typically depends on an external stronger teacher and incurs nontrivial rollout\-time supervision cost\(Liet al\.,[2026b](https://arxiv.org/html/2606.07000#bib.bib50)\)\. Recent self\-distillation methods further reduce this dependency by letting the same model serve as both student and teacher under different contexts, e\.g\., conditioning the teacher on ground\-truth solutions, rich feedback, successful sibling rollouts, or other privileged signals to provide dense token\-level correction\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib20); Hübotteret al\.,[2026](https://arxiv.org/html/2606.07000#bib.bib11); Liet al\.,[2026a](https://arxiv.org/html/2606.07000#bib.bib51)\)\. Most relevant to our work, HDPO introduces privileged self\-distillation into RL by using ground\-truth answers to construct a self\-teacher for cliff prompts where standard RL yields vanishing gradients\(Ding,[2026](https://arxiv.org/html/2606.07000#bib.bib52)\)\. However, HDPO focuses on text\-only mathematical reasoning and relies on answer\-revealing privileged information, whereas our method targets multimodal reasoning and uses non\-answer\-revealing spatial and textual hints to correct failed trajectories\.

## Appendix BMore Implementation Details

#### Overall Setup\.

All experiments are implemented with PyTorch and conducted on Qwen3\-VL\-Thinking models with different parameter scales\. For all training methods, we use ViRL39K as the training corpus and train each model for two epochs\. The vision encoder is kept consistent with the original model configuration, and all RL\-based methods are trained under the same data mixture, rollout setting, response length budget, and evaluation protocol to ensure fair comparison\. For PTD\-PO, privileged hints are only used during teacher\-side distribution generation and are never included in the student rollout or evaluation context\.

#### Training Details\.

For all methods, the maximum response length is set to 4096 tokens during both training and evaluation\. SFT is optimized with standard next\-token prediction on the constructed solution trajectories\. For RL\-based methods, each prompt is sampled under the original question\-only context, and the policy generates multiple rollouts for verifiable reward optimization\. The reward is composed of a format reward and an accuracy reward, with weights of0\.10\.1and0\.90\.9, respectively, where the accuracy reward is computed by rule\-based answer matching whenever possible\. GRPO and PTD\-PO share the same rollout configuration and reward design\. OPSD follows its original training settings, with no additional modification except using the same training data, response length budget, and evaluation protocol for fair comparison\. For PTD\-PO, teacher\-side logits are computed only for failed trajectories activated by the PTD criterion and are further compressed by the Top\-KKJSD objective\. All experiments are conducted on 8 or 16 NVIDIA H100 GPUs depending on the model size\.

#### Evaluation Details\.

All experiments follow a train\-inference consistent setting\. During both student rollout training and benchmark evaluation, the policy model is conditioned only on the original question context, without access to privileged hints, spatial guidance, intermediate textual hints, or ground\-truth answers\. We use the same response length budget and answer extraction protocol for all compared methods\. The final answer is extracted from the model response and evaluated by dataset\-specific rule\-based matching whenever possible, and free\-form instances are filtered following the main evaluation protocol to avoid LLM\-as\-a\-judge\.

Table 3:Key hyperparameters for training and evaluation\.

## Appendix CData Construction and Quality Control

### C\.1Privileged Hints Construction

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/hint_pipeline.png)Figure 6:Hint construction pipeline\.![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/hint_system_prompt.png)Figure 7:Hint generation prompt\.As shown in Figure[6](https://arxiv.org/html/2606.07000#A3.F6), privileged hints are constructed offline from the image\-context, the question, and the verified ground\-truth answer\. The external model is used only as a hint generator before PTD training, and is not involved in rollout, distillation, or evaluation\. For each instance, we provide the generator with\(I,q,a⋆\)\(I,q,a^\{\\star\}\)and the system prompt in Figure[7](https://arxiv.org/html/2606.07000#A3.F7), and ask it to reverse\-engineer a concise hint that points to the necessary visual evidence and reasoning route without producing a complete solution trace\.

We mainly use Qwen3\-VL\-235B\-A22B\-Thinking for hint generation\. For particularly difficult cases where this model fails to produce a complete hint within the maximum generation budget of 16,384 tokens, we fall back to Gemini\-3\.0\-Pro to generate the hint\. The same system prompt is used for both generators to ensure a consistent annotation style\. Specifically, the prompt enforces three requirements: the hint should be aligned with the verified reasoning path, should not reveal the final answer or exact intermediate results, and should explicitly suppress irrelevant visual elements or misleading textual cues\. It also encourages concise imperative instructions, with slightly more detailed guidance only for visually ambiguous or logically difficult steps\.

After generation, we apply a lightweight quality check to remove or regenerate invalid hints\. A hint is rejected if it directly contains the final answer, exposes decisive intermediate numerical results, omits the key visual evidence, or degenerates into a full chain\-of\-thought solution\. The remaining hints are used as privileged contexts for the hint\-augmented reference model, enabling it to provide reasoning\-path distillation signals during PTD training\.

### C\.2Chain\-of\-Thought Answer Construction

We use Qwen3\-VL\-235B\-A22B\-Thinking and Gemini\-3\.0\-Pro as teacher models to construct SFT chain\-of\-thought answers for ViRL39K\. For each training instance, the input image, question, and thinking instruction are provided to the teacher, requiring the response to contain a reasoning process enclosed by<think\>and</think\>and a final answer enclosed by\\boxed\{\}\. We sample multiple responses for each question and retain only candidates whose extracted boxed answer matches the ground\-truth answer under the rule\-based accuracy checker\. Among all correct candidates, we keep the shortest one to reduce unnecessary verbosity\. The accepted samples are saved in ShareGPT\-style JSONL format, together with the corresponding image path\.

For questions not solved in the initial generation pass, we apply an additional repair stage\. We first re\-sample missing instances with a larger generation budget\. If they are still unresolved, we optionally use a reverse\-thinking pass, where the teacher is given the ground\-truth answer only during data construction and is asked to back\-construct a coherent reasoning chain\. The stored SFT example still contains only the original question and standard thinking instruction, without exposing the answer hint in the user message\. All repaired responses are again verified by the same rule\-based answer checker, and samples that fail format extraction or answer matching are discarded\.

## Appendix DProofs for Theoretical Analysis

### D\.1A Unified View of GRPO and Token\-Level Distillation

We provide a distributional view of GRPO and token\-level distillation\. Lets=\(x,y<t\)s=\(x,y\_\{<t\}\)denote a token\-generation state and leta∈𝒱a\\in\\mathcal\{V\}denote the next token\. For a fixed statess, consider the following regularized policy improvement problem:

π\+\(⋅\|s\)=argmaxπ\(⋅\|s\)\[𝔼a∼π\(⋅\|s\)A\(s,a\)−τDKL\(π\(⋅\|s\)∥πold\(⋅\|s\)\)\],\\pi^\{\+\}\(\\cdot\|s\)=\\arg\\max\_\{\\pi\(\\cdot\|s\)\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\(s,a\)\-\\tau D\_\{\\mathrm\{KL\}\}\\left\(\\pi\(\\cdot\|s\)\\\|\\pi\_\{\\mathrm\{old\}\}\(\\cdot\|s\)\\right\)\\right\],\(18\)whereA​\(s,a\)A\(s,a\)is the token\-level advantage,πold\\pi\_\{\\mathrm\{old\}\}is the rollout policy, andτ\>0\\tau\>0is a temperature coefficient\.

#### Lemma D\.1\.

The solution of Eq\. equation[18](https://arxiv.org/html/2606.07000#A4.E18)is

π\+​\(a\|s\)=πold​\(a\|s\)​exp⁡\(A​\(s,a\)/τ\)Z​\(s\),Z​\(s\)=∑a′∈𝒱πold​\(a′\|s\)​exp⁡\(A​\(s,a′\)/τ\)\.\\pi^\{\+\}\(a\|s\)=\\frac\{\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\\exp\(A\(s,a\)/\\tau\)\}\{Z\(s\)\},\\quad Z\(s\)=\\sum\_\{a^\{\\prime\}\\in\\mathcal\{V\}\}\\pi\_\{\\mathrm\{old\}\}\(a^\{\\prime\}\|s\)\\exp\(A\(s,a^\{\\prime\}\)/\\tau\)\.\(19\)

#### Proof\.

The Lagrangian of Eq\. equation[18](https://arxiv.org/html/2606.07000#A4.E18)is

ℒ​\(π,η\)=∑a∈𝒱π​\(a\|s\)​A​\(s,a\)−τ​∑a∈𝒱π​\(a\|s\)​log⁡π​\(a\|s\)πold​\(a\|s\)\+η​\(∑a∈𝒱π​\(a\|s\)−1\)\.\\mathcal\{L\}\(\\pi,\\eta\)=\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\(a\|s\)A\(s,a\)\-\\tau\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\(a\|s\)\\log\\frac\{\\pi\(a\|s\)\}\{\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\}\+\\eta\\left\(\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\(a\|s\)\-1\\right\)\.\(20\)Taking the derivative with respect toπ​\(a\|s\)\\pi\(a\|s\)gives

A​\(s,a\)−τ​\[log⁡π​\(a\|s\)πold​\(a\|s\)\+1\]\+η=0\.A\(s,a\)\-\\tau\\left\[\\log\\frac\{\\pi\(a\|s\)\}\{\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\}\+1\\right\]\+\\eta=0\.\(21\)Thus,

π​\(a\|s\)=πold​\(a\|s\)​exp⁡\(A​\(s,a\)τ\)​exp⁡\(η−ττ\)\.\\pi\(a\|s\)=\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\\exp\\left\(\\frac\{A\(s,a\)\}\{\\tau\}\\right\)\\exp\\left\(\\frac\{\\eta\-\\tau\}\{\\tau\}\\right\)\.\(22\)Normalizing over𝒱\\mathcal\{V\}yields Eq\. equation[19](https://arxiv.org/html/2606.07000#A4.E19)\.□\\square

Eq\. equation[19](https://arxiv.org/html/2606.07000#A4.E19)shows that the advantage function induces a reward\-improved target distribution\. Equivalently, from Eq\. equation[19](https://arxiv.org/html/2606.07000#A4.E19),

log⁡π\+​\(a\|s\)=log⁡πold​\(a\|s\)\+1τ​A​\(s,a\)−log⁡Z​\(s\)\.\\log\\pi^\{\+\}\(a\|s\)=\\log\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\+\\frac\{1\}\{\\tau\}A\(s,a\)\-\\log Z\(s\)\.\(23\)For any policyπ\\pi, we have

DKL\(π\(⋅\|s\)∥π\+\(⋅\|s\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\left\(\\pi\(\\cdot\|s\)\\\|\\pi^\{\+\}\(\\cdot\|s\)\\right\)=∑a∈𝒱π​\(a\|s\)​log⁡π​\(a\|s\)π\+​\(a\|s\)\\displaystyle=\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\(a\|s\)\\log\\frac\{\\pi\(a\|s\)\}\{\\pi^\{\+\}\(a\|s\)\}=∑a∈𝒱π​\(a\|s\)​log⁡π​\(a\|s\)πold​\(a\|s\)−1τ​𝔼a∼π\(⋅\|s\)​A​\(s,a\)\+log⁡Z​\(s\)\\displaystyle=\\sum\_\{a\\in\\mathcal\{V\}\}\\pi\(a\|s\)\\log\\frac\{\\pi\(a\|s\)\}\{\\pi\_\{\\mathrm\{old\}\}\(a\|s\)\}\-\\frac\{1\}\{\\tau\}\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\(s,a\)\+\\log Z\(s\)=DKL\(π\(⋅\|s\)∥πold\(⋅\|s\)\)−1τ𝔼a∼π\(⋅\|s\)A\(s,a\)\+logZ\(s\)\.\\displaystyle=D\_\{\\mathrm\{KL\}\}\\left\(\\pi\(\\cdot\|s\)\\\|\\pi\_\{\\mathrm\{old\}\}\(\\cdot\|s\)\\right\)\-\\frac\{1\}\{\\tau\}\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\|s\)\}A\(s,a\)\+\\log Z\(s\)\.\(24\)Sincelog⁡Z​\(s\)\\log Z\(s\)is independent ofπ\\pi, maximizing Eq\. equation[18](https://arxiv.org/html/2606.07000#A4.E18)is equivalent to minimizing

DKL\(π\(⋅\|s\)∥π\+\(⋅\|s\)\)\.D\_\{\\mathrm\{KL\}\}\\left\(\\pi\(\\cdot\|s\)\\\|\\pi^\{\+\}\(\\cdot\|s\)\\right\)\.\(25\)
Therefore, GRPO can be interpreted as implicit distributional policy improvement\. Its scalar advantage defines an implicit target distributionπ\+\\pi^\{\+\}, and the policy update movesπθ\\pi\_\{\\theta\}toward this reward\-improved distribution\.

In practical GRPO, the token\-level advantage is estimated from group\-relative outcome rewards\. For a sampled responseyiy\_\{i\}, the same group\-level advantageAiA\_\{i\}is assigned to all valid tokens in the response:

A​\(si,t,yi,t\)≈Ai,si,t=\(x,yi,<t\)\.A\(s\_\{i,t\},y\_\{i,t\}\)\\approx A\_\{i\},\\quad s\_\{i,t\}=\(x,y\_\{i,<t\}\)\.\(26\)The clipped GRPO surrogate can thus be viewed as a sample\-based approximation to the distributional improvement in Eq\. equation[25](https://arxiv.org/html/2606.07000#A4.E25), with clipping used to constrain the update aroundπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\.

Token\-level distillation provides an explicit distributional target\. Given a teacher distributionq\(⋅\|s\)q\(\\cdot\|s\)and a student distributionpθ\(⋅\|s\)p\_\{\\theta\}\(\\cdot\|s\), the standard distillation objective is

ℒOPD=𝔼s\[D\(q\(⋅\|s\),pθ\(⋅\|s\)\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=\\mathbb\{E\}\_\{s\}\\left\[D\\left\(q\(\\cdot\|s\),p\_\{\\theta\}\(\\cdot\|s\)\\right\)\\right\],\(27\)whereD​\(⋅,⋅\)D\(\\cdot,\\cdot\)can be instantiated as KL divergence, Jensen\-Shannon divergence, or other distributional discrepancies\. WhenD=DKL​\(q∥pθ\)D=D\_\{\\mathrm\{KL\}\}\(q\\\|p\_\{\\theta\}\), the objective becomes

ℒOPD=𝔼s​\[∑a∈𝒱q​\(a\|s\)​log⁡q​\(a\|s\)pθ​\(a\|s\)\]\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=\\mathbb\{E\}\_\{s\}\\left\[\\sum\_\{a\\in\\mathcal\{V\}\}q\(a\|s\)\\log\\frac\{q\(a\|s\)\}\{p\_\{\\theta\}\(a\|s\)\}\\right\]\.\(28\)Withqqfixed, minimizing Eq\. equation[28](https://arxiv.org/html/2606.07000#A4.E28)is equivalent to minimizing the cross\-entropy

−𝔼s​\[∑a∈𝒱q​\(a\|s\)​log⁡pθ​\(a\|s\)\],\-\\mathbb\{E\}\_\{s\}\\left\[\\sum\_\{a\\in\\mathcal\{V\}\}q\(a\|s\)\\log p\_\{\\theta\}\(a\|s\)\\right\],\(29\)which directly matches the student distribution to the teacher distribution at each token position\.

GRPO and token\-level distillation can therefore be written in a unified form:

minθ⁡𝔼s​\[DKL\(pθ\(⋅\|s\)∥π\+\(⋅\|s\)\)⏟implicit reward\-induced target\+λ​D\(q\(⋅\|s\),pθ\(⋅\|s\)\)⏟explicit teacher\-induced target\]\.\\min\_\{\\theta\}\\;\\mathbb\{E\}\_\{s\}\\left\[\\underbrace\{D\_\{\\mathrm\{KL\}\}\\left\(p\_\{\\theta\}\(\\cdot\|s\)\\\|\\pi^\{\+\}\(\\cdot\|s\)\\right\)\}\_\{\\text\{implicit reward\-induced target\}\}\+\\lambda\\underbrace\{D\\left\(q\(\\cdot\|s\),p\_\{\\theta\}\(\\cdot\|s\)\\right\)\}\_\{\\text\{explicit teacher\-induced target\}\}\\right\]\.\(30\)The first term corresponds to reward\-driven policy improvement, where the target distribution is induced by group\-relative advantages\. The second term corresponds to token\-level distillation, where the target distribution is explicitly provided by a teacher\.

In PTD\-PO, the explicit teacher target is instantiated as a hint\-conditioned self\-teacher:

qi,th=πref\(⋅∣xh,yi,<t\),xh=\[x;h\],q^\{h\}\_\{i,t\}=\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\\mid x^\{h\},y\_\{i,<t\}\\right\),\\quad x^\{h\}=\[x;h\],\(31\)while the student remains conditioned on the original question\-only context:

pi,t=πθ\(⋅∣x,yi,<t\)\.p\_\{i,t\}=\\pi\_\{\\theta\}\\left\(\\cdot\\mid x,y\_\{i,<t\}\\right\)\.\(32\)Thus, PTD\-PO does not replace reward optimization with imitation learning\. It augments the implicit reward\-induced target of GRPO with an explicit token\-level corrective target on selected trajectories:

ℒPTD​\-​PO=LGRPO​\(θ\)\+λPTD​ℒPTD\.\\mathcal\{L\}\_\{\\mathrm\{PTD\\mbox\{\-\}PO\}\}=L\_\{\\mathrm\{GRPO\}\}\(\\theta\)\+\\lambda\_\{\\mathrm\{PTD\}\}\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}\.\(33\)
This establishes the theoretical role of PTD: GRPO provides an on\-policy reward\-improvement direction, while PTD provides a token\-level distributional correction when the reward\-induced target is under\-informative\.

### D\.2Why Jensen\-Shannon Divergence Instead of Directional KL

We analyze the optimization behavior of directional KL objectives and Jensen\-Shannon divergence under the asymmetric context setting of PTD\-PO\. Let

q\(⋅\|s\)=πref\(⋅\|xh,y<t\),pθ\(⋅\|s\)=πθ\(⋅\|x,y<t\),q\(\\cdot\|s\)=\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x^\{h\},y\_\{<t\}\),\\quad p\_\{\\theta\}\(\\cdot\|s\)=\\pi\_\{\\theta\}\(\\cdot\|x,y\_\{<t\}\),\(34\)where the teacher distributionqqis conditioned on the hint\-augmented contextxh=\[x;h\]x^\{h\}=\[x;h\], while the student distributionpθp\_\{\\theta\}is conditioned on the original question\-only contextxx\.

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/kl_gradient_norm_reward_first50_legend_lower.png)Figure 8:Comparison between forward KL and reverse KL during PTD\-style token\-level distillation\. Forward KL produces a more moderate gradient norm but yields weaker reward improvement, while reverse KL introduces substantially larger early\-stage gradients and quickly collapses the reward signal\.#### Forward KL\.

The forward KL objective is

DKL​\(q∥pθ\)=∑v∈𝒱q​\(v\)​log⁡q​\(v\)pθ​\(v\)\.D\_\{\\mathrm\{KL\}\}\(q\\\|p\_\{\\theta\}\)=\\sum\_\{v\\in\\mathcal\{V\}\}q\(v\)\\log\\frac\{q\(v\)\}\{p\_\{\\theta\}\(v\)\}\.\(35\)Letzvz\_\{v\}denote the student logit andpθ​\(v\)=softmax​\(z\)vp\_\{\\theta\}\(v\)=\\mathrm\{softmax\}\(z\)\_\{v\}\. Sinceqqis fixed, the gradient of forward KL with respect to the student logit is

∂DKL​\(q∥pθ\)∂zv=pθ​\(v\)−q​\(v\)\.\\frac\{\\partial D\_\{\\mathrm\{KL\}\}\(q\\\|p\_\{\\theta\}\)\}\{\\partial z\_\{v\}\}=p\_\{\\theta\}\(v\)\-q\(v\)\.\(36\)Thus, forward KL directly pulls the student distribution toward the teacher distribution at every token position\. Whenqqis produced under a hint\-augmented context, the differenceq​\(v\)−pθ​\(v\)q\(v\)\-p\_\{\\theta\}\(v\)contains not only corrective reasoning information, but also context\-induced distribution shift\. Therefore, forward KL may force the question\-only student to imitate probability mass that is specific to the privileged context, rather than learning a stable reward\-aligned correction\.

#### Reverse KL\.

The reverse KL objective is

DKL​\(pθ∥q\)=∑v∈𝒱pθ​\(v\)​log⁡pθ​\(v\)q​\(v\)\.D\_\{\\mathrm\{KL\}\}\(p\_\{\\theta\}\\\|q\)=\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{\\theta\}\(v\)\\log\\frac\{p\_\{\\theta\}\(v\)\}\{q\(v\)\}\.\(37\)Its gradient with respect to the student logit is

∂DKL​\(pθ∥q\)∂zv=pθ​\(v\)​\[log⁡pθ​\(v\)q​\(v\)−𝔼u∼pθ​\(log⁡pθ​\(u\)q​\(u\)\)\]\.\\frac\{\\partial D\_\{\\mathrm\{KL\}\}\(p\_\{\\theta\}\\\|q\)\}\{\\partial z\_\{v\}\}=p\_\{\\theta\}\(v\)\\left\[\\log\\frac\{p\_\{\\theta\}\(v\)\}\{q\(v\)\}\-\\mathbb\{E\}\_\{u\\sim p\_\{\\theta\}\}\\left\(\\log\\frac\{p\_\{\\theta\}\(u\)\}\{q\(u\)\}\\right\)\\right\]\.\(38\)Reverse KL penalizes assigning probability mass to tokens with small teacher probability\. When the hint\-conditioned teacher becomes sharper than the question\-only student, tokens that remain plausible under the student but receive low probability under the teacher can induce large logarithmic penalties\. This produces a mode\-seeking effect and can suppress the exploratory distribution maintained by RLVR\.

Figure[8](https://arxiv.org/html/2606.07000#A4.F8)empirically illustrates this behavior\. Reverse KL yields much larger gradient norms in the early optimization stage, while its reward quickly degenerates\. Forward KL is more stable in gradient magnitude, but its reward improvement is weaker\. These results indicate that directional KL objectives are not well matched to the asymmetric teacher\-student contexts in PTD\-PO\.

#### Jensen\-Shannon divergence\.

PTD\-PO instead uses Jensen\-Shannon divergence:

DJSD​\(q,pθ\)=12​DKL​\(q∥m\)\+12​DKL​\(pθ∥m\),m=12​\(q\+pθ\)\.D\_\{\\mathrm\{JSD\}\}\(q,p\_\{\\theta\}\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(q\\\|m\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\theta\}\\\|m\),\\quad m=\\frac\{1\}\{2\}\(q\+p\_\{\\theta\}\)\.\(39\)Different from directional KL, JSD aligns both distributions through the mixture distributionmm\. For each tokenvv, the denominator in the logarithmic ratio is not the opposite distribution alone, but the smoothed mixture

m​\(v\)=12​\(q​\(v\)\+pθ​\(v\)\)\.m\(v\)=\\frac\{1\}\{2\}\\left\(q\(v\)\+p\_\{\\theta\}\(v\)\\right\)\.\(40\)This prevents either distribution from acting as an overly strict one\-sided target\.

The derivative of JSD with respect to the student probability has a simple form:

∂DJSD​\(q,pθ\)∂pθ​\(v\)=12​log⁡pθ​\(v\)m​\(v\)=12​log⁡2​pθ​\(v\)pθ​\(v\)\+q​\(v\)\.\\frac\{\\partial D\_\{\\mathrm\{JSD\}\}\(q,p\_\{\\theta\}\)\}\{\\partial p\_\{\\theta\}\(v\)\}=\\frac\{1\}\{2\}\\log\\frac\{p\_\{\\theta\}\(v\)\}\{m\(v\)\}=\\frac\{1\}\{2\}\\log\\frac\{2p\_\{\\theta\}\(v\)\}\{p\_\{\\theta\}\(v\)\+q\(v\)\}\.\(41\)Accordingly, the gradient with respect to the student logit is

∂DJSD​\(q,pθ\)∂zv=12​pθ​\(v\)​\[log⁡pθ​\(v\)m​\(v\)−𝔼u∼pθ​\(log⁡pθ​\(u\)m​\(u\)\)\]\.\\frac\{\\partial D\_\{\\mathrm\{JSD\}\}\(q,p\_\{\\theta\}\)\}\{\\partial z\_\{v\}\}=\\frac\{1\}\{2\}p\_\{\\theta\}\(v\)\\left\[\\log\\frac\{p\_\{\\theta\}\(v\)\}\{m\(v\)\}\-\\mathbb\{E\}\_\{u\\sim p\_\{\\theta\}\}\\left\(\\log\\frac\{p\_\{\\theta\}\(u\)\}\{m\(u\)\}\\right\)\\right\]\.\(42\)Compared with Eq\. equation[38](https://arxiv.org/html/2606.07000#A4.E38), JSD replaces the teacher denominatorq​\(v\)q\(v\)with the mixture denominatorm​\(v\)m\(v\)\. Thus, even whenq​\(v\)q\(v\)is very small, the penalty is softened by the student probabilitypθ​\(v\)p\_\{\\theta\}\(v\)throughm​\(v\)m\(v\)\. This reduces the risk of extremely aggressive gradients caused by context\-induced teacher sharpness\.

#### Boundedness\.

JSD is bounded:

0≤DJSD​\(q,pθ\)≤log⁡2\.0\\leq D\_\{\\mathrm\{JSD\}\}\(q,p\_\{\\theta\}\)\\leq\\log 2\.\(43\)The lower bound follows from the non\-negativity of KL divergence\. For the upper bound, since

m​\(v\)=q​\(v\)\+pθ​\(v\)2≥q​\(v\)2,m​\(v\)≥pθ​\(v\)2,m\(v\)=\\frac\{q\(v\)\+p\_\{\\theta\}\(v\)\}\{2\}\\geq\\frac\{q\(v\)\}\{2\},\\quad m\(v\)\\geq\\frac\{p\_\{\\theta\}\(v\)\}\{2\},\(44\)we have

DKL​\(q∥m\)=∑vq​\(v\)​log⁡q​\(v\)m​\(v\)≤∑vq​\(v\)​log⁡2=log⁡2,D\_\{\\mathrm\{KL\}\}\(q\\\|m\)=\\sum\_\{v\}q\(v\)\\log\\frac\{q\(v\)\}\{m\(v\)\}\\leq\\sum\_\{v\}q\(v\)\\log 2=\\log 2,\(45\)and similarly,

DKL​\(pθ∥m\)≤log⁡2\.D\_\{\\mathrm\{KL\}\}\(p\_\{\\theta\}\\\|m\)\\leq\\log 2\.\(46\)Substituting these two inequalities into Eq\. equation[39](https://arxiv.org/html/2606.07000#A4.E39)gives Eq\. equation[43](https://arxiv.org/html/2606.07000#A4.E43)\.

Therefore, JSD provides a symmetric and bounded distributional discrepancy\. In PTD\-PO, this property is important because the teacher and student are intentionally conditioned on different contexts:

q\(⋅\|s\)=πref\(⋅\|xh,y<t\),pθ\(⋅\|s\)=πθ\(⋅\|x,y<t\)\.q\(\\cdot\|s\)=\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|x^\{h\},y\_\{<t\}\),\\quad p\_\{\\theta\}\(\\cdot\|s\)=\\pi\_\{\\theta\}\(\\cdot\|x,y\_\{<t\}\)\.\(47\)The objective should transfer corrective reasoning information from the hint\-conditioned teacher, but should not force the student to fully reproduce the privileged\-context distribution\. JSD satisfies this requirement by using a softened mixture target and by avoiding the one\-sided optimization bias of directional KL objectives\.

### D\.3Approximation Property of Top\-K JSD with Tail Compensation

We analyze the approximation property of the proposed Top\-K JSD with tail compensation\. For simplicity, we omit the token position index\(i,t\)\(i,t\)and write the student and teacher distributions asppandqqover the vocabulary𝒱\\mathcal\{V\}\. LetS⊂𝒱S\\subset\\mathcal\{V\}be the compact support used by Top\-K JSD:

S=TopK⁡\(p\)∪TopK⁡\(q\)\.S=\\operatorname\{TopK\}\(p\)\\cup\\operatorname\{TopK\}\(q\)\.\(48\)The remaining vocabulary set is denoted by

S¯=𝒱∖S\.\\bar\{S\}=\\mathcal\{V\}\\setminus S\.\(49\)The tail masses ofppandqqare

ptail=∑v∈S¯p​\(v\)=1−∑v∈Sp​\(v\),qtail=∑v∈S¯q​\(v\)=1−∑v∈Sq​\(v\)\.p\_\{\\mathrm\{tail\}\}=\\sum\_\{v\\in\\bar\{S\}\}p\(v\)=1\-\\sum\_\{v\\in S\}p\(v\),\\quad q\_\{\\mathrm\{tail\}\}=\\sum\_\{v\\in\\bar\{S\}\}q\(v\)=1\-\\sum\_\{v\\in S\}q\(v\)\.\(50\)The compressed distributions are defined as

p~=\(\{p​\(v\)\}v∈S,ptail\),q~=\(\{q​\(v\)\}v∈S,qtail\)\.\\tilde\{p\}=\\left\(\\\{p\(v\)\\\}\_\{v\\in S\},p\_\{\\mathrm\{tail\}\}\\right\),\\quad\\tilde\{q\}=\\left\(\\\{q\(v\)\\\}\_\{v\\in S\},q\_\{\\mathrm\{tail\}\}\\right\)\.\(51\)
#### Validity of the compressed distributions\.

The compressed distributions remain valid probability distributions:

∑v∈Sp~​\(v\)\+p~tail=∑v∈Sp​\(v\)\+\(1−∑v∈Sp​\(v\)\)=1,\\sum\_\{v\\in S\}\\tilde\{p\}\(v\)\+\\tilde\{p\}\_\{\\mathrm\{tail\}\}=\\sum\_\{v\\in S\}p\(v\)\+\\left\(1\-\\sum\_\{v\\in S\}p\(v\)\\right\)=1,\(52\)and similarly,

∑v∈Sq~​\(v\)\+q~tail=1\.\\sum\_\{v\\in S\}\\tilde\{q\}\(v\)\+\\tilde\{q\}\_\{\\mathrm\{tail\}\}=1\.\(53\)Thus, Top\-K JSD is computed on a normalized distribution rather than on a hard\-truncated and renormalized distribution\. This distinction is important: hard truncation discards the probability mass outsideSS, whereas tail compensation preserves this mass as an additional bucket\.

#### Coarse\-graining view\.

Let

m=12​\(p\+q\),m~=12​\(p~\+q~\)\.m=\\frac\{1\}\{2\}\(p\+q\),\\quad\\tilde\{m\}=\\frac\{1\}\{2\}\(\\tilde\{p\}\+\\tilde\{q\}\)\.\(54\)The tail mass ofmmis

mtail=∑v∈S¯m​\(v\)=12​\(ptail\+qtail\)\.m\_\{\\mathrm\{tail\}\}=\\sum\_\{v\\in\\bar\{S\}\}m\(v\)=\\frac\{1\}\{2\}\\left\(p\_\{\\mathrm\{tail\}\}\+q\_\{\\mathrm\{tail\}\}\\right\)\.\(55\)The full\-vocabulary JSD is

DJSD​\(q,p\)=12​DKL​\(q∥m\)\+12​DKL​\(p∥m\),D\_\{\\mathrm\{JSD\}\}\(q,p\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(q\\\|m\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\\\|m\),\(56\)while the Top\-K JSD with tail compensation is

DTopK​\-​JSD​\(q,p\)=DJSD​\(q~,p~\)=12​DKL​\(q~∥m~\)\+12​DKL​\(p~∥m~\)\.D\_\{\\mathrm\{TopK\\mbox\{\-\}JSD\}\}\(q,p\)=D\_\{\\mathrm\{JSD\}\}\(\\tilde\{q\},\\tilde\{p\}\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(\\tilde\{q\}\\\|\\tilde\{m\}\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(\\tilde\{p\}\\\|\\tilde\{m\}\)\.\(57\)
We next decompose the difference between full JSD and Top\-K JSD\. Assumeqtail\>0q\_\{\\mathrm\{tail\}\}\>0,ptail\>0p\_\{\\mathrm\{tail\}\}\>0, andmtail\>0m\_\{\\mathrm\{tail\}\}\>0\. Define the conditional tail distributions

qS¯​\(v\)=q​\(v\)qtail,pS¯​\(v\)=p​\(v\)ptail,mS¯​\(v\)=m​\(v\)mtail,v∈S¯\.q\_\{\\bar\{S\}\}\(v\)=\\frac\{q\(v\)\}\{q\_\{\\mathrm\{tail\}\}\},\\quad p\_\{\\bar\{S\}\}\(v\)=\\frac\{p\(v\)\}\{p\_\{\\mathrm\{tail\}\}\},\\quad m\_\{\\bar\{S\}\}\(v\)=\\frac\{m\(v\)\}\{m\_\{\\mathrm\{tail\}\}\},\\quad v\\in\\bar\{S\}\.\(58\)By the chain rule of KL divergence under a partition\{S,S¯\}\\\{S,\\bar\{S\}\\\}, we have

DKL​\(q∥m\)=DKL​\(q~∥m~\)\+qtail​DKL​\(qS¯∥mS¯\),D\_\{\\mathrm\{KL\}\}\(q\\\|m\)=D\_\{\\mathrm\{KL\}\}\(\\tilde\{q\}\\\|\\tilde\{m\}\)\+q\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(q\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\),\(59\)and

DKL​\(p∥m\)=DKL​\(p~∥m~\)\+ptail​DKL​\(pS¯∥mS¯\)\.D\_\{\\mathrm\{KL\}\}\(p\\\|m\)=D\_\{\\mathrm\{KL\}\}\(\\tilde\{p\}\\\|\\tilde\{m\}\)\+p\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\)\.\(60\)Substituting Eq\. equation[59](https://arxiv.org/html/2606.07000#A4.E59)and Eq\. equation[60](https://arxiv.org/html/2606.07000#A4.E60)into the definition of JSD gives

DJSD​\(q,p\)\\displaystyle D\_\{\\mathrm\{JSD\}\}\(q,p\)=DJSD​\(q~,p~\)\\displaystyle=D\_\{\\mathrm\{JSD\}\}\(\\tilde\{q\},\\tilde\{p\}\)\+12​qtail​DKL​\(qS¯∥mS¯\)\+12​ptail​DKL​\(pS¯∥mS¯\)\.\\displaystyle\\quad\+\\frac\{1\}\{2\}q\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(q\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\)\+\\frac\{1\}\{2\}p\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\)\.\(61\)Therefore,

DTopK​\-​JSD​\(q,p\)=DJSD​\(q~,p~\)≤DJSD​\(q,p\)\.D\_\{\\mathrm\{TopK\\mbox\{\-\}JSD\}\}\(q,p\)=D\_\{\\mathrm\{JSD\}\}\(\\tilde\{q\},\\tilde\{p\}\)\\leq D\_\{\\mathrm\{JSD\}\}\(q,p\)\.\(62\)The approximation error is

Δtail=DJSD​\(q,p\)−DTopK​\-​JSD​\(q,p\)=12​qtail​DKL​\(qS¯∥mS¯\)\+12​ptail​DKL​\(pS¯∥mS¯\)\.\\Delta\_\{\\mathrm\{tail\}\}=D\_\{\\mathrm\{JSD\}\}\(q,p\)\-D\_\{\\mathrm\{TopK\\mbox\{\-\}JSD\}\}\(q,p\)=\\frac\{1\}\{2\}q\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(q\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\)\+\\frac\{1\}\{2\}p\_\{\\mathrm\{tail\}\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\bar\{S\}\}\\\|m\_\{\\bar\{S\}\}\)\.\(63\)Thus, Top\-K JSD with tail compensation preserves the exact JSD contribution on the selected supportSSand only discards the fine\-grained divergence inside the aggregated tail regionS¯\\bar\{S\}\. The total probability mass of the tail is still preserved byptailp\_\{\\mathrm\{tail\}\}andqtailq\_\{\\mathrm\{tail\}\}\.

#### Error interpretation\.

Eq\. equation[63](https://arxiv.org/html/2606.07000#A4.E63)shows that the approximation error depends on two factors: the tail massesptailp\_\{\\mathrm\{tail\}\}andqtailq\_\{\\mathrm\{tail\}\}, and the internal discrepancy between the conditional tail distributions\. When the selected Top\-K support captures most of the probability mass,ptailp\_\{\\mathrm\{tail\}\}andqtailq\_\{\\mathrm\{tail\}\}are small, and the residual term is correspondingly limited\. Moreover, the high\-probability tokens inSSare kept explicitly, so the dominant token\-level alignment signal is retained\.

This property differs from hard truncation\. If the tail is simply removed and the remaining probabilities are renormalized, the resulting objective changes the relative scale of the retained tokens and ignores the mismatch in the total omitted probability mass\. In contrast, tail compensation preserves the mass outsideSS:

p~tail=ptail,q~tail=qtail,\\tilde\{p\}\_\{\\mathrm\{tail\}\}=p\_\{\\mathrm\{tail\}\},\\quad\\tilde\{q\}\_\{\\mathrm\{tail\}\}=q\_\{\\mathrm\{tail\}\},\(64\)so the objective still penalizes cases where the student and teacher assign substantially different total probability to the omitted vocabulary\.

#### Memory complexity\.

Full\-vocabulary token\-level distillation requires storing or computing distributions over all vocabulary tokens for each valid response position\. For batch sizeBB, response lengthTT, and vocabulary sizeVV, the memory complexity is

Top\-K JSD keeps only the union of two Top\-K supports and one tail bucket\. The resulting complexity becomes

O​\(B​T​K\),K≪V,O\(BTK\),\\quad K\\ll V,\(66\)up to a small constant factor from the union support and the tail bucket\. Therefore, Top\-K JSD with tail compensation is a mass\-preserving coarse\-grained approximation to full\-vocabulary JSD, retaining the dominant distributional alignment signal while substantially reducing the memory overhead of online token\-level distillation\.

### D\.4Optimization Rationale for Applying PTD to Failed Trajectories

We analyze why PTD is applied to failed trajectories rather than all sampled trajectories\. For a questionxx, let the policy sample a group of responses\{yi\}i=1G\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}with binary verifiable rewardsri∈\{0,1\}r\_\{i\}\\in\\\{0,1\\\}\. The group\-relative advantage used by GRPO is

Ai=ri−r¯σr\+ϵ,r¯=1G​∑j=1Grj,σr=std⁡\(\{rj\}j=1G\)\.A\_\{i\}=\\frac\{r\_\{i\}\-\\bar\{r\}\}\{\\sigma\_\{r\}\+\\epsilon\},\\quad\\bar\{r\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\_\{j\},\\quad\\sigma\_\{r\}=\\operatorname\{std\}\\left\(\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\\right\)\.\(67\)
#### Degeneration under all\-fail groups\.

When all sampled trajectories fail, we have

r1=r2=⋯=rG=0\.r\_\{1\}=r\_\{2\}=\\cdots=r\_\{G\}=0\.\(68\)Then

r¯=0,σr=0,Ai=0,∀i\.\\bar\{r\}=0,\\quad\\sigma\_\{r\}=0,\\quad A\_\{i\}=0,\\quad\\forall i\.\(69\)Thus, the clipped policy\-gradient term in GRPO provides no discriminative improvement signal for the failed trajectories\. Ignoring the KL regularization term, the token\-level GRPO gradient for a sampled tokenyi,ty\_\{i,t\}is proportional to

−Ai​∇θlog⁡πθ​\(yi,t∣x,yi,<t\),\-A\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\left\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\\right\),\(70\)which vanishes whenAi=0A\_\{i\}=0\. Therefore, all\-fail groups require an additional corrective signal that can provide token\-level guidance beyond scalar outcome rewards\.

#### Negative advantage is not equivalent to corrective supervision\.

When a group contains both successful and failed responses, let

m=∑i=1Gri,0<m<G\.m=\\sum\_\{i=1\}^\{G\}r\_\{i\},\\quad 0<m<G\.\(71\)For a failed trajectory withri=0r\_\{i\}=0, its advantage is

Ai=−r¯σr\+ϵ<0\.A\_\{i\}=\\frac\{\-\\bar\{r\}\}\{\\sigma\_\{r\}\+\\epsilon\}<0\.\(72\)For a successful trajectory withri=1r\_\{i\}=1, its advantage is

Ai=1−r¯σr\+ϵ\>0\.A\_\{i\}=\\frac\{1\-\\bar\{r\}\}\{\\sigma\_\{r\}\+\\epsilon\}\>0\.\(73\)The negative advantage in Eq\. equation[72](https://arxiv.org/html/2606.07000#A4.E72)discourages the sampled failed trajectory\. However, it does not specify which alternative tokens should receive higher probability at each intermediate reasoning step\. In other words, GRPO can suppress low\-reward sampled actions, but it does not directly construct a token\-level corrective distribution for the failed reasoning path\.

PTD supplies such a distributional correction\. For a failed trajectory, the PTD gradient is induced by

DPTD\(qi,th,pi,t\),qi,th=πref\(⋅∣xh,yi,<t\),pi,t=πθ\(⋅∣x,yi,<t\)\.D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\),\\quad q^\{h\}\_\{i,t\}=\\pi\_\{\\mathrm\{ref\}\}\\left\(\\cdot\\mid x^\{h\},y\_\{i,<t\}\\right\),\\quad p\_\{i,t\}=\\pi\_\{\\theta\}\\left\(\\cdot\\mid x,y\_\{i,<t\}\\right\)\.\(74\)Thus, PTD complements the scalar reward signal with a dense token\-level target generated from the hint\-conditioned self\-teacher\.

#### Over\-regularization on successful trajectories\.

Applying PTD to successful trajectories is not always desirable\. A successful trajectory already receives a positive reward\-driven update from GRPO, as shown in Eq\. equation[73](https://arxiv.org/html/2606.07000#A4.E73)\. If PTD is additionally applied to this trajectory, the optimization objective contains

λPTD​∑t=1Timi,t​DPTD​\(qi,th,pi,t\),ri=1\.\\lambda\_\{\\mathrm\{PTD\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\),\\quad r\_\{i\}=1\.\(75\)Sinceqi,thq^\{h\}\_\{i,t\}is conditioned on the privileged contextxh=\[x;h\]x^\{h\}=\[x;h\], it is not guaranteed to coincide with the question\-only distribution that produced a correct trajectory\. Therefore, Eq\. equation[75](https://arxiv.org/html/2606.07000#A4.E75)may pull a reward\-consistent on\-policy behavior toward a hint\-conditioned distribution\. This introduces an additional regularization bias that is not required for solving the instance\.

This can be made explicit by decomposing an all\-trajectory PTD objective into failed and successful parts:

ℒPTDall=ℒPTDfail\+ℒPTDsucc,\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{all\}\}=\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{fail\}\}\+\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{succ\}\},\(76\)where

ℒPTDfail=∑i:ri=0∑t=1Timi,t​DPTD​\(qi,th,pi,t\)∑i:ri=0∑t=1Timi,t\+ϵ,\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{fail\}\}=\\frac\{\\sum\_\{i:r\_\{i\}=0\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\)\}\{\\sum\_\{i:r\_\{i\}=0\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}\+\\epsilon\},\(77\)and

ℒPTDsucc=∑i:ri=1∑t=1Timi,t​DPTD​\(qi,th,pi,t\)∑i:ri=1∑t=1Timi,t\+ϵ\.\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{succ\}\}=\\frac\{\\sum\_\{i:r\_\{i\}=1\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\)\}\{\\sum\_\{i:r\_\{i\}=1\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}\+\\epsilon\}\.\(78\)The failed part provides corrective supervision for trajectories with insufficient reward feedback\. The successful part instead constrains already correct trajectories to match the hint\-conditioned self\-teacher\. Since the teacher and student are conditioned on different contexts, this term can conflict with the on\-policy reward\-improvement direction\.

#### Selective PTD objective\.

PTD\-PO therefore applies distillation only to failed trajectories in PTD\-activated groups\. Let

αi=𝟏​\[r¯​\(x\)<τptd\]​𝟏​\[ri=0\],\\alpha\_\{i\}=\\mathbf\{1\}\\left\[\\bar\{r\}\(x\)<\\tau\_\{\\mathrm\{ptd\}\}\\right\]\\mathbf\{1\}\\left\[r\_\{i\}=0\\right\],\(79\)whereτptd\\tau\_\{\\mathrm\{ptd\}\}is the group\-level activation threshold\. The selective PTD objective can be written as

ℒPTD=∑i=1Gαi​∑t=1Timi,t​DPTD​\(qi,th,pi,t\)∑i=1Gαi​∑t=1Timi,t\+ϵ\.\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}=\\frac\{\\sum\_\{i=1\}^\{G\}\\alpha\_\{i\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}D\_\{\\mathrm\{PTD\}\}\\left\(q^\{h\}\_\{i,t\},p\_\{i,t\}\\right\)\}\{\\sum\_\{i=1\}^\{G\}\\alpha\_\{i\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}\+\\epsilon\}\.\(80\)Withτptd=1\.0\\tau\_\{\\mathrm\{ptd\}\}=1\.0, PTD is activated for groups that contain at least one failed trajectory, while fully correct groups are excluded from distillation\. Within activated groups, only failed trajectories are distilled\.

The final objective is

ℒPTD​\-​GRPO=−JGRPO​\(θ\)\+λPTD​ℒPTD\.\\mathcal\{L\}\_\{\\mathrm\{PTD\\mbox\{\-\}GRPO\}\}=\-J\_\{\\mathrm\{GRPO\}\}\(\\theta\)\+\\lambda\_\{\\mathrm\{PTD\}\}\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}\.\(81\)This objective preserves the reward\-driven update on successful trajectories and injects privileged token\-level correction only where the outcome reward is insufficient\.

#### Gradient decomposition\.

Let

gGRPO=∇θ\(−JGRPO\),gPTD=∇θℒPTD\.g\_\{\\mathrm\{GRPO\}\}=\\nabla\_\{\\theta\}\\left\(\-J\_\{\\mathrm\{GRPO\}\}\\right\),\\quad g\_\{\\mathrm\{PTD\}\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{PTD\}\}\.\(82\)The total gradient is

gtotal=gGRPO\+λPTD​gPTD\.g\_\{\\mathrm\{total\}\}=g\_\{\\mathrm\{GRPO\}\}\+\\lambda\_\{\\mathrm\{PTD\}\}g\_\{\\mathrm\{PTD\}\}\.\(83\)Under all\-trajectory distillation, the PTD gradient can be decomposed as

gPTDall=gPTDfail\+gPTDsucc\.g\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{all\}\}=g\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{fail\}\}\+g\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{succ\}\}\.\(84\)The termgPTDfailg\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{fail\}\}provides a corrective gradient for failed trajectories\. The termgPTDsuccg\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{succ\}\}imposes hint\-conditioned regularization on successful trajectories and is not guaranteed to align withgGRPOg\_\{\\mathrm\{GRPO\}\}\. Selective PTD removes this potentially conflicting term:

gPTDselective=gPTDfail\.g\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{selective\}\}=g\_\{\\mathrm\{PTD\}\}^\{\\mathrm\{fail\}\}\.\(85\)Therefore, applying PTD to failed trajectories reduces unnecessary regularization on reward\-consistent behaviors while preserving dense correction for trajectories where GRPO provides weak or incomplete token\-level supervision\.

## Appendix EAdditional Experiments and Analysis

### E\.1Compatibility With Other RLVR Optimizers

Table 4:PTD consistently improves different RLVR optimizers on Qwen3\-VL\-4B\-Thinking\.MethodGeneral Multimodal ReasoningVision\-Dependent Multimodal ReasoningOverallMMK12Geo3KMathVerseMathVistaWe\-MathAVGMMMU\-ProCountingMathVerseVLogicVistaAVGAVGΔr​e​l%\\Delta\_\{rel\}^\{\\%\}GRPO65\.9258\.1574\.3867\.3375\.6968\.2935\.7492\.2570\.3052\.0162\.5865\.75–PTD\-POGRPO71\.3361\.0277\.5669\.5579\.3971\.7738\.5692\.8874\.1657\.6365\.8169\.12↑\\uparrow5\.13DAPO64\.0353\.7270\.3467\.3973\.4665\.7935\.0192\.2566\.7452\.2961\.5763\.91–PTD\-PODAPO67\.5156\.0373\.8569\.0174\.9768\.2735\.1492\.6270\.5052\.8262\.7765\.83↑\\uparrow3\.00GSPO70\.4360\.7778\.5169\.5879\.4671\.7538\.7592\.6274\.2559\.0666\.1769\.27–PTD\-POGSPO71\.8261\.6379\.1273\.8979\.8673\.2640\.3092\.9475\.2360\.6567\.2870\.60↑\\uparrow1\.92

Table[4](https://arxiv.org/html/2606.07000#A5.T4)evaluates whether PTD can be combined with different RLVR optimizers\. Across GRPO, DAPO, and GSPO, adding PTD consistently improves the overall average performance, with gains of5\.13%5\.13\\%,3\.00%3\.00\\%, and1\.92%1\.92\\%, respectively\. These results indicate that PTD is not tied to a specific policy optimization objective, but can serve as a general auxiliary distillation module for RLVR by providing dense corrective guidance to failed trajectories\.

### E\.2Sensitivity Analysis on the Top\-KKSupport Size

Table 5:Effect of the Top\-KKsupport size in PTD on Qwen3\-VL\-4B\- and 8B\-Thinking\.Table[5](https://arxiv.org/html/2606.07000#A5.T5)studies the effect of the Top\-KKsupport size used in the PTD divergence\. We varyKKwhile keeping all other training settings unchanged, and report results on four representative benchmarks covering both general reasoning and vision\-dependent reasoning\. This experiment examines whether the default choiceK=100K=100provides a good trade\-off between retaining informative token\-level distribution signals and reducing the memory cost of full\-vocabulary distillation\.

## Appendix FCase Study

We provide qualitative examples to further understand the behavior of PTD\-PO\. Figures[9](https://arxiv.org/html/2606.07000#A7.F9)–[11](https://arxiv.org/html/2606.07000#A7.F11)show representative successful cases covering physics reasoning, geometry reasoning, and science reasoning with visual diagrams\. Across these examples, PTD\-PO identifies task\-relevant visual or structural information and follows a coherent reasoning path before producing the correct answer\. This indicates that PTD\-PO improves multimodal grounding and intermediate reasoning, rather than merely fitting final answers\.

## Appendix GLimitations and Future Work

#### Limitations\.

Although PTD\-PO improves failed\-trajectory learning through dense privileged guidance, it does not fully eliminate reasoning failures in highly compositional multimodal problems\. As shown in Figure[12](https://arxiv.org/html/2606.07000#A7.F12), the model may follow a generally plausible reasoning direction but still produce an unreliable final answer when the task requires fine\-grained verification, such as chemical equation balancing, multi\-option checking, or strict consistency between intermediate reasoning and the final decision\. This suggests that when the hint is insufficient to resolve the core difficulty, the model may over\-rely on partial guidance and terminate the reasoning chain prematurely or inconsistently\. In addition, the relative gains of PTD\-PO become less pronounced on larger models compared with smaller ones\. One possible reason is the mismatch between model capability and training\-data difficulty: stronger models produce fewer failed trajectories during RLVR training, which reduces the number of cases where PTD can be activated to provide corrective supervision\. Since PTD\-PO is mainly designed to improve learning from failed rollouts, its effective scope naturally depends on the availability of sufficiently challenging examples that expose such failures\. Therefore, PTD\-PO should be viewed as an effective way to improve corrective supervision for failed rollouts, rather than a complete solution to all visual grounding and symbolic reasoning errors\. Constructing harder and more capability\-matched multimodal RLVR training data may further enlarge the benefit of privileged\-information distillation, especially for stronger models\.

#### Future Work\.

An important future direction is to construct harder and more capability\-matched RLVR training data for stronger LVLMs\. Since PTD\-PO mainly provides corrective supervision for failed trajectories, larger models may benefit less when the current training data is not sufficiently challenging and produces too few informative failures\. Designing more difficult multimodal reasoning tasks, especially those requiring fine\-grained visual grounding, symbolic verification, and long\-chain reasoning, could expose more failure cases and further enlarge the effective scope of privileged\-information distillation\. Beyond static multimodal reasoning, another promising direction is to extend PTD\-style training to multimodal agent settings\. In agentic tasks, models need to interact with environments, use tools, update observations, and make long\-horizon decisions\. Future work may construct answer\-free privileged hints from intermediate observations, tool feedback, or expert action traces, so as to guide failed agent trajectories without directly revealing the final goal state\. Such an extension could help improve exploration, grounding, and recovery behaviors in multimodal agents\.

![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PTD-GRPO_True1.png)Figure 9:Physics reasoning case\.![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PTD-GRPO_True2.png)Figure 10:Geometry reasoning case\.![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PTD-GRPO_True3.png)Figure 11:Science reasoning case\.![Refer to caption](https://arxiv.org/html/2606.07000v1/figs/PTD-GRPO_Wrong.png)Figure 12:Failure case\.