Token Utility for On-Policy Distillation via Local Trust-Region Optimization

# Token Utility for On-Policy Distillation via Local Trust-Region Optimization
Source: [https://arxiv.org/html/2610.18883](https://arxiv.org/html/2610.18883)
Narayanan Sadagopan1Yilan Chen2Vijay Lingam11AWS AI Labs2UC San Diego††thanks:Corresponding author\.††thanks:Work done during an internship at AWS AI Labs\.

###### Abstract

Token\-selection methods for on\-policy distillation \(OPD\) prioritize student\-visited positions using diverse signals, including uncertainty, teacher–student disagreement, and mismatch direction\. We instead derive token importance from the distillation objective itself: how much can each position’s loss decrease under an equal local KL budget on the student’s predictive distribution? The answer is the utilityst=gt⊤​Ft†​gts\_\{t\}=g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}, wheregtg\_\{t\}is the logit gradient andFtF\_\{t\}is the categorical Fisher matrix\. For reverse\-KL it equals the student\-weighted variance of the student–teacher log\-probability ratio, and for forward\-KL it equals the Pearsonχ2\\chi^\{2\}divergence, so the appropriate utility follows from the choice of objective\. On shared training checkpoints, several independently motivated criteria, including student entropy, KL disagreement, and TIP, concentrate their importance on nearly the same positions as the derived utility at a 50% selection fraction, while diverging at stricter fractions and for direction\-specific criteria\. We instantiate the utility as Fisher Utility Weighting \(FUW\), a continuous, mean\-normalized token weighting\. Across seven mathematical reasoning benchmarks and three Qwen3 student–teacher pairs, FUW improves over matched uniform OPD under both reverse\-KL \(1\.35–4\.0 macro points\) and forward\-KL \(1\.08–2\.71 macro points\), and is competitive with existing selective methods using a single closed\-form score derived from the training objective\.

## 1Introduction

On\-policy distillation \(OPD\) transfers capabilities from a stronger teacher while retaining the student’s state distribution\([Agarwal et al\., 2024](https://arxiv.org/html/2610.18883#bib.bib1);[Gu et al\., 2024](https://arxiv.org/html/2610.18883#bib.bib8)\)\. The student generates the training trajectories, and the teacher supplies dense distributional supervision at the prefixes that the student actually visits\. This avoids the state\-distribution mismatch of offline distillation and has become an effective recipe for improving reasoning models\. Yet dense supervision does not imply that every supervised position is equally useful\. Some student and teacher distributions are already closely aligned; elsewhere they disagree, but the correction may require a large change in the student’s predictive distribution\. Uniform OPD, which weighs all token positions the same, does not distinguish between these cases\.

Recent work addresses this heterogeneity using properties of the visited positions\. Entropy\-based methods emphasize uncertain decisions\([Jin et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib11)\)\. TIP\([Xu et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib12)\)combines student uncertainty with teacher–student divergence; DEAR\([Xiao et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib2)\)separates decision tokens from supporting evidence tokens; Token Teachability\([Wang et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib15)\)asks whether teacher corrections lie within compatible student support; and TIDE\([Yu et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib14)\)distinguishes student\-excess from student\-deficit probability mass\. These approaches operationalize token importance through properties such as uncertainty, disagreement, reasoning role, support compatibility, or mismatch direction\. We take a different approach, defining importance through the optimization problem itself: how much can each position’s expected distillation objective improve under an equal local budget on predictive\-distribution change?

Suppose that every position is allowed the same infinitesimal change in the student’s next\-token distribution\. Which position permits the largest decrease in its expected distillation loss? This equal\-budget comparison separates the amount of disagreement from how effectively a local update can resolve it\. We formulate an independent local trust\-region problem at each student\-visited prefix\. Using the Fisher geometry induced by KL divergence, the maximum first\-order improvement under a common radiusε\\varepsilonis2​ε​st\\sqrt\{2\\varepsilon s\_\{t\}\}, where

st=gt⊤​Ft†​gt\.s\_\{t\}=g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\.Here,gtg\_\{t\}is the gradient of the token\-level distillation objective with respect to the student logits andFtF\_\{t\}is the categorical Fisher matrix\. For reverse\-KL OPD, the general score admits the closed form

st=Vara∼pS,t⁡\[log⁡pS,t​\(a\)pT,t​\(a\)\]\.s\_\{t\}=\\operatorname\{Var\}\_\{a\\sim p\_\{S,t\}\}\\left\[\\log\\frac\{p\_\{S,t\}\(a\)\}\{p\_\{T,t\}\(a\)\}\\right\]\.Thus, the variance of the log\-probability ratio is not introduced as a disagreement heuristic\. It is the optimization utility induced by an equal local distributional budget constraint\. We convert this score into Fisher Utility Weighting \(FUW\), which applies damped, mean\-normalized weights to the token\-level OPD objective\. Unlike selection schemes such as TIP and TIDE, which decide whether a position contributes, FUW uses the utility’s magnitude to set how much each position contributes, with weights proportional to the improvement scalest\\sqrt\{s\_\{t\}\}\. Since the utility is derived from the objective rather than chosen, it also provides a reference against which existing criteria can be compared\. We find that several of them concentrate importance on largely the same positions as the derived utility, and that the framework accounts for where they diverge\. Our contributions are threefold:

Main Contributions\.•We develop a common optimization framework for token utility in OPD\. Comparing the maximum first\-order loss reduction under an equal local KL budget yieldsst=gt⊤​Ft†​gts\_\{t\}=g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\. This utility reduces exactly to student\-weighted log\-ratio variance for reverse\-KL and toχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)for forward\-KL\. \(§[2](https://arxiv.org/html/2610.18883#S2),[3](https://arxiv.org/html/2610.18883#S3),[4](https://arxiv.org/html/2610.18883#S4),[5\.4](https://arxiv.org/html/2610.18883#S5.SS4)\)•On shared checkpoints, student entropy, KL disagreement, gradient norm, TIP Soft\-OR, and TIDE\-deficit retain nearly all of the derived utility’s top\-half importance mass, and conversely\. This suggests that these independently motivated criteria often identify similar high\-importance support despite their different constructions\. They diverge at stricter selection fractions and for the direction\-specific TIDE\-excess criterion\. \(§[5\.2](https://arxiv.org/html/2610.18883#S5.SS2)\)•We turn the utility into Fisher Utility Weighting \(FUW\) and evaluate it across seven mathematical reasoning benchmarks and three student–teacher pairs\. FUW improves macro mean accuracy over uniform OPD on all three pairs under the same candidate\-set construction and loss implementation\. Controlled studies examine the roles of score\-to\-position alignment, weight concentration, continuous weighting, and the distillation objective\. \(§[5\.1](https://arxiv.org/html/2610.18883#S5.SS1),[5\.3](https://arxiv.org/html/2610.18883#S5.SS3),[5\.4](https://arxiv.org/html/2610.18883#S5.SS4)\)

## 2Theory

The KL–Fisher expansion and the natural\-gradient solution to a Fisher\-constrained steepest\-descent problem are classical results\([Amari and Nagaoka, 2000](https://arxiv.org/html/2610.18883#bib.bib5);[Kakade, 2001](https://arxiv.org/html/2610.18883#bib.bib6)\)\. Our contribution is to view token importance in on\-policy distillation through the lens of local optimization: comparing positions under a common distributional budget yields objective\-specific utilities in closed form\. The trust region serves only to derive these per\-position utilities; it is not imposed as a constraint during training\.

### 2\.1Problem Formulation

On\-policy distillation \(OPD\) trains the student on trajectories sampled from its own policy while using a teacher model to provide token\-level supervision\. For a student\-generated trajectoryx=\(x1,…,xT\)∼pSx=\(x\_\{1\},\\ldots,x\_\{T\}\)\\sim p\_\{S\}, define the sampled token\-level reverse\-KL log\-ratio at positionttas

ℓt=log⁡pS​\(xt∣x<t\)pT​\(xt∣x<t\)\.\\ell\_\{t\}=\\log\\frac\{p\_\{S\}\(x\_\{t\}\\mid x\_\{<t\}\)\}\{p\_\{T\}\(x\_\{t\}\\mid x\_\{<t\}\)\}\.
The standard reverse\-KL OPD objective is

ℒOPD=𝔼x∼pS​\[∑t=1Tℓt\]\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=\\mathbb\{E\}\_\{x\\sim p\_\{S\}\}\\left\[\\sum\_\{t=1\}^\{T\}\\ell\_\{t\}\\right\]\.
Our goal is to determine how the optimization effort should be allocated across the token positions in this loss\. To assess the optimization opportunity at a particular student\-visited prefixht=x<th\_\{t\}=x\_\{<t\}, define

pS,t\(⋅\)=pS\(⋅∣ht\),pT,t\(⋅\)=pT\(⋅∣ht\)\.p\_\{S,t\}\(\\cdot\)=p\_\{S\}\(\\cdot\\mid h\_\{t\}\),\\qquad p\_\{T,t\}\(\\cdot\)=p\_\{T\}\(\\cdot\\mid h\_\{t\}\)\.Rather than assessing a position using only the particular tokenxtx\_\{t\}sampled on the current rollout, we consider the expected token\-level loss over the student’s next\-token distribution\. ForAt∼pS,tA\_\{t\}\\sim p\_\{S,t\},

Lt:=𝔼At∼pS,t​\[log⁡pS,t​\(At\)pT,t​\(At\)\]\.L\_\{t\}:=\\mathbb\{E\}\_\{A\_\{t\}\\sim p\_\{S,t\}\}\\left\[\\log\\frac\{p\_\{S,t\}\(A\_\{t\}\)\}\{p\_\{T,t\}\(A\_\{t\}\)\}\\right\]\.This expectation is exactly the full\-vocabulary reverse\-KL:

Lt=DKL\(pS,t∥pT,t\)\.L\_\{t\}=D\_\{\\mathrm\{KL\}\}\\left\(p\_\{S,t\}\\\|p\_\{T,t\}\\right\)\.
Thus,LtL\_\{t\}is the conditional expectation of the sampled token\-level OPD log\-ratio,

Lt=𝔼⁡\[ℓt∣ht\]\.L\_\{t\}=\\mathbb\{E\}\[\\ell\_\{t\}\\mid h\_\{t\}\]\.
LtL\_\{t\}marginalizes the next\-token sampling randomness while keeping the student\-visited prefix fixed\. We use the full\-distribution loss\{Lt\}t=1T\\\{L\_\{t\}\\\}\_\{t=1\}^\{T\}at each student visited position to determine how optimization effort should be allocated across these positions\.

Letzt∈ℝVz\_\{t\}\\in\\mathbb\{R\}^\{V\}denote the student logits at positiontt, and define

gt=∇ztLt\.g\_\{t\}=\\nabla\_\{z\_\{t\}\}L\_\{t\}\.A natural measure of token importance would be the Euclidean gradient magnitude‖gt‖2\\\|g\_\{t\}\\\|\_\{2\}\. However, this measures sensitivity per unit movement in logit coordinates\. Equal Euclidean perturbations of the logits can induce very different changes in the corresponding predictive distributions\.

We instead compare positions under an equal local change in the student’s predictive distribution\. Specifically, for each positiontt, we ask for the largest first\-order reduction in the expected token\-level loss that can be achieved while restricting the local change in the student distribution:

maxδt\\displaystyle\\max\_\{\\delta\_\{t\}\}\\quad−gt⊤​δt\\displaystyle\-g\_\{t\}^\{\\top\}\\delta\_\{t\}\(1\)subject toDKL\(pS,t\(⋅;zt\)∥pS,t\(⋅;zt\+δt\)\)≤ε\.\\displaystyle D\_\{\\mathrm\{KL\}\}\\left\(p\_\{S,t\}\(\\cdot;z\_\{t\}\)\\\|p\_\{S,t\}\(\\cdot;z\_\{t\}\+\\delta\_\{t\}\)\\right\)\\leq\\varepsilon\.\(2\)
The variableδt\\delta\_\{t\}is an infinitesimal logit perturbation\. The constraint compares positions by the induced change in their predictive distributions, rather than by an arbitrary Euclidean logit displacement\.

### 2\.2Maximum Improvement Within the Local KL Ball

Using the local Fisher approximation, the token\-specific optimization problem becomes

Δt⋆=maxδt−gt⊤​δtsubject toδt⊤​Ft​δt≤2​ε\.\\Delta\_\{t\}^\{\\star\}=\\max\_\{\\delta\_\{t\}\}\-g\_\{t\}^\{\\top\}\\delta\_\{t\}\\qquad\\text\{subject to\}\\qquad\\delta\_\{t\}^\{\\top\}F\_\{t\}\\delta\_\{t\}\\leq 2\\varepsilon\.
This is the classical Fisher\-constrained steepest\-descent problem\([Kakade, 2001](https://arxiv.org/html/2610.18883#bib.bib6)\)\. Its solution points in the natural\-gradient direction:

δt⋆=−2​εgt⊤​Ft†​gt​Ft†​gt,\\delta\_\{t\}^\{\\star\}=\-\\sqrt\{\\frac\{2\\varepsilon\}\{g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\}\}F\_\{t\}^\{\\dagger\}g\_\{t\},whereFt†F\_\{t\}^\{\\dagger\}is the Moore–Penrose pseudoinverse\. The pseudoinverse is required because the categorical Fisher is singular along the constant\-logit\-shift direction\. SinceLtL\_\{t\}depends on the logits only through softmax,gtg\_\{t\}lies in the range ofFtF\_\{t\}\. Whengt=0g\_\{t\}=0, we takeδt⋆=0\\delta\_\{t\}^\{\\star\}=0\.

The corresponding maximum first\-order improvement is

Δt⋆=−gt⊤​δt⋆=2​ε​gt⊤​Ft†​gt\.\\Delta\_\{t\}^\{\\star\}=\-g\_\{t\}^\{\\top\}\\delta\_\{t\}^\{\\star\}=\\sqrt\{2\\varepsilon\\,g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\}\.
Because every token position is evaluated under the same maximum radiusε\\varepsilon, the ordering induced byΔt⋆\\Delta\_\{t\}^\{\\star\}is identical to the ordering induced by the normalized score

st:=\(Δt⋆\)22​ε=gt⊤​Ft†​gt\.s\_\{t\}:=\\frac\{\(\\Delta\_\{t\}^\{\\star\}\)^\{2\}\}\{2\\varepsilon\}=g\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\.
We callsts\_\{t\}the*local token optimization utility*\. It measures the maximum first\-order improvement available at token positionttafter normalizing out the common maximum distributional budget\. The full Lagrangian derivation appears in Appendix[A\.2](https://arxiv.org/html/2610.18883#A1.SS2)\.

Importantly, this local geometry is determined entirely by the trust\-region constraint on the student distribution\. It does not depend on the direction of the teacher–student distillation divergence\. The choice of distillation objective enters through the gradientgg, while the local metric is determined byFF\.

## 3Closed\-Form Utility for Reverse\-KL OPD

We now specialize the general token utility to the reverse\-KL objective used in OPD\. The same framework applies to other distillation objectives through their objective\-specific gradientgg; we derive the corresponding forward\-KL utility in Appendix[A\.4](https://arxiv.org/html/2610.18883#A1.SS4)\.

Define the vocabulary\-wide log\-probability ratio

r⁡\(v\)=log⁡pS​\(v\)pT​\(v\)\.r\(v\)=\\log\\frac\{p\_\{S\}\(v\)\}\{p\_\{T\}\(v\)\}\.
###### Theorem 1\(Reverse\-KL Token Optimization Utility\)\.

For the token\-level reverse\-KL objective

L=DKL\(pS∥pT\),L=D\_\{\\mathrm\{KL\}\}\(p\_\{S\}\\\|p\_\{T\}\),the local token optimization utility satisfies

s=g⊤​F†​g=Varv∼pS⁡\[log⁡pS​\(v\)pT​\(v\)\]\.s=g^\{\\top\}F^\{\\dagger\}g=\\operatorname\{Var\}\_\{v\\sim p\_\{S\}\}\\left\[\\log\\frac\{p\_\{S\}\(v\)\}\{p\_\{T\}\(v\)\}\\right\]\.

#### Proof sketch\.

Letr∈ℝVr\\in\\mathbb\{R\}^\{V\}denote the vector with entriesrv=log⁡\(pS​\(v\)/pT​\(v\)\)r\_\{v\}=\\log\(p\_\{S\}\(v\)/p\_\{T\}\(v\)\)\. Differentiating the reverse\-KL with respect to the student logits gives

where

F=diag⁡\(pS\)−pS​pS⊤\.F=\\operatorname\{diag\}\(p\_\{S\}\)\-p\_\{S\}p\_\{S\}^\{\\top\}\.Therefore,

g⊤​F†​g\\displaystyle g^\{\\top\}F^\{\\dagger\}g=r⊤​F​F†​F​r=r⊤​F​r\.\\displaystyle=r^\{\\top\}FF^\{\\dagger\}Fr=r^\{\\top\}Fr\.\(3\)For the categorical Fisher,

r⊤​F​r\\displaystyle r^\{\\top\}Fr=∑vpS​\(v\)​rv2−\(∑vpS​\(v\)​rv\)2\\displaystyle=\\sum\_\{v\}p\_\{S\}\(v\)r\_\{v\}^\{2\}\-\\left\(\\sum\_\{v\}p\_\{S\}\(v\)r\_\{v\}\\right\)^\{2\}\(4\)=Varv∼pS⁡\[rv\]\.\\displaystyle=\\operatorname\{Var\}\_\{v\\sim p\_\{S\}\}\[r\_\{v\}\]\.\(5\)Substituting the definition ofrvr\_\{v\}gives the result\. A complete derivation is provided in Appendix[A\.3](https://arxiv.org/html/2610.18883#A1.SS3)\.

#### Practical computation\.

The utility can be computed directly inO⁡\(V\)O\(V\)without constructing or inverting the Fisher matrix:

s=∑vpS​\(v\)​\(rv−∑upS​\(u\)​ru\)2\.s=\\sum\_\{v\}p\_\{S\}\(v\)\\left\(r\_\{v\}\-\\sum\_\{u\}p\_\{S\}\(u\)r\_\{u\}\\right\)^\{2\}\.Since student and teacher logits are already available during distillation, this requires only elementwise operations and reductions\.

Importantly, variance is not posited as a heuristic measure of token importance; it arises from the Fisher\-constrained optimization problem\. Utility is high when student\-supported actions have sufficiently different teacher–student log\-ratios to permit substantial improvement through local probability redistribution, and low when those log\-ratios are nearly constant\.

#### Scope\.

The analysis is local to each student\-visited predictive distribution and does not model coupling through shared transformer parameters\. Thus, the derived utility characterizes local optimization opportunity rather than a globally optimal weighting rule; our end\-to\-end experiments test whether this signal remains useful under parameter coupling\.

## 4Token Importance under Reverse\-KL OPD

#### Relationship to existing criteria\.

Existing criteria capture different notions of token importance\. TIP develops an optimization\-motivated account but operationalizes its practical Soft\-OR criterion through student uncertainty and teacher–student disagreement, while TIDE distinguishes student\-excess from student\-deficit mismatch\. FUW instead derives a continuous weighting signal directly from the distillation objective throughgt⊤​Ft†​gtg\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}, which changes with the objective\. Table[4](https://arxiv.org/html/2610.18883#A2.T4)in Appendix[B\.2](https://arxiv.org/html/2610.18883#A2.SS2)summarizes the existing criteria and Section[5](https://arxiv.org/html/2610.18883#S5)compares their effective support empirically\.

#### Fisher Utility Weighting \(FUW\)

We now turn the reverse\-KL utility derived in Section[3](https://arxiv.org/html/2610.18883#S3)into a practical training rule\.[Theorem 1](https://arxiv.org/html/2610.18883#Thmtheorem1)defines Fisher utility on the full vocabulary\. For training, we use the candidate set

𝒞t=Top16⁡\(pS,t\)∪Top16⁡\(pT,t\),\|𝒞t\|≤32,\\mathcal\{C\}\_\{t\}=\\operatorname\{Top\}\_\{16\}\(p\_\{S,t\}\)\\cup\\operatorname\{Top\}\_\{16\}\(p\_\{T,t\}\),\\qquad\|\\mathcal\{C\}\_\{t\}\|\\leq 32,\(6\)and renormalize each distribution on this set\. Form∈\{S,T\}m\\in\\\{S,T\\\}andv∈𝒞tv\\in\\mathcal\{C\}\_\{t\}, let

p^m,t​\(v\)=pm,t​\(v\)∑u∈𝒞tpm,t​\(u\)\.\\widehat\{p\}\_\{m,t\}\(v\)=\\frac\{p\_\{m,t\}\(v\)\}\{\\sum\_\{u\\in\\mathcal\{C\}\_\{t\}\}p\_\{m,t\}\(u\)\}\.\(7\)We compute the restricted log\-ratio, clip it, and evaluate its student\-weighted variance:

rt𝒞​\(v\)=log⁡p^S,t​\(v\)p^T,t​\(v\),rtclip​\(v\)=clip⁡\(rt𝒞​\(v\),−c,c\),s^t=Varv∼p^S,t⁡\[rtclip​\(v\)\]\.r^\{\\mathcal\{C\}\}\_\{t\}\(v\)=\\log\\frac\{\\widehat\{p\}\_\{S,t\}\(v\)\}\{\\widehat\{p\}\_\{T,t\}\(v\)\},\\qquad r^\{\\mathrm\{clip\}\}\_\{t\}\(v\)=\\operatorname\{clip\}\\\!\\left\(r^\{\\mathcal\{C\}\}\_\{t\}\(v\),\-c,c\\right\),\\qquad\\widehat\{s\}\_\{t\}=\\operatorname\{Var\}\_\{v\\sim\\widehat\{p\}\_\{S,t\}\}\\\!\\left\[r^\{\\mathrm\{clip\}\}\_\{t\}\(v\)\\right\]\.\(8\)We setc=5c=5, so0≤s^t≤c20\\leq\\widehat\{s\}\_\{t\}\\leq c^\{2\}\. Without clipping,s^t\\widehat\{s\}\_\{t\}is the Fisher utility of the reverse\-KL loss on𝒞t\\mathcal\{C\}\_\{t\}, with the candidate set held fixed\. Clipping stabilizes this score and makes it an approximation to that utility\.

The per\-position training loss is the reverse\-KL on the same renormalized candidate set:

L^t=DKL​\(p^S,t∥p^T,t\)\.\\widehat\{L\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\widehat\{p\}\_\{S,t\}\\middle\\\|\\widehat\{p\}\_\{T,t\}\\right\)\.\(9\)For theNNvalid positions in a batch, we setut=s^tu\_\{t\}=\\sqrt\{\\widehat\{s\}\_\{t\}\}and use mean\-one weights

wt=\{N​ut∑j=1Nuj,∑j=1Nuj\>0,1,∑j=1Nuj=0\.w\_\{t\}=\\begin\{cases\}\\displaystyle\\frac\{Nu\_\{t\}\}\{\\sum\_\{j=1\}^\{N\}u\_\{j\}\},&\\sum\_\{j=1\}^\{N\}u\_\{j\}\>0,\\\\\[7\.0pt\] 1,&\\sum\_\{j=1\}^\{N\}u\_\{j\}=0\.\\end\{cases\}\(10\)The resulting objective is

ℒFUW=1N​∑t=1Nstopgrad⁡\(wt\)​L^t\.\\mathcal\{L\}\_\{\\mathrm\{FUW\}\}=\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\operatorname\{stopgrad\}\(w\_\{t\}\)\\widehat\{L\}\_\{t\}\.\(11\)Uniform OPD uses the same candidate set, renormalization, and per\-position loss, withwt=1w\_\{t\}=1for every position\. Thus, its comparison with FUW changes the weighting rule while holding this loss implementation fixed\.

The square\-root transform follows from the equal\-radius improvement scale,Δt⋆∝st\\Delta\_\{t\}^\{\\star\}\\propto\\sqrt\{s\_\{t\}\}, and also dampens score concentration\. We therefore useut=s^tu\_\{t\}=\\sqrt\{\\hat\{s\}\_\{t\}\}as a theoretically motivated weighting signal\. We normalize these utilities to have unit mean and treat the resulting weights as stop\-gradient during optimization\. Unless otherwise specified, Fisher Utility Weighting \(FUW\) refers to this reverse\-KL instantiation; Section[5\.4](https://arxiv.org/html/2610.18883#S5.SS4)applies the same construction to forward KL using its corresponding objective\-specific utility\.

## 5Experiments

Our experiments evaluate the derived token utility from these complementary perspectives: end\-to\-end distillation performance, its relationship to existing token\-importance criteria, and controlled variations of the scoring and weighting procedure\. We evaluate both reverse\- and forward\-KL OPD across three student–teacher pairs and seven mathematical\-reasoning benchmarks, using the objective\-specific utility induced by each distillation objective\. Additional concentration, robustness, and diagnostic analyses are reported in Appendix[B](https://arxiv.org/html/2610.18883#A2)\.

### 5\.1RQ1: Does FUW improve over uniform reverse\-KL OPD across model pairs?

We evaluate reverse\-KL OPD on three Qwen3 student–teacher pairs: Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B, Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B, and Qwen3\-4B\-Base←\\leftarrowQwen3\-30B\-A3B\-Instruct\-2507\. For all model pairs, training uses DAPO\-Math\-17k\([Yu et al\., 2025](https://arxiv.org/html/2610.18883#bib.bib16)\)for 279 steps \(one epoch\), 256 sequences per step, and an 8,192 tokens maximum training context with maximum response length set to 7,168 tokens and maximum prompt length set to 1,024 tokens\. FUW useswt∝s^tw\_\{t\}\\propto\\sqrt\{\\widehat\{s\}\_\{t\}\}\. Figure[1](https://arxiv.org/html/2610.18883#S5.F1)summarizes aggregate performance; per\-benchmark results along with details of the evaluation setup and baselines are in Appendix[B\.1](https://arxiv.org/html/2610.18883#A2.SS1)\.

Figure 1:Macro and pooled average gains over uniform OPD across three Qwen3 student–teacher pairs\. Bars show percentage\-point differences in the reported point estimates\. Macro weights the seven benchmarks equally; pooled weights all 1,620 problems equally\. Uniform OPD and FUW use the same candidate\-set construction and restricted reverse\-KL loss, differing in their token weights\. All methods share the same training setup\. Uniform OPD, TIP, and FUW use the same candidate\-set construction; TIDE uses its native support \(Appendix[B\.1](https://arxiv.org/html/2610.18883#A2.SS1)\)\.#### Benchmarks\.

We evaluate on 1,620 problems from seven mathematical\-reasoning benchmarks: AIME24 \(30\), AIME25 \(30\), AMC23 \(83\), BRUMO25 \(30\), MATH\-500 \(500\), Minerva \(272\), and OlympiadBench \(675\)\. AIME24/25, AMC23, and BRUMO25 form the four smaller benchmarks\. Evaluations use Macro Avg@8, which weights benchmarks equally, and pooled Avg@8, which weights problems equally\.

#### Results\.

FUW improves on uniform OPD on all three pairs\. Its Macro Avg@8 gains are4\.04\.0,3\.13\.1, and1\.351\.35percentage points for the 1\.7B←\\leftarrow8B, 1\.7B←\\leftarrow14B, and 4B←\\leftarrow30B\-A3B pairs, respectively\. The corresponding pooled gains are4\.64\.6,4\.74\.7, and1\.341\.34points\. Thus, the gain over uniform OPD persists across the evaluated model pairs, although its size varies\.

On the 1\.7B←\\leftarrow8B pair, FUW achieves the highest aggregate scores among the trained students:26\.826\.8Macro Avg@8 and39\.239\.2pooled Avg@8\. It exceeds TIP Soft\-OR by1\.51\.5and0\.70\.7points, respectively, and has the highest point estimate on five of the seven individual benchmarks among the trained methods\. The pooled FUW–TIP difference is small: its paired\-bootstrap 95% confidence interval includes zero\.

FUW also has the highest aggregate point estimates among trained methods on the 1\.7B←\\leftarrow14B pair\. On the 4B←\\leftarrow30B\-A3B pair, TIDE is slightly ahead: its Macro and pooled Avg@8 are46\.446\.4and58\.058\.0, compared with FUW’s46\.1546\.15and57\.5457\.54\. Thus, FUW consistently improves over matched\-implementation uniform OPD, while its differences from selective baselines are smaller and less uniform\.

### 5\.2RQ2: Do different token\-importance criteria prioritize the same positions?

We compare FUW with Euclidean gradient magnitude, reverse\-KL disagreement, student entropy, TIP Soft\-OR, and the excess and deficit components of TIDE; their definitions and relationships to FUW are summarized in Table[4](https://arxiv.org/html/2610.18883#A2.T4)in Appendix[B\.2](https://arxiv.org/html/2610.18883#A2.SS2)\. We compute all these token importance criteria on the checkpoints obtained during uniformly weighted reverse\-KL OPD of Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B\. Since we use uniformly weighted OPD the token\-importance criteria do not determine the model states being compared\.

Different importance criteria may disagree on the exact ordering of positions while still concentrating their weight on similar positions of the trajectory\. We therefore measure agreement in both directions\. LetSXρS\_\{X\}^\{\\rho\}denote the top\-ρ\\rhofraction of positions selected by criterionXX, and letwX,iw\_\{X,i\}denote its weight on positionii\. We define

REVρ​\(X\)=∑i∈SXρwFUW,i∑i∈SFUWρwFUW,i,CONTρ​\(X\)=∑i∈SFUWρwX,i∑i∈SXρwX,i\.\\mathrm\{REV\}\_\{\\rho\}\(X\)=\\frac\{\\sum\_\{i\\in S\_\{X\}^\{\\rho\}\}w\_\{\\mathrm\{FUW\},i\}\}\{\\sum\_\{i\\in S\_\{\\mathrm\{FUW\}\}^\{\\rho\}\}w\_\{\\mathrm\{FUW\},i\}\},\\qquad\\mathrm\{CONT\}\_\{\\rho\}\(X\)=\\frac\{\\sum\_\{i\\in S\_\{\\mathrm\{FUW\}\}^\{\\rho\}\}w\_\{X,i\}\}\{\\sum\_\{i\\in S\_\{X\}^\{\\rho\}\}w\_\{X,i\}\}\.
REV asks how much of FUW’s attainable top\-ρ\\rhoweight is retained when positions are selected byXX, while CONT asks the converse\. We summarize the two directions using

Bρ​\(X\)=min⁡\{REVρ​\(X\),CONTρ​\(X\)\},B\_\{\\rho\}\(X\)=\\min\\\{\\mathrm\{REV\}\_\{\\rho\}\(X\),\\mathrm\{CONT\}\_\{\\rho\}\(X\)\\\},which we call bidirectional weight retention\. Importantly, highBρB\_\{\\rho\}measures agreement in effective high\-importance support, not equality of the underlying weight vectors\. Two criteria can attainBρ≈1B\_\{\\rho\}\\approx 1while assigning different relative weights to individual positions within that shared support\.

![Refer to caption](https://arxiv.org/html/2610.18883v1/figures/criterion_equivalence_ovl_x1_pair.png)Figure 2:Effective\-support agreementBρB\_\{\\rho\}with FUW during uniform reverse\-KL OPD of Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B\. Atρ=0\.2\\rho=0\.2, agreement is criterion\- and checkpoint\-dependent; atρ=0\.5\\rho=0\.5, most criteria approach near\-complete retention, while TIDE\-excess remains distinct\. The random\-selection control accounts for agreement induced by the concentration of FUW’s weights\.We evaluateρ=0\.2\\rho=0\.2, close to TIDE’s realized active fraction \(0\.264\), andρ=0\.5\\rho=0\.5, matching TIP’s 50% Soft\-OR gate\. As shown in Figure[2](https://arxiv.org/html/2610.18883#S5.F2), atρ=0\.2\\rho=0\.2, agreement is substantial but criterion\- and checkpoint\-dependent, with gradient magnitude remaining particularly close to FUW\. Atρ=0\.5\\rho=0\.5, gradient magnitude, student entropy, KL disagreement, TIP Soft\-OR, and TIDE\-deficit achieveB0\.5≥0\.985B\_\{0\.5\}\\geq 0\.985from step 61 onward, whereas TIDE\-excess remains distinct\. Thus, for the Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B pair, several independently motivated criteria identify nearly the same effective high\-importance support at the broader operating fraction, although they may assign different relative weights within that support\. Appendix[B\.2](https://arxiv.org/html/2610.18883#A2.SS2)further analyzesBρ​\(X\)B\_\{\\rho\}\(X\)into its directional componentsREVρ​\(X\)\\mathrm\{REV\}\_\{\\rho\}\(X\)andCONTρ​\(X\)\\mathrm\{CONT\}\_\{\\rho\}\(X\)and defines the random\-control, showing that the near\-unitB0\.5B\_\{0\.5\}values for most criteria reflect nearly symmetric agreement, whereas TIDE\-excess exhibits asymmetric containment\. It also reports a cross\-pair robustness analysis on Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B along an FUW\-trained trajectory; although these checkpoints are not independent of the reference criterion, the same qualitative effective\-support structure appears at bothρ=0\.2\\rho=0\.2and0\.50\.5\.

### 5\.3RQ3: Which components of FUW matter?

These ablations use reverse\-KL distillation on the Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B pair with the same training setup as RQ1\. We separate the roles of the metric, position\-specific utility information, and weighting form\. Replacinggt⊤​Ft†​gtg\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}with the Euclidean score‖gt‖22\\\|g\_\{t\}\\\|\_\{2\}^\{2\}gives comparable downstream accuracy, although its weights become substantially more concentrated during training: effective retention falls to0\.1040\.104and the maximum token weight reaches9696\. FUW avoids this late\-stage concentration, although the two metrics are not separated by downstream accuracy at the present training budget\. Permuting FUW weights within each batch, while preserving their marginal distribution, lowers pooled Avg@8 by1\.021\.02points \(p=0\.028p=0\.028; Table[6](https://arxiv.org/html/2610.18883#A2.T6)\), showing that position\-specific utility information matters\. In contrast, a hard top\-half FUW mask reaches38\.438\.4pooled Avg@8, compared with39\.239\.2for continuous FUW and38\.438\.4for TIP Soft\-OR at the same 50% retention fraction\. This is consistent with RQ2, where FUW and TIP identify nearly the same effective support atρ=0\.5\\rho=0\.5\. The complete paired contrasts appear in Appendix[B\.3](https://arxiv.org/html/2610.18883#A2.SS3)\.

### 5\.4RQ4: How does the token\-utility framework extend to forward\-KL OPD?

The token\-utility framework extends directly to forward\-KL distillation: applying the same local trust\-region formulation yields the Pearsonχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)divergence as the induced utility, rather than the log\-ratio variance obtained for reverse\-KL\. A complete derivation of the forward\-KL utility is provided in Appendix[A\.4](https://arxiv.org/html/2610.18883#A1.SS4)\. For practical weighting, we upper\-clip the log\-ratior~=log⁡\(pT/pS\)\\tilde\{r\}=\\log\(p\_\{T\}/p\_\{S\}\)atτ=3\\tau=3, i\.e\.,pT/pS≤eτ≈20p\_\{T\}/p\_\{S\}\\leq e^\{\\tau\}\\approx 20, when computing the detached utility weights\. The utility is computed on the same candidate setCtC\_\{t\}, and weights are formed as in Eqs\. \(10\)–\(11\)\. The forward\-KL loss itself is unchanged\.

We test whether weighting forward\-KL OPD using its corresponding utility improves over uniform forward\-KL distillation\. We use the same experimental setup as in RQ1, changing only the objective\-specific components: the forward\-KL loss and its corresponding Pearsonχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)utility\. Uniform forward\-KL uses the same forward\-KL loss with uniform token weights\. Objective\-specific utility weighting improves both Macro and pooled Avg@8 on all three model pairs\. Averaged equally across the three model pairs, utility weighting improves Macro Avg@8 by 1\.65 points and pooled Avg@8 by 1\.95 points\. Detailed results are shown in Table[7](https://arxiv.org/html/2610.18883#A2.T7)in Appendix[B\.4](https://arxiv.org/html/2610.18883#A2.SS4)\.

Thus, the same local trust\-region principle developed in Section[2](https://arxiv.org/html/2610.18883#S2)yields a usable utility for a different objective: the Pearsonχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)divergence for forward\-KL, in place of the log\-ratio variance for reverse\-KL\. Weighting by the corresponding objective\-specific utility improves aggregate Avg@8 over uniform weighting under both reverse\- and forward\-KL distillation\.

## 6Related Work

#### On\-policy distillation\.

On\-policy distillation \(OPD\) trains a student model on trajectories sampled from its own policy while receiving token\-level supervision from a stronger teacher\([Agarwal et al\., 2024](https://arxiv.org/html/2610.18883#bib.bib1);[Gu et al\., 2024](https://arxiv.org/html/2610.18883#bib.bib8)\)\. Recent work has demonstrated that OPD substantially improves reasoning performance over offline distillation, while simultaneously exposing new optimization challenges unique to student\-generated trajectories\.[Luo et al\. \(2026\)](https://arxiv.org/html/2610.18883#bib.bib9)attribute instability to repetitive\-token advantage amplification, while[Li et al\. \(2026\)](https://arxiv.org/html/2610.18883#bib.bib3)characterize failure modes arising from signal degradation on drifted prefixes\. Other work has proposed stabilizing modifications including reward extrapolation\([Yang et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib4)\), relaxed clipping strategies\([Ko et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib10)\), and entropy\-aware training recipes\([Jin et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib11)\)\. Collectively, these studies suggest that the distillation signal is highly non\-uniform across token positions and motivate selective supervision strategies\.

#### Token importance and selective supervision\.

Entropy\-based methods prioritize uncertain positions\([Wang et al\., 2025](https://arxiv.org/html/2610.18883#bib.bib17);[Jin et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib11)\)\. TIP\([Xu et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib12)\)combines uncertainty and teacher–student divergence; DEAR\([Xiao et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib2)\)distinguishes decision and supporting evidence tokens; Token Teachability\([Wang et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib15)\)measures whether disagreement lies within compatible student support; and TIDE\([Yu et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib14)\)separates student\-excess from student\-deficit mass\. REOPOLD\([Ko et al\., 2026](https://arxiv.org/html/2610.18883#bib.bib10)\)instead treats token log\-ratios as rewards and combines clipping with entropy\-guided sampling\. FUW is complementary: it derives a score from achievable loss reduction under a common distributional budget rather than prescribing a token property\.

#### Information geometry and optimization\-based token utility\.

Our work is based on the information\-geometric view of optimization\. The local Fisher geometry induced by KL divergence and the resulting natural\-gradient formulation have a long history in machine learning\([Amari and Nagaoka, 2000](https://arxiv.org/html/2610.18883#bib.bib5);[Kakade, 2001](https://arxiv.org/html/2610.18883#bib.bib6)\)\. Trust\-region methods such as TRPO similarly constrain policy updates using KL divergence\([Schulman et al\., 2015](https://arxiv.org/html/2610.18883#bib.bib7)\), while PPO provides a practical approximation to trust\-region policy optimization\([Schulman et al\., 2017](https://arxiv.org/html/2610.18883#bib.bib13)\)\. Rather than applying these ideas to a shared parameter\-space update, we use the local KL geometry as an analytical lens for characterizing individual student\-visited positions\. Specifically, we define token utility as the maximum first\-order reduction in the token\-level distillation objective under a common local budget on predictive\-distribution change\. This yields the Fisher\-normalized quantitygt⊤​Ft†​gtg\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}, which reduces to student\-weighted log\-ratio variance for reverse KL and Pearsonχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)divergence for forward KL\. This provides a complementary perspective to recent notions based on uncertainty, disagreement, reasoning role, support compatibility, or mismatch type: rather than specifying a property that makes a token informative or learnable, we define the local optimization problem and let utility emerge from its solution\.

## 7Discussion and Conclusion

We derived token utility for on\-policy distillation \(OPD\) from the objective being optimized\. The local trust\-region formulation defines utility through the optimization opportunity induced by the distillation objective, yielding the general quantitygt⊤​Ft†​gtg\_\{t\}^\{\\top\}F\_\{t\}^\{\\dagger\}g\_\{t\}\. Under reverse\-KL this reduces to the student\-weighted variance of the log\-probability ratio, while under forward\-KL it becomes the Pearsonχ2\(pT∥pS\)\\chi^\{2\}\(p\_\{T\}\\\|p\_\{S\}\)divergence\. Across three student–teacher pairs, weighting by the corresponding utility improves aggregate performance over uniform weighting under both objectives\.

Distinct importance criteria can nevertheless identify similar effective high\-importance support\. Under reverse\-KL OPD, several criteria with different motivations exhibit near\-complete bidirectional retention with FUW at the broader operating fraction, while agreement is more criterion\- and checkpoint\-dependent at the more selective fraction and TIDE\-excess remains distinct\. Importantly, shared effective support does not imply identical weighting behavior, as the criteria can distribute importance differently within that support\.

Our analysis is local and first\-order: it evaluates predictive distributions independently at fixed student\-visited prefixes, whereas shared transformer parameters couple positions during training\. Accordingly, the derived utility characterizes local optimization opportunity rather than a globally optimal shared\-parameter weighting rule\. Our experiments are restricted to Qwen3 model pairs and mathematical reasoning, leaving broader architectures and agentic tasks for future study\.

Overall, the framework provides an optimization\-based construction for token utility in OPD: specify the distillation objective, derive its local utility under a common predictive\-distribution budget, and use that utility to characterize or prioritize student\-visited positions\.

## 8Acknowledgments

The authors thank Prof\. Tong Zhang of UIUC for his valuable feedback on an earlier version of this work\.

## AI Use Statement

The authors used LLM assistants to edit prose, check consistency between tables and text, and implement plotting functions for figures\. The authors designed the experiments, verified every reported number against the run artifacts, and take full responsibility for the content\.

## Ethics Statement

This work studies token weighting for on\-policy distillation of language models on publicly available mathematical\-reasoning data and benchmarks\. It involves no human subjects or personal data\. Distillation transfers capabilities from a larger teacher to a smaller student; the student may therefore inherit the teacher’s errors and biases, and more efficient distillation lowers the cost of deploying capable models\. We evaluate only mathematical reasoning and make no claims about safety\-relevant behavior of the distilled students\.

## Reproducibility Statement

Section[4](https://arxiv.org/html/2610.18883#S4)specifies the Fisher Utility Weighting \(FUW\) score, candidate\-set construction, clipping, and weighting\. Section[5\.1](https://arxiv.org/html/2610.18883#S5.SS1)and Appendix[B\.1](https://arxiv.org/html/2610.18883#A2.SS1)give the model pairs, training data, training budget, sampling settings, and evaluation protocol\. The Appendix[A](https://arxiv.org/html/2610.18883#A1)contains complete derivations for all theoretical results\. Each configuration is trained once; reported uncertainty reflects paired bootstrap resampling over evaluation problems\. For every run we retain the configuration, intermediate checkpoints, and raw evaluation outputs\. We will release code and configurations upon publication\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2306.13649)Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1)\.
- Amari and Nagaoka \(2000\)S\. Amari and H\. NagaokaMethods of information geometry\.Fields Institute Communications,American Mathematical Society\.External Links:ISBN 9780821805312,LCCN 00059362,[Link](https://books.google.com/books?id=qGzCwAEACAAJ)Cited by:[§A\.1](https://arxiv.org/html/2610.18883#A1.SS1.p32.1),[§2](https://arxiv.org/html/2610.18883#S2.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px3.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: on\-policy distillation of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1)\.
- Jinet al\.\(2026\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. LeeEntropy\-aware on\-policy distillation of language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p2.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Kakade \(2001\)S\. KakadeA natural policy gradient\.InProceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic,NIPS’01,Cambridge, MA, USA,pp\. 1531–1538\.Cited by:[§A\.2](https://arxiv.org/html/2610.18883#A1.SS2.p2.1),[§2\.2](https://arxiv.org/html/2610.18883#S2.SS2.p2.1),[§2](https://arxiv.org/html/2610.18883#S2.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px3.p1.1)\.
- Koet al\.\(2026\)J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. CameronScaling reasoning efficiently via relaxed on\-policy distillation\.arXiv:2603\.11137\.External Links:[Link](https://arxiv.org/abs/2603.11137)Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. DingRethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2026\)F\. Luo, Y\. Chuang, G\. Wang, Z\. Xu, X\. Han, T\. Zhang, and V\. BravermanDemystifying opd: length inflation and stabilization strategies for large language models\.arXiv preprint arXiv:2604\.08527\.Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1)\.
- Schulmanet al\.\(2015\)J\. Schulman, S\. Levine, P\. Moritz, M\. Jordan, and P\. AbbeelTrust region policy optimization\.InInternational Conference on Machine Learning,Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px3.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang, Y\. Liu, A\. Yang, A\. Zhao, Y\. Yue, S\. Song, B\. Yu, G\. Huang, and J\. LinBeyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.External Links:2506\.01939,[Link](https://arxiv.org/abs/2506.01939)Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, S\. Lu, Y\. Gu, P\. Wang, Y\. Yang, Z\. Yan, C\. Xie, J\. Wu, and H\. YangNot all disagreement is learnable: token teachability in on\-policy distillation\.arXiv: 2605\.26844\.Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p2.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Xiaoet al\.\(2026\)J\. Xiao, Z\. Han, Y\. Sun, Z\. Lu, Y\. Liu, Z\. Yao, W\. Chen, Q\. Gu, and X\. CaiFinding the evidence: discovering decision\-supporting tokens for on\-policy reasoning distillation\.arXiv:2606\.22830\.External Links:[Link](https://arxiv.org/abs/2606.22830)Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p2.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026\)Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. GeramifardTIP: token importance in on\-policy distillation\.arXiv preprint arXiv:2604\.14084\.External Links:[Link](https://arxiv.org/abs/2604.14084)Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p2.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026\)W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. LinLearning beyond teacher: generalized on\-policy distillation with reward extrapolation\.arXiv preprint arXiv:2602\.12125\.Cited by:[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. WangDAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§5\.1](https://arxiv.org/html/2610.18883#S5.SS1.p1.1)\.
- Yuet al\.\(2026\)Z\. Yu, C\. Yu, S\. Xu, Y\. Han, B\. Jiang, X\. Wang, and D\. ZouMismatch matters: on\-policy distillation beyond token agreement\.arXiv:2608\.09836\.External Links:[Link](https://arxiv.org/abs/2608.09836)Cited by:[§1](https://arxiv.org/html/2610.18883#S1.p2.1),[§6](https://arxiv.org/html/2610.18883#S6.SS0.SSS0.Px2.p1.1)\.

## Appendix AAdditional Theoretical Derivations

### A\.1Derivation of the Fisher\-Metric Constraint

In the main text we constrain the amount by which the student’s predictive distribution changes,

DKL​\(pS​\(⋅,z\)∥pS​\(⋅,z\+δ\)\)≤ε\.D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{S\}\(\\cdot;z\)\\,\\middle\\\|\\,p\_\{S\}\(\\cdot;z\+\\delta\)\\right\)\\leq\\varepsilon\.
This appendix derives the corresponding local quadratic approximation\.

Define

C⁡\(δ\)=DKL​\(pS​\(⋅,z\)∥pS​\(⋅,z\+δ\)\)\.C\(\\delta\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{S\}\(\\cdot;z\)\\,\\middle\\\|\\,p\_\{S\}\(\\cdot;z\+\\delta\)\\right\)\.
Using the definition of KL divergence,

C⁡\(δ\)=∑ipS​\(i,z\)​log⁡pS​\(i,z\)pS​\(i,z\+δ\)\.C\(\\delta\)=\\sum\_\{i\}p\_\{S\}\(i;z\)\\log\\frac\{p\_\{S\}\(i;z\)\}\{p\_\{S\}\(i;z\+\\delta\)\}\.
Since the first term is independent ofδ\\delta,

C\(δ\)=−∑ipS\(i;z\)logpS\(i;z\+δ\)\+const\.C\(\\delta\)=\-\\sum\_\{i\}p\_\{S\}\(i;z\)\\log p\_\{S\}\(i;z\+\\delta\)\+\\mathrm\{const\}\.
Taking the derivative with respect toδj\\delta\_\{j\},

∂C∂δj=−∑ipS\(i\)∂∂δjlogpS\(i;z\+δ\)\.\\frac\{\\partial C\}\{\\partial\\delta\_\{j\}\}=\-\\sum\_\{i\}p\_\{S\}\(i\)\\frac\{\\partial\}\{\\partial\\delta\_\{j\}\}\\log p\_\{S\}\(i;z\+\\delta\)\.
Using the softmax identity

∂log⁡p⁡\(i\)∂zj=𝟏\[i=j\]−p\(j\),\\frac\{\\partial\\log p\(i\)\}\{\\partial z\_\{j\}\}=\\mathbf\{1\}\[i=j\]\-p\(j\),
gives

∂C∂δj=pS​\(j,z\+δ\)−pS​\(j,z\)\.\\frac\{\\partial C\}\{\\partial\\delta\_\{j\}\}=p\_\{S\}\(j;z\+\\delta\)\-p\_\{S\}\(j;z\)\.
Evaluating at the expansion point,

yields

Similarly,

Therefore,

C⁡\(δ\)=12​δ⊤​H​δ\+O⁡\(‖δ‖3\),C\(\\delta\)=\\frac\{1\}\{2\}\\delta^\{\\top\}H\\delta\+O\(\\\|\\delta\\\|^\{3\}\),
where

H=∇δ2​C​\(0\)\.H=\\nabla\_\{\\delta\}^\{2\}C\(0\)\.
Differentiating once more,

Hi​j=∂pS​\(j\)∂zi\.H\_\{ij\}=\\frac\{\\partial p\_\{S\}\(j\)\}\{\\partial z\_\{i\}\}\.
Using the softmax derivative,

∂p⁡\(j\)∂zi=p\(j\)\(𝟏\[i=j\]−p\(i\)\),\\frac\{\\partial p\(j\)\}\{\\partial z\_\{i\}\}=p\(j\)\(\\mathbf\{1\}\[i=j\]\-p\(i\)\),
gives

H=diag⁡\(pS\)−pS​pS⊤\.H=\\operatorname\{diag\}\(p\_\{S\}\)\-p\_\{S\}p\_\{S\}^\{\\top\}\.
Recognizing this as the Fisher information matrix\([Amari and Nagaoka, 2000](https://arxiv.org/html/2610.18883#bib.bib5)\),

F=diag⁡\(pS\)−pS​pS⊤,F=\\operatorname\{diag\}\(p\_\{S\}\)\-p\_\{S\}p\_\{S\}^\{\\top\},
we obtain

DKL​\(pS​\(⋅,z\)∥pS​\(⋅,z\+δ\)\)=12​δ⊤​F​δ\+O⁡\(‖δ‖3\),D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{S\}\(\\cdot;z\)\\,\\middle\\\|\\,p\_\{S\}\(\\cdot;z\+\\delta\)\\right\)=\\frac\{1\}\{2\}\\delta^\{\\top\}F\\delta\+O\(\\\|\\delta\\\|^\{3\}\),
which is the local Fisher\-metric constraint used in the main text\.

### A\.2Solution of the Fisher\-Constrained Optimization

We derive the solution to

maxδ−g⊤​δsubject toδ⊤​F​δ≤2​ε\.\\max\_\{\\delta\}\\;\-g^\{\\top\}\\delta\\qquad\\text\{subject to\}\\qquad\\delta^\{\\top\}F\\delta\\leq 2\\varepsilon\.
HereFFis the Fisher information matrix of the categorical student distribution\. The derivation follows the classical Fisher\-constrained steepest\-descent argument of natural gradient methods\([Kakade, 2001](https://arxiv.org/html/2610.18883#bib.bib6)\)\. Since the softmax is invariant to constant logit shifts,FFis positive semi\-definite rather than positive definite\. Its null space contains the all\-ones vector𝟏\\mathbf\{1\}\. We therefore restrict the optimization to the simplex tangent space

𝒯=\{δ∈ℝV:𝟏⊤​δ=0\},\\mathcal\{T\}=\\left\\\{\\delta\\in\\mathbb\{R\}^\{V\}:\\mathbf\{1\}^\{\\top\}\\delta=0\\right\\\},on which the Fisher is positive definite when the student distribution has full support\. Equivalently, its inverse action in the ambient logit space is represented by the Moore–Penrose pseudoinverseF†F^\{\\dagger\}\.

Because the objective is linear and the feasible set is convex, a nonzero optimum lies on the boundary:

δ⊤​F​δ=2​ε\.\\delta^\{\\top\}F\\delta=2\\varepsilon\.
Introduce a Lagrange multiplierλ\\lambdaand define

𝒥⁡\(δ,λ\)=−g⊤​δ−λ⁡\(δ⊤​F​δ−2​ε\)\.\\mathcal\{J\}\(\\delta,\\lambda\)=\-g^\{\\top\}\\delta\-\\lambda\\left\(\\delta^\{\\top\}F\\delta\-2\\varepsilon\\right\)\.
The sign convention forλ\\lambdais arbitrary; we choose it so that the resulting optimizer points in the descent direction\. The first\-order stationarity condition with respect toδ\\deltais

∇δ𝒥\\displaystyle\\nabla\_\{\\delta\}\\mathcal\{J\}=−g−2​λ​F​δ=0\.\\displaystyle=\-g\-2\\lambda F\\delta=0\.
Hence,

F​δ=−12​λ​g\.F\\delta=\-\\frac\{1\}\{2\\lambda\}g\.
Since the gradient of the categorical KL satisfies𝟏⊤​g=0\\mathbf\{1\}^\{\\top\}g=0, it lies in the range ofFF\. Applying the pseudoinverse gives

δ=−12​λ​F†​g\+u,\\delta=\-\\frac\{1\}\{2\\lambda\}F^\{\\dagger\}g\+u,whereuulies in the null space ofFF\. A null\-space component changes all logits by a common shift and therefore changes neither the softmax distribution nor the objective\. We select the minimum\-norm representative by takingu=0u=0:

δ=−12​λ​F†​g\.\\delta=\-\\frac\{1\}\{2\\lambda\}F^\{\\dagger\}g\.\(12\)
Substituting Equation \([12](https://arxiv.org/html/2610.18883#A1.E12)\) into the active constraint gives

2​ε\\displaystyle 2\\varepsilon=δ⊤​F​δ\\displaystyle=\\delta^\{\\top\}F\\delta=14​λ2​g⊤​F†​F​F†​g\.\\displaystyle=\\frac\{1\}\{4\\lambda^\{2\}\}g^\{\\top\}F^\{\\dagger\}FF^\{\\dagger\}g\.
Becausegglies in the range ofFF,

F​F†​g=g,FF^\{\\dagger\}g=g,and therefore

2​ε=14​λ2​g⊤​F†​g\.2\\varepsilon=\\frac\{1\}\{4\\lambda^\{2\}\}g^\{\\top\}F^\{\\dagger\}g\.
Solving for the positive multiplier magnitude,

λ=12​g⊤​F†​g2​ε\.\\lambda=\\frac\{1\}\{2\}\\sqrt\{\\frac\{g^\{\\top\}F^\{\\dagger\}g\}\{2\\varepsilon\}\}\.
Substituting back into Equation \([12](https://arxiv.org/html/2610.18883#A1.E12)\) yields

δ⋆=−2​εg⊤​F†​g​F†​g\.\\boxed\{\\delta^\{\\star\}=\-\\sqrt\{\\frac\{2\\varepsilon\}\{g^\{\\top\}F^\{\\dagger\}g\}\}F^\{\\dagger\}g\}\.
Thus, the optimal perturbation points in the natural\-gradient direction−F†​g\-F^\{\\dagger\}g, with its scale chosen so that it exhausts the permitted local distributional budget\.

Finally, its first\-order improvement is

−g⊤​δ⋆\\displaystyle\-g^\{\\top\}\\delta^\{\\star\}=2​εg⊤​F†​g​g⊤​F†​g\\displaystyle=\\sqrt\{\\frac\{2\\varepsilon\}\{g^\{\\top\}F^\{\\dagger\}g\}\}g^\{\\top\}F^\{\\dagger\}g=2​ε​g⊤​F†​g\.\\displaystyle=\\boxed\{\\sqrt\{2\\varepsilon\\,g^\{\\top\}F^\{\\dagger\}g\}\}\.
For a common budgetε\\varepsilonacross token positions, the factor2​ε2\\varepsilonand the square root are common monotone transformations\. Consequently, ranking tokens by maximum local first\-order improvement is equivalent to ranking them by

g⊤​F†​g\.g^\{\\top\}F^\{\\dagger\}g\.

### A\.3Closed Form for Reverse\-KL Token Utility

For a fixed student\-visited prefix, let

pi=pS​\(i\),qi=pT​\(i\),p\_\{i\}=p\_\{S\}\(i\),\\qquad q\_\{i\}=p\_\{T\}\(i\),and define

ri=log⁡piqi\.r\_\{i\}=\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\.The reverse\-KL objective is

L=DKL\(p∥q\)=∑ipilogpiqi=∑ipiri\.L=D\_\{\\mathrm\{KL\}\}\(p\\\|q\)=\\sum\_\{i\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}=\\sum\_\{i\}p\_\{i\}r\_\{i\}\.
We first derive its gradient with respect to the student logitszz\. For the softmax distribution,

∂pi∂zj=pi\(𝟏\[i=j\]−pj\)\.\\frac\{\\partial p\_\{i\}\}\{\\partial z\_\{j\}\}=p\_\{i\}\(\\mathbf\{1\}\[i=j\]\-p\_\{j\}\)\.Moreover,

∂∂pi​\(pi​log⁡piqi\)=log⁡piqi\+1=ri\+1\.\\frac\{\\partial\}\{\\partial p\_\{i\}\}\\left\(p\_\{i\}\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\)=\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\+1=r\_\{i\}\+1\.Applying the chain rule,

gj=∂L∂zj\\displaystyle g\_\{j\}=\\frac\{\\partial L\}\{\\partial z\_\{j\}\}=∑i\(ri\+1\)​∂pi∂zj\\displaystyle=\\sum\_\{i\}\(r\_\{i\}\+1\)\\frac\{\\partial p\_\{i\}\}\{\\partial z\_\{j\}\}=∑i\(ri\+1\)pi\(𝟏\[i=j\]−pj\)\\displaystyle=\\sum\_\{i\}\(r\_\{i\}\+1\)p\_\{i\}\(\\mathbf\{1\}\[i=j\]\-p\_\{j\}\)=pj​\(rj\+1\)−pj​∑ipi​\(ri\+1\)\.\\displaystyle=p\_\{j\}\(r\_\{j\}\+1\)\-p\_\{j\}\\sum\_\{i\}p\_\{i\}\(r\_\{i\}\+1\)\.Since∑ipi=1\\sum\_\{i\}p\_\{i\}=1,

∑ipi​\(ri\+1\)=∑ipi​ri\+1\.\\sum\_\{i\}p\_\{i\}\(r\_\{i\}\+1\)=\\sum\_\{i\}p\_\{i\}r\_\{i\}\+1\.Therefore,

gj\\displaystyle g\_\{j\}=pj​\(rj−∑ipi​ri\)\\displaystyle=p\_\{j\}\\left\(r\_\{j\}\-\\sum\_\{i\}p\_\{i\}r\_\{i\}\\right\)=pj​\(rj−𝔼i∼p​\[ri\]\)\.\\displaystyle=p\_\{j\}\\left\(r\_\{j\}\-\\mathbb\{E\}\_\{i\\sim p\}\[r\_\{i\}\]\\right\)\.
Now recall that the categorical Fisher information matrix is

F=diag⁡\(p\)−p​p⊤\.F=\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\.MultiplyingFFbyrrgives

\(F​r\)j=pj​rj−pj​∑ipi​ri,\(Fr\)\_\{j\}=p\_\{j\}r\_\{j\}\-p\_\{j\}\\sum\_\{i\}p\_\{i\}r\_\{i\},which is exactly the expression above forgjg\_\{j\}\. Hence,

The Fisher\-constrained token utility derived in the main text is

s=g⊤​F†​g\.s=g^\{\\top\}F^\{\\dagger\}g\.Substitutingg=F​rg=Fr,

s\\displaystyle s=\(F​r\)⊤​F†​\(F​r\)\\displaystyle=\(Fr\)^\{\\top\}F^\{\\dagger\}\(Fr\)=r⊤​F​F†​F​r\\displaystyle=r^\{\\top\}FF^\{\\dagger\}Fr=r⊤​F​r,\\displaystyle=r^\{\\top\}Fr,where we use the Moore–Penrose identity

F​F†​F=F\.FF^\{\\dagger\}F=F\.
Expanding the categorical Fisher,

r⊤​F​r\\displaystyle r^\{\\top\}Fr=r⊤​\(diag⁡\(p\)−p​p⊤\)​r\\displaystyle=r^\{\\top\}\\left\(\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\\right\)r=∑ipi​ri2−\(∑ipi​ri\)2\\displaystyle=\\sum\_\{i\}p\_\{i\}r\_\{i\}^\{2\}\-\\left\(\\sum\_\{i\}p\_\{i\}r\_\{i\}\\right\)^\{2\}=𝔼i∼p​\[ri2\]−𝔼i∼p​\[ri\]2\\displaystyle=\\mathbb\{E\}\_\{i\\sim p\}\[r\_\{i\}^\{2\}\]\-\\mathbb\{E\}\_\{i\\sim p\}\[r\_\{i\}\]^\{2\}=Vari∼p⁡\[ri\]\.\\displaystyle=\\operatorname\{Var\}\_\{i\\sim p\}\[r\_\{i\}\]\.Substitutingri=log⁡\(pS​\(i\)/pT​\(i\)\)r\_\{i\}=\\log\(p\_\{S\}\(i\)/p\_\{T\}\(i\)\)therefore yields

s=Vari∼pS⁡\[log⁡pS​\(i\)pT​\(i\)\]\.\\boxed\{s=\\operatorname\{Var\}\_\{i\\sim p\_\{S\}\}\\left\[\\log\\frac\{p\_\{S\}\(i\)\}\{p\_\{T\}\(i\)\}\\right\]\.\}

### A\.4Closed Form for Forward\-KL Token Utility

The local Fisher geometry derived in Appendix A\.1 depends only on the constraint on changes to the student distribution\. Changing the distillation objective therefore changes the utility through its gradientgg, while the Fisher metric remains the same\.

Letp=pSp=p\_\{S\}andq=pTq=p\_\{T\}\. For the forward\-KL objective,

LFKL=DKL\(q∥p\)=∑iqilogqipi\.L\_\{\\mathrm\{FKL\}\}=D\_\{\\mathrm\{KL\}\}\(q\\\|p\)=\\sum\_\{i\}q\_\{i\}\\log\\frac\{q\_\{i\}\}\{p\_\{i\}\}\.Differentiating with respect to the student logits gives

Sinceppandqqare probability distributions,

𝟏⊤​g=0\.\\mathbf\{1\}^\{\\top\}g=0\.
From Appendix A\.2, the local token utility is

sFKL=g⊤​F†​g,F=diag⁡\(p\)−p​p⊤\.s\_\{\\mathrm\{FKL\}\}=g^\{\\top\}F^\{\\dagger\}g,\\qquad F=\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\.LetD=diag⁡\(p\)D=\\operatorname\{diag\}\(p\)\. Sincep⊤​D−1=𝟏⊤p^\{\\top\}D^\{\-1\}=\\mathbf\{1\}^\{\\top\},

F​D−1​g=\(D−p​p⊤\)​D−1​g=g−p⁡\(𝟏⊤​g\)=g\.FD^\{\-1\}g=\(D\-pp^\{\\top\}\)D^\{\-1\}g=g\-p\(\\mathbf\{1\}^\{\\top\}g\)=g\.ThusD−1​gD^\{\-1\}gis a solution to the Fisher systemF​v=gFv=g\. Any two solutions differ only by a null\-space component ofFF\. As established in Appendix A\.2, this component is a constant\-logit shift and is orthogonal togg\. Therefore,

g⊤​F†​g=g⊤​D−1​g=∑igi2pi\.g^\{\\top\}F^\{\\dagger\}g=g^\{\\top\}D^\{\-1\}g=\\sum\_\{i\}\\frac\{g\_\{i\}^\{2\}\}\{p\_\{i\}\}\.Substitutinggi=pi−qig\_\{i\}=p\_\{i\}\-q\_\{i\}gives

sFKL=∑i\(qi−pi\)2pi=χ2\(q∥p\)\.\\boxed\{s\_\{\\mathrm\{FKL\}\}=\\sum\_\{i\}\\frac\{\(q\_\{i\}\-p\_\{i\}\)^\{2\}\}\{p\_\{i\}\}=\\chi^\{2\}\(q\\\|p\)\}\.
Thus, under the same local predictive\-distribution geometry, reverse\-KL and forward\-KL induce different token utilities: reverse\-KL yields the student\-weighted variance of the log\-probability ratio, whereas forward KL yields the Pearsonχ2\\chi^\{2\}divergence\.

### A\.5Exact Finite\-Radius Utility for Reverse\-KL

Relation to the local Fisher utility\. The preceding derivations define token utility through a local quadratic approximation to the predictive\-distribution KL constraint\. We now ask whether this quantity also corresponds to an exact distribution\-space optimization problem before taking the local approximation\. For reverse\-KL, we solve the finite\-radius problem exactly and show that the Fisher utility is recovered as its normalized infinitesimal limit\.

At one student\-visited prefix, write

pi=pS​\(i,z\),qi=pT​\(i\),xi​\(δ\)=pS​\(i,z\+δ\),p\_\{i\}=p\_\{S\}\(i;z\),\\qquad q\_\{i\}=p\_\{T\}\(i\),\\qquad x\_\{i\}\(\\delta\)=p\_\{S\}\(i;z\+\\delta\),and assumepi,qi\>0p\_\{i\},q\_\{i\}\>0\. Since softmax can represent every distribution in the interior of the simplex, optimizing overδ\\deltaup to a common logit shift is equivalent to

minxDKL\(x∥q\)subject toDKL\(p∥x\)≤ε,∑ixi=1\.\\min\_\{x\}\\quad D\_\{\\mathrm\{KL\}\}\(x\\\|q\)\\qquad\\text\{subject to\}\\qquad D\_\{\\mathrm\{KL\}\}\(p\\\|x\)\\leq\\varepsilon,\\quad\\sum\_\{i\}x\_\{i\}=1\.The objective and feasible set are convex inxx, so the optimizer is unique\. IfDKL\(p∥q\)≤εD\_\{\\mathrm\{KL\}\}\(p\\\|q\)\\leq\\varepsilon, thenx⋆=qx^\{\\star\}=q\. Otherwise, the constraint is active\.

For the active case, introduce multipliersλ\>0\\lambda\>0andν\\nu:

𝒥⁡\(x,λ,ν\)\\displaystyle\\mathcal\{J\}\(x,\\lambda,\\nu\)=∑ixi​log⁡xiqi\+λ⁡\(∑ipi​log⁡pixi−ε\)\+ν⁡\(∑ixi−1\)\.\\displaystyle=\\sum\_\{i\}x\_\{i\}\\log\\frac\{x\_\{i\}\}\{q\_\{i\}\}\+\\lambda\\left\(\\sum\_\{i\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{x\_\{i\}\}\-\\varepsilon\\right\)\+\\nu\\left\(\\sum\_\{i\}x\_\{i\}\-1\\right\)\.Stationarity with respect toxix\_\{i\}gives

log⁡xiqi\+1\+ν−λ​pixi=0\.\\log\\frac\{x\_\{i\}\}\{q\_\{i\}\}\+1\+\\nu\-\\lambda\\frac\{p\_\{i\}\}\{x\_\{i\}\}=0\.\(13\)Letui=λ​pi/xiu\_\{i\}=\\lambda p\_\{i\}/x\_\{i\}andα=λ​e1\+ν\\alpha=\\lambda e^\{1\+\\nu\}\. Equation \([13](https://arxiv.org/html/2610.18883#A1.E13)\) then implies

ui​eui=α​piqi\.u\_\{i\}e^\{u\_\{i\}\}=\\alpha\\frac\{p\_\{i\}\}\{q\_\{i\}\}\.Using the principal Lambert functionW0W\_\{0\}, defined byW0​\(y\)​eW0​\(y\)=yW\_\{0\}\(y\)e^\{W\_\{0\}\(y\)\}=y, yields

ui=W0​\(α​piqi\)\.u\_\{i\}=W\_\{0\}\\\!\\left\(\\alpha\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\)\.Normalizingxxtherefore gives

xi⋆​\(α\)=pi/W0​\(α​pi/qi\)∑jpj/W0​\(α​pj/qj\)\.\\boxed\{x\_\{i\}^\{\\star\}\(\\alpha\)=\\frac\{p\_\{i\}/W\_\{0\}\\\!\\left\(\\alpha p\_\{i\}/q\_\{i\}\\right\)\}\{\\displaystyle\\sum\_\{j\}p\_\{j\}/W\_\{0\}\\\!\\left\(\\alpha p\_\{j\}/q\_\{j\}\\right\)\}\}\.\(14\)The scalarα\>0\\alpha\>0is chosen so that

DKL​\(p∥x⋆​\(α\)\)=ε\.D\_\{\\mathrm\{KL\}\}\\left\(p\\middle\\\|x^\{\\star\}\(\\alpha\)\\right\)=\\varepsilon\.\(15\)Asα↓0\\alpha\\downarrow 0,x⋆​\(α\)→qx^\{\\star\}\(\\alpha\)\\to q, while asα↑∞\\alpha\\uparrow\\infty,x⋆​\(α\)→px^\{\\star\}\(\\alpha\)\\to p\. Thus Equation \([15](https://arxiv.org/html/2610.18883#A1.E15)\) is a one\-dimensional root problem for0<ε<DKL\(p∥q\)0<\\varepsilon<D\_\{\\mathrm\{KL\}\}\(p\\\|q\)\.

Becausexi∝pi​eδix\_\{i\}\\propto p\_\{i\}e^\{\\delta\_\{i\}\}, an exact optimal logit perturbation is

δi⋆=−log⁡W0​\(α​piqi\)\+c,\\boxed\{\\delta\_\{i\}^\{\\star\}=\-\\log W\_\{0\}\\\!\\left\(\\alpha\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\)\+c\},where the common shiftccdoes not affect the softmax distribution\.

The exact finite\-radius improvement is

U\(ε\):=DKL\(p∥q\)−DKL\(x⋆∥q\)\.U\(\\varepsilon\):=D\_\{\\mathrm\{KL\}\}\(p\\\|q\)\-D\_\{\\mathrm\{KL\}\}\(x^\{\\star\}\\\|q\)\.To connect it to the local utility, letri=log⁡\(pi/qi\)r\_\{i\}=\\log\(p\_\{i\}/q\_\{i\}\)\. Asε↓0\\varepsilon\\downarrow 0, the exact optimizer satisfiesx⋆=p\+O⁡\(ε\)x^\{\\star\}=p\+O\(\\sqrt\{\\varepsilon\}\)\. The objective and constraint expansions in logit space give

DKL\(p∥x\(δ\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(p\\\|x\(\\delta\)\)=12​δ⊤​F​δ\+O⁡\(‖δ‖3\),\\displaystyle=\\frac\{1\}\{2\}\\delta^\{\\top\}F\\delta\+O\(\\\|\\delta\\\|^\{3\}\),DKL\(x\(δ\)∥q\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(x\(\\delta\)\\\|q\)=DKL\(p∥q\)\+g⊤δ\+O\(∥δ∥2\)\.\\displaystyle=D\_\{\\mathrm\{KL\}\}\(p\\\|q\)\+g^\{\\top\}\\delta\+O\(\\\|\\delta\\\|^\{2\}\)\.Applying the Fisher\-constrained solution and usingg⊤​F†​g=Vari∼p⁡\[ri\]g^\{\\top\}F^\{\\dagger\}g=\\operatorname\{Var\}\_\{i\\sim p\}\[r\_\{i\}\]yields

U⁡\(ε\)=2​ε​Vari∼p​\[log⁡piqi\]\+O⁡\(ε\)\.U\(\\varepsilon\)=\\sqrt\{2\\varepsilon\\,\\operatorname\{Var\}\_\{i\\sim p\}\\left\[\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\]\}\+O\(\\varepsilon\)\.Consequently,

s=limε↓0U​\(ε\)22​ε=Vari∼p⁡\[log⁡piqi\]\.\\boxed\{s=\\lim\_\{\\varepsilon\\downarrow 0\}\\frac\{U\(\\varepsilon\)^\{2\}\}\{2\\varepsilon\}=\\operatorname\{Var\}\_\{i\\sim p\}\\left\[\\log\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\]\}\.
Thus,ssis not merely a quantity produced by the quadratic Fisher approximation\. It is the radius\-independent coefficient governing the leading\-order improvement of the exact constrained problem\. At finiteϵ\\epsilon, however, the exact utilityU⁡\(ϵ\)U\(\\epsilon\)depends on the chosen radius and need not preserve the ranking induced byss\.

## Appendix BAdditional Experimental Studies

This appendix collects the controlled studies that complement the main end\-to\-end comparison\. Unless stated otherwise, the student, teacher, data, training budget, generation settings, and paired bootstrap procedure match Section[5](https://arxiv.org/html/2610.18883#S5)\.

### B\.1Reverse\-KL Evaluation Results Across Model Pairs

#### Evaluation\.

We sample at temperature0\.70\.7and top\-p=0\.95p=0\.95\. For each problem, we generateKb=16K\_\{b\}=16responses on AIME24, AIME25, AMC23, and BRUMO25, andKb=8K\_\{b\}=8responses on MATH\-500, Minerva, and OlympiadBench\. The accuracy of problemiiin benchmarkbbis the fraction of itsKbK\_\{b\}responses that are correct, denotedab,ia\_\{b,i\}\. We report

Macro=17​∑b=171nb​∑i=1nbab,i,Pooled=11620​∑b=17∑i=1nbab,i,\\operatorname\{Macro\}=\\frac\{1\}\{7\}\\sum\_\{b=1\}^\{7\}\\frac\{1\}\{n\_\{b\}\}\\sum\_\{i=1\}^\{n\_\{b\}\}a\_\{b,i\},\\qquad\\operatorname\{Pooled\}=\\frac\{1\}\{1620\}\\sum\_\{b=1\}^\{7\}\\sum\_\{i=1\}^\{n\_\{b\}\}a\_\{b,i\},\(16\)wherenbn\_\{b\}is the number of problems in benchmarkbb\. Thus, Macro weights benchmarks equally, while Pooled weights problems equally regardless of their number of sampled responses\. We estimate aggregate uncertainty and paired contrasts using 20,000 paired bootstrap resamples over problems\.

#### Baselines\.

Uniform OPD uses the same candidate\-set construction, renormalized reverse\-KL loss, and training setup as FUW, withwt=1w\_\{t\}=1at every position\. Each candidate set is the union of the student’s top\-16 and teacher’s top\-16 tokens\. On average we observe the union consists of about 19 tokens\. We also compare against TIDE using its native support and TIP Soft\-OR at 50% retention using FUW’s candidate\-set construction\.

End\-to\-end mathematical reasoning results for the different model pairs are shown in Tables[1](https://arxiv.org/html/2610.18883#A2.T1),[2](https://arxiv.org/html/2610.18883#A2.T2)and[3](https://arxiv.org/html/2610.18883#A2.T3)\. Macro averages benchmarks equally; Pooled averages all 1,620 problems\. Best trained\-student result in each column isbold\.

Table 1:Qwen3\-1\.7B\-Base distilled from Qwen3\-8BTable 2:Qwen3\-4B\-Base distilled from Qwen3\-30B\-A3B\-Instruct\-2507Table 3:Qwen3\-1\.7B\-Base distilled from Qwen3\-14B

### B\.2Token\-Importance Agreement Analysis

#### Token\-importance criteria\.

Table[4](https://arxiv.org/html/2610.18883#A2.T4)summarizes the criteria considered in our agreement analysis and their relationship to the reverse\-KL Fisher utility\. KL disagreement and the squared Euclidean gradient norm are analytically related to the same log\-ratio underlying FUW, whereas student entropy, TIP Soft\-OR, and TIDE characterize different properties of the position\.

Table 4:Token\-importance criteria used in the agreement analysis\. Hereppandqqdenote the student and teacher next\-token distributions at a fixed prefix,ri=log⁡\(pi/qi\)r\_\{i\}=\\log\(p\_\{i\}/q\_\{i\}\), andr¯=𝔼i∼p​\[ri\]\\bar\{r\}=\\mathbb\{E\}\_\{i\\sim p\}\[r\_\{i\}\]\. The first three rows have the stated analytical relationships; the remaining criteria characterize different properties of the position\.
#### Agreement analysis\.

We compute all criteria on checkpoints obtained during uniformly weighted reverse\-KL OPD of Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B\. Since none of the importance criteria determines this training trajectory, it provides a controlled setting for analyzing criterion agreement\.

The main\-text analysis summarizes the two directions using bidirectional weight retention,Bρ​\(X\)=min⁡\{REVρ​\(X\),CONTρ​\(X\)\}B\_\{\\rho\}\(X\)=\\min\\\{\\mathrm\{REV\}\_\{\\rho\}\(X\),\\mathrm\{CONT\}\_\{\\rho\}\(X\)\\\}\. Here, we decomposeBρB\_\{\\rho\}into its directional components to determine whether high effective\-support agreement is symmetric or instead reflects one\-sided containment\. Importantly, high bidirectional weight retention measures agreement in effective high\-importance support, not equality of the underlying weight vectors\. Two criteria can attainBρ≈1B\_\{\\rho\}\\approx 1while assigning different relative weights to individual positions within that shared support\. This distinction is reflected in the substantial differences in score concentration across criteria shown in Table[5](https://arxiv.org/html/2610.18883#A2.T5)\.

Table 5:Fraction of positions needed to account for a given fraction of score mass\.LetSXρS\_\{X\}^\{\\rho\}denote the top\-ρ\\rhofraction of positions according to criterionXX, and letwX,iw\_\{X,i\}denote the corresponding importance weight assigned to positionii\. As defined in Section[5\.2](https://arxiv.org/html/2610.18883#S5.SS2)

REVρ​\(X\)=∑i∈SXρwFUW,i∑i∈SFUWρwFUW,i,\\mathrm\{REV\}\_\{\\rho\}\(X\)=\\frac\{\\sum\_\{i\\in S\_\{X\}^\{\\rho\}\}w\_\{\\mathrm\{FUW\},i\}\}\{\\sum\_\{i\\in S\_\{\\mathrm\{FUW\}\}^\{\\rho\}\}w\_\{\\mathrm\{FUW\},i\}\},measures how much of FUW’s attainable top\-ρ\\rhoweight is retained when positions are instead selected byXX, and

CONTρ​\(X\)=∑i∈SFUWρwX,i∑i∈SXρwX,i,\\mathrm\{CONT\}\_\{\\rho\}\(X\)=\\frac\{\\sum\_\{i\\in S\_\{\\mathrm\{FUW\}\}^\{\\rho\}\}w\_\{X,i\}\}\{\\sum\_\{i\\in S\_\{X\}^\{\\rho\}\}w\_\{X,i\}\},measures the converse\. Their difference,

Aρ​\(X\)=CONTρ​\(X\)−REVρ​\(X\),A\_\{\\rho\}\(X\)=\\mathrm\{CONT\}\_\{\\rho\}\(X\)\-\\mathrm\{REV\}\_\{\\rho\}\(X\),characterizes directional asymmetry\. Positive values indicate that FUW’s selected set captures more ofXX’s attainable weight thanXX’s selected set captures of FUW’s; negative values indicate the reverse\. Values near zero indicate approximately symmetric retention\.

#### Random selection controls\.

To distinguish criterion agreement from retention expected under random selection, we use a permutation control that preserves the weight multiset while destroying the correspondence between positions and scores\. Let

Dρ=∑i∈SFUWρwFUW,i∑iwFUW,iD\_\{\\rho\}=\\frac\{\\sum\_\{i\\in S^\{\\rho\}\_\{\\mathrm\{FUW\}\}\}w\_\{\\mathrm\{FUW\},i\}\}\{\\sum\_\{i\}w\_\{\\mathrm\{FUW\},i\}\}denote the fraction of FUW’s total weight contained in its top\-ρ\\rhopositions\. Since each position is included in a uniformly random top\-ρ\\rhoset with probabilityρ\\rho,

𝔼⁡\[REVρ,random\]=ρDρ\.\\mathbb\{E\}\[\\mathrm\{REV\}\_\{\\rho,\\mathrm\{random\}\}\]=\\frac\{\\rho\}\{D\_\{\\rho\}\}\.Thus, the control accounts for retention induced by the selection fraction and the concentration of FUW’s weight distribution\. The measured permutation control agrees with this analytic expectation to within0\.0030\.003at all 13 checkpoints\.

Figures[3](https://arxiv.org/html/2610.18883#A2.F3)and[4](https://arxiv.org/html/2610.18883#A2.F4)report the decomposition ofBρB\_\{\\rho\}atρ=0\.2\\rho=0\.2andρ=0\.5\\rho=0\.5, corresponding to the 20% and 50% retention fractions\. The former is close to TIDE’s realized active fraction in our implementation, while the latter matches the 50% Soft\-OR configuration recommended by TIP\.

Atρ=0\.2\\rho=0\.2, the two directions can differ materially for student entropy and TIP Soft\-OR, with asymmetry reaching approximately0\.180\.18over the latter half of training\. These differences coincide with changes in the relative concentration of the criteria’s weights, so we do not interpret them as evidence of directional containment\.

Atρ=0\.5\\rho=0\.5, Euclidean gradient magnitude, student entropy, KL disagreement, TIP Soft\-OR, and TIDE\-deficit exhibit nearly symmetric agreement with FUW: their two directional retention scores remain close throughout the later checkpoints\. Their near\-unitB0\.5B\_\{0\.5\}values therefore reflect high retention in both directions\. TIDE\-excess exhibits a qualitatively different pattern\. Its asymmetry is large and positive, indicating that FUW’s top\-half set captures substantially more of TIDE\-excess’s attainable weight than the TIDE\-excess set captures of FUW’s\. By the later checkpoints, FUW’s selected set captures nearly all of the high\-weight support identified by TIDE\-excess, while the converse retention is substantially lower\. Thus, unlike the near\-symmetric agreement observed for the other criteria, the lowerB0\.5B\_\{0\.5\}of TIDE\-excess reflects asymmetric effective\-support containment\. This behavior is consistent with the distinct role of TIDE\-excess, which explicitly isolates one direction of student–teacher mismatch rather than measuring scalar disagreement, uncertainty, or local optimization opportunity\.

![Refer to caption](https://arxiv.org/html/2610.18883v1/figures/criterion_containment_ovl_x1_rho02_row.png)Figure 3:Directional decomposition atρ=0\.2\\rho=0\.2\.REV measures the fraction of FUW’s attainable top\-ρ\\rhoweight retained when positions are selected by criterionXX, while CONT measures the converse\. Directional agreement is more sensitive to differences in weight concentration at this operating fraction\.![Refer to caption](https://arxiv.org/html/2610.18883v1/figures/criterion_containment_ovl_x1_rho05_row.png)Figure 4:Directional decomposition atρ=0\.5\\rho=0\.5\.Most criteria exhibit nearly symmetric, near\-complete retention with FUW, whereas TIDE\-excess exhibits asymmetric effective\-support containment\.Overall, for the Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-14B pair studied under uniform OPD, several independently motivated criteria concentrate importance on largely the same effective support as the Fisher\-derived utility atρ=0\.5\\rho=0\.5\. For gradient magnitude, student entropy, KL disagreement, TIP Soft\-OR, and TIDE\-deficit, this relationship is nearly symmetric, indicating shared effective high\-importance support rather than one\-sided containment\. TIDE\-excess is the notable exception: its high\-importance support is largely contained within FUW’s broader support, while the converse does not hold\. These results concern effective support rather than point\-wise equivalence of the weighting rules\.

#### Cross\-pair robustness\.

We additionally evaluate bidirectional weight retention on a second student–teacher pair using checkpoints generated during FUW training\. Unlike the uniformly weighted trajectory above, these checkpoints are not independent of the reference criterion, since FUW itself determines the training trajectory\. We therefore treat this experiment as a cross\-pair robustness check rather than an independent test of criterion agreement\.

The similar qualitative structure observed earlier is preserved across both operating fractions as shown in Figure[5](https://arxiv.org/html/2610.18883#A2.F5)\. Atρ=0\.5\\rho=0\.5, gradient magnitude, student entropy, KL disagreement, TIP Soft\-OR, and TIDE\-deficit exhibit near\-complete bidirectional retention with FUW throughout most of training, whereas TIDE\-excess remains substantially lower\. Atρ=0\.2\\rho=0\.2, the same separation emerges after the earliest checkpoints: the scalar criteria approach high effective\-support agreement with FUW, while TIDE\-excess remains distinctly separated\. In both cases, agreement is substantially above the random\-selection control\. Thus, the effective\-support structure observed under uniform OPD for the primary model pair is qualitatively reproduced on a different model pair and training trajectory\.

![Refer to caption](https://arxiv.org/html/2610.18883v1/figures/criterion_equivalence_p3_pair.png)Figure 5:Cross\-pair robustness of effective\-support agreement\. Bidirectional weight retentionBρB\_\{\\rho\}with FUW on Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B along an FUW\-trained trajectory, atρ=0\.2\\rho=0\.2\(left\) andρ=0\.5\\rho=0\.5\(right\)\. Beyond the early training steps, most criteria show near\-complete effective\-support agreement with FUW at both retention fractions, while TIDE\-excess remains distinctly lower\.

### B\.3Weighting, metric, and ranking controls

All controls in this subsection use the same re\-normalized union of the student’s and teacher’s top\-16 candidates obtained during the reverse\-KL distillation of the Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B pair\. They therefore isolate the scoring or weighting rule without changing candidate support\. Table[6](https://arxiv.org/html/2610.18883#A2.T6)reports paired pooled\-score differences relative to FUW\. A negative value favors FUW\.

Table 6:Paired controls on the union top\-16 support for reverse\-KL OPD of Qwen3\-1\.7B\-Base←\\leftarrowQwen3\-8B pair\. The Euclidean variant replaces the Fisher utility by‖gt‖22\\\|g\_\{t\}\\\|\_\{2\}^\{2\}; the permutation control randomly reassigns FUW weights within each batch\. NegativeΔ\\Deltafavors FUW\.The Euclidean variant reaches pooled Avg@8 of38\.938\.9, close to FUW’s39\.239\.2, with no significant paired difference \(p=0\.527p=0\.527\)\. However, its weighting dynamics differ substantially from FUW\. Effective retention decreases from0\.3270\.327early in training to0\.1040\.104by step 118, while the maximum token weight rises from2929to9696\. The additional factor ofpSp\_\{S\}implicit in the Euclidean score increasingly concentrates weight on high\-probability student outcomes\. Fisher normalization avoids this late\-stage concentration, although the two metrics are not separated by downstream accuracy at the present training budget\.

The permutation control preserves the complete multiset of FUW weights and changes only their assignment to positions\. Its significant reduction \(−1\.02\-1\.02,p=0\.028p=0\.028\) shows that position\-specific utility information contributes beyond the marginal weight distribution: the positions FUW identifies are not interchangeable\. Hard selection does the opposite, preserving which positions FUW identifies as high utility while discarding their relative weights\. Used as a top\-half mask, the FUW score reaches38\.438\.4pooled Avg@8, matching TIP Soft\-OR at the same 50% retention fraction \(38\.438\.4\)\. This matches the effective\-support analysis in Section[5\.2](https://arxiv.org/html/2610.18883#S5.SS2): the two criteria identify nearly the same support atρ=0\.5\\rho=0\.5\. Continuous FUW reaches39\.239\.2; on this pair, its difference from the hard mask is not significant \(p=0\.087p=0\.087\)\.

### B\.4Forward\-KL Evaluation Results Across Model Pairs

Table[7](https://arxiv.org/html/2610.18883#A2.T7)reports the per\-benchmark results for the forward\-KL experiments summarized in Section[5\.4](https://arxiv.org/html/2610.18883#S5.SS4)\.

Table 7:Forward\-KL mathematical\-reasoning performance across three student–teacher pairs\. Benchmark columns report Avg@k, while aggregate columns report macro\- and pooled\-average Avg@k\. Values are reported as mean±\\pmstandard error\. The best result within each model pair is shown in bold\.