Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillation

# Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillation
Source: [https://arxiv.org/html/2607.16246](https://arxiv.org/html/2607.16246)
Jiangan Yuan Baidu Inc\. jianganyuan@link\.cuhk\.edu\.cn&Zhixuan Li11footnotemark:1 Baidu Inc\. lizhixuan\.2017@tsinghua\.org\.cn&Han Xu Baidu Inc\. xhbj66@gmail\.com

###### Abstract

Off\-policy distillation is now central to large language model pre\-training, yet how training data, objective parameterization, and model capabilities interact remains poorly characterized\. We studies top\-kk\-truncated, temperature\-scaled off\-policy distillation by decomposing this problem into two questions: an*objective\-to\-capability*analysis of how the training objective shapes token\-level supervision and downstream performance, and a*data\-to\-objective*analysis of how data heterogeneity should inform objective routing\. We first show that the language\-modeling objective \(LLML\_\{\\mathrm\{LM\}\}\) and the knowledge\-distillation objective \(LKDL\_\{\\mathrm\{KD\}\}\) induce systematically different capability profiles, and trace this divergence to a gradient\-level tension between*direct observed\-token reinforcement*and*teacher\-supported alternative supervision*\. To quantify this tension, we introduce diagnostic metrics—support coverage, observed\-token probability mass, and teacher\-distribution concentration—and show via controlled sweeps that the support sizekkgoverns a coverage–sharpness trade\-off, while distillation temperature controls within\-support probability allocation\. We then examine adaptive objective routing: a domain\-level policy that appliesLLML\_\{\\mathrm\{LM\}\}to math and code andLKDL\_\{\\mathrm\{KD\}\}to general\-domain data yields consistent gains over both single\-objective baselines, whereas token\-level routing based on observed\-token probability mass or teacher entropy fails to consistently match the single\-objective baseline\. These results suggest that effective objective routing depends less on routing granularity than on the quality of the routing signal, reframing continued pre\-training via off\-policy distillation as a structured, data\-conditional supervision\-design problem rather than a global hyperparameter choice\.

## 1Introduction

Off\-policy distillation has become an increasingly important ingredient in large language model \(LLM\) pre\-training\. In this setting, a student model is trained on a fixed corpus while receiving supplementary supervision from a larger, frozen teacher model, typically in the form of soft next\-token distributions over the vocabulary\. Compared with standard language\-modeling \(LM\) training on one\-hot targets, knowledge distillation \(KD\) can expose the student to the teacher’s distributional uncertainty over plausible continuations—an effect commonly attributed to the “dark knowledge” encoded in soft probability distributions\(Hintonet al\.,[2015](https://arxiv.org/html/2607.16246#bib.bib1)\)—rather than grounding training exclusively on the single observed token\. These properties make off\-policy distillation attractive for improving data efficiency, transferring capabilities from stronger models, and recovering performance after structural compression\. Recent industrial model families have demonstrated that distillation can serve not merely as a post\-training compression tool but as a central mechanism for constructing smaller and more capable base models\(Teamet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib3); Gemma Teamet al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib4); Muralidharanet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib5); Liuet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib6); Tanget al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib7)\)\.

Despite its practical importance, off\-policy distillation remains insufficiently characterized as a training objective\. Existing work has produced a range of useful empirical recipes, but two broader limitations persist\. First, most analyses treat data composition, objective choice, sparse teacher\-target construction, and downstream benchmark behavior as separate design dimensions, even though they jointly determine the token\-level supervision signal that the student receives at each training position\. This fragmentation makes it difficult to reason systematically about how the training objective interacts with data heterogeneity to shape downstream model capabilities\. Second, many reported findings remain primarily phenomenological: they document that adjusting a loss weight, a top\-kkcutoff, a temperature parameter, or a routing heuristic changes benchmark scores, but offer limited explanation of which token\-level properties of the supervision signal are responsible or why the effect takes the observed form\. As a result, off\-policy distillation is often treated as a global recipe to be tuned empirically rather than as a structured, data\-dependent training objective whose effects can be systematically analyzed and anticipated\.

These limitations manifest in three concrete ways\. The first concerns the relationship between training objective and model capability\. Most distillation pipelines apply a fixed interpolation between LM and KD losses uniformly across all token positions\. Some work reports that pure KD can outperform LM–KD mixtures in certain settings\(Muralidharanet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib5); Liuet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib6)\), while others study static or dynamic weighting schedules\(Penget al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib8)\)\. These findings establish that the LM–KD balance is a consequential design choice, but evaluations are typically reported as aggregate benchmark scores\. They do not identify which capability dimensions each objective preferentially strengthens, whether benchmark performance diverges systematically across task types under fixed global objectives, or whether a single global objective necessarily incurs an avoidable capability trade\-off across heterogeneous tasks\.

The second limitation concerns the parameterization of the KD objective itself\. Modern LLM vocabularies make full\-vocabulary teacher logits prohibitively expensive to store and transmit at scale, so practical systems typically construct sparse teacher targets via top\-kktruncation or sampling, followed by temperature scaling and renormalization\(Gemma Teamet al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib4); Penget al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib8); Goyalet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib2)\)\. These choices directly determine the support, sharpness, and probability allocation of the target distribution, and therefore materially affect the optimization signal that the student receives at each position\. Existing work has explored sweeps overkkor temperature and reported the resulting downstream benchmark scores, but without a detailed account of how each parameter reshapes the token\-level supervision landscape or why its downstream effect takes the observed form\.

The third limitation concerns the relationship between data heterogeneity and objective routing\. Pre\-training corpora are intrinsically heterogeneous, and different domains, examples, or individual token positions may benefit from different training objectives\. Recent work has begun to explore token\-level routing policies, using teacher entropy, distribution sharpness, or related difficulty signals to determine when to apply KD and when to default to standard LM supervision\(Penget al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib8); Goyalet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib2); Zhanget al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib9); Xieet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib10); Huanget al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib16)\)\. However, the design of routing signals is rarely analyzed in depth\. In particular, it remains unclear whether teacher\-side statistics such as entropy are sufficient for reliable routing, or whether effective objective routing requires a joint measure of alignment between the teacher distribution and the observed training token\.

This paper studies continued pre\-training via off\-policy distillation through the interplay of data heterogeneity, training objective, and downstream capability\. We decompose the problem into two questions\. The first is an*objective\-to\-capability*question: given the same training data and model initialization, how do LM and KD supervision differ at the token level, and how do KD\-specific parameters such as top\-kkand temperature reshape the resulting capability profile? The second is a*data\-to\-objective*question: given a heterogeneous training corpus, can different data subsets or individual token positions be routed to different objectives to better align supervision with the underlying training distribution, and what properties make a routing signal reliable? Together, these questions reframe continued pre\-training via off\-policy distillation not as a monolithic hyperparameter choice but as a problem of structured, data\-conditional supervision design\.

We address both questions through a combination of gradient\-level analysis, diagnostic measurement, controlled hyperparameter sweeps, and objective routing experiments\. We begin by directly comparing a standard LM\-trained student with an off\-policy KD\-trained student under otherwise identical training conditions\. The results reveal that neither objective uniformly dominates: the two objectives induce systematically distinct capability profiles\. LM supervision is consistently stronger on high\-difficulty reasoning, mathematical problem solving, and knowledge\-intensive evaluations, with the performance gap amplified substantially under Pass@KKevaluation at larger sampling budgets\. KD supervision is more favorable on benchmarks associated with commonsense plausibility, factual retrieval, local reading comprehension, and structured program synthesis\. This systematic, benchmark\-dependent divergence demonstrates that a fixed global objective commits to a particular capability trade\-off rather than being uniformly optimal across heterogeneous tasks, directly motivating a principled analysis of where and why the two objectives differ\.

To explain this divergence, we analyze the two objectives at the gradient level\. Standard LM training concentrates all supervision on the observed token, while top\-kk\-truncated KD distributes target mass across the teacher support, either reinforcing or suppressing the observed\-token update depending on whether and to what degree the teacher assigns probability mass to that token\. This analysis identifies two complementary training signals—*direct observed\-token reinforcement*and*teacher\-supported alternative supervision*—whose relative balance governs the capability trade\-off between the objectives\. To quantify this balance at each training position, we introduce a suite of diagnostic metrics: observed\-token coverage by the teacher support, observed\-token rank, observed\-token mass \(OTMass\), conditional observed\-token mass \(CondOTMass\), and teacher\-support entropy\. These metrics characterize how the sparse teacher target departs from the hard LM target along each dimension of the supervision signal and serve as instruments for connecting token\-level optimization statistics to downstream benchmark behavior\.

Using these diagnostics, we conduct controlled sweeps over the two principal degrees of freedom in sparse KD: support sizekkand distillation temperatureτ\\tau\. The support size governs a coverage–sharpness trade\-off: small supports produce sharp, concentrated supervision but frequently exclude the observed token, while large supports reduce coverage gaps at the cost of a progressively more diffuse target\. Temperature, by contrast, does not alter support membership; at fixedkk, it controls within\-support probability allocation, sharpening the target toward the teacher’s top\-ranked prediction at low temperatures and redistributing mass toward lower\-ranked alternatives at high temperatures\. Across standard benchmarks and Pass@KKevaluations, the effects of these two parameters are qualitatively distinct and task\-dependent, consistent with the interpretation thatkkandτ\\taujointly position the supervision signal along the spectrum between the two complementary factors identified above, and that different tasks favor different positions along this spectrum\.

We then investigate whether objective routing can be made adaptive to the training data\. At the coarsest granularity, a domain\-based routing policy that applies LM supervision to mathematics and code data and KD supervision to general\-domain data yields consistent aggregate improvements over both single\-objective baselines: it largely recovers the LM advantage on reasoning\-intensive benchmarks and Pass@KKevaluations, preserves the KD advantage on knowledge\-oriented and commonsense benchmarks, and in several cases—including MBPP and the AIME benchmarks—exceeds both baselines simultaneously\. Analysis of the diagnostic metrics stratified by domain provides a mechanistic account of this behavior: mathematics and code data exhibit substantially higher observed\-token mass and lower teacher\-support entropy than general\-domain data, indicating that domain labels serve as a reliable proxy for teacher–data alignment rather than merely indexing surface\-level content categories\.

Finally, we compare domain\-level routing with finer\-grained token\-level routing policies that use OTMass and teacher entropy directly as per\-token routing signals\. Despite operating at finer granularity, neither token\-level policy consistently matches the stronger single\-objective baseline, and both fall substantially short of domain\-level routing on high\-difficulty reasoning evaluations\. Between the two token\-level signals, OTMass is the more reliable routing criterion: it directly measures the teacher probability mass assigned to the observed training token and thereby captures teacher–data alignment, whereas teacher entropy characterizes only the concentration of the teacher distribution independently of the observed token and cannot distinguish a teacher that agrees with the data from one that confidently predicts an alternative\. These results suggest that routing effectiveness need not increase monotonically with granularity, and that reliable objective routing may instead call for signals that jointly reflect the teacher distribution and its alignment with the observed training token—a criterion that teacher\-side uncertainty measures alone do not appear to satisfy\.

## 2Related Work

### 2\.1Off\-policy Distillation in Pre\-training

Knowledge distillation has long been used to compress pretrained language models, and several early works already operate in an off\-policy manner by training a student on a fixed corpus with a frozen teacher\. DistilBERT introduces a pre\-training recipe that combines masked\-language modeling, soft\-target distillation, and cosine representation matching, demonstrating that distillation can be applied during pre\-training rather than solely at downstream fine\-tuning time\(Sanhet al\.,[2019](https://arxiv.org/html/2607.16246#bib.bib17)\)\. TinyBERT extends this approach with a two\-stage framework that distills both general\-domain pre\-training knowledge and task\-specific knowledge through Transformer\-layer supervision\(Jiaoet al\.,[2020](https://arxiv.org/html/2607.16246#bib.bib18)\)\. MobileBERT, MiniLM, and MiniLMv2 further show that architecture\-aware distillation and self\-attention relation transfer can preserve much of the teacher quality with fewer layers and lower latency\(Sunet al\.,[2020](https://arxiv.org/html/2607.16246#bib.bib19); Wanget al\.,[2020](https://arxiv.org/html/2607.16246#bib.bib20),[2021](https://arxiv.org/html/2607.16246#bib.bib21)\)\. Although these works focus primarily on encoder\-style PLMs rather than decoder\-only LLMs, they establish the foundational template of offline teacher–student transfer on a fixed corpus\.

More recent work has revisited this paradigm at the scale of modern LLM pre\-training\. The Gemma series offers a prominent industrial\-scale demonstration\. Gemma 2 trains its 2B and 9B models with knowledge distillation, motivating this choice as a richer training signal capable of providing denser gradients than the observed token alone\(Teamet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib3)\)\. Gemma 3 further extends this recipe to all released model sizes and reports a practical sparse logit\-sampling implementation for efficient teacher\-target construction\(Gemma Teamet al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib4)\)\.

A related line of work combines structured pruning with distillation to produce model families spanning multiple deployment scales\. Minitron demonstrates that a pretrained dense model can be pruned and retrained with knowledge distillation using only a small fraction of the original training data, and its empirical study emphasizes that distillation is critical for performance recovery after pruning\(Muralidharanet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib5)\)\. Ministral 3 follows a similar compression\-oriented direction through cascade distillation\(Liuet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib6)\), and SlimQwen extends this theme to MoE compression\(Tanget al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib7)\)\. These works position distillation not only as a standalone pre\-training objective but also as a recovery mechanism that makes aggressive architecture compression feasible\.

Beyond industrial recipes, recent academic studies have begun to analyze the design space and behavioral consequences of pre\-training distillation\.Penget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)systematically investigate top\-pp–top\-kktruncation for reducing logit storage, temperature normalization of teacher distributions, static and scheduled mixtures of LM and KD losses, and adaptive temperature variants\.Goyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\)offer a complementary mechanistic perspective by analyzing distilled pre\-training through the lens of in\-context learning and test\-time scaling, showing that distillation can improve test\-time scaling by increasing generation diversity while simultaneously impairing in\-context learning at low\-entropy positions\. Together, these works motivate a closer examination of how off\-policy distillation reshapes token\-level supervision during pre\-training\.

### 2\.2KL Objectives for Off\-policy Distillation

The standard logits\-based distillation objective traces back to the soft\-target formulation ofHintonet al\.\([2015](https://arxiv.org/html/2607.16246#bib.bib1)\)\. The student is trained to match the softened teacher distribution, typically in combination with a hard\-label cross\-entropy term\. In token\-level language\-model pre\-training, this idea is most commonly instantiated as a forward KL loss over next\-token distributions\. LetpT\(⋅∣x<r\)p\_\{T\}\(\\cdot\\mid x\_\{<r\}\)andpS\(⋅∣x<r\)p\_\{S\}\(\\cdot\\mid x\_\{<r\}\)denote the teacher and student distributions, respectively\. A generic off\-policy distillation objective can be written as

L=\(1−α\)LLM\+αLKD,LKD=1T∑t=1TDKL\(p~T\(⋅∣x<r\)∥pS\(⋅∣x<r\)\),L=\(1\-\\alpha\)L\_\{\\mathrm\{LM\}\}\+\\alpha L\_\{\\mathrm\{KD\}\},\\qquad L\_\{\\mathrm\{KD\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\tilde\{p\}\_\{T\}\(\\cdot\\mid x\_\{<r\}\)\\,\\\|\\,p\_\{S\}\(\\cdot\\mid x\_\{<r\}\)\\right\),wherep~T\\tilde\{p\}\_\{T\}denotes the processed teacher target distribution after operations such as sampling, scaling and renormalization\.

Most large\-scale pre\-training distillation recipes adopt forward KL as the default token\-level objective, differing primarily in how it is combined with the LM loss\.Penget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)explicitly formulate pre\-training distillation as a weighted combination ofLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}, and explore schedules for the mixing coefficientα\\alpha\.Goyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\)adopt a similar weighted objective to study distilled pre\-training under modern LLM scaling conditions\. By contrast, several pruning\-and\-distillation recipes find that pure logits distillation can be preferable following model compression\(Muralidharanet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib5); Liuet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib6)\)\. These findings suggest that the LM–KD mixing ratio is not merely an implementation detail but a substantive design choice whose optimal value depends on the training regime, model initialization, and teacher–student capacity gap\.

Although forward KL is the dominant choice in off\-policy pre\-training, several works study alternative divergence formulations in adjacent LLM distillation settings\. MiniLLM argues that forward KL can be suboptimal for open\-ended generation because it may force a capacity\-limited student to cover too many teacher modes, and instead optimizes reverse KL with a policy\-gradient\-style procedure\(Guet al\.,[2024](https://arxiv.org/html/2607.16246#bib.bib14)\)\. TAID takes a different approach by constructing an adaptive intermediate distribution between the student and teacher that is progressively interpolated over the course of training\(Shinget al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib15)\)\. These methods are developed primarily for post\-training or general LLM distillation rather than the pre\-training setting studied here; we therefore focus on the standard forward KL objective while treating these works as evidence that the choice and parameterization of the distillation target can materially affect student behavior\.

Full\-vocabulary teacher logits are expensive to store and transfer in practice since modern LLM vocabularies often contain hundreds of thousands of tokens\. Off\-policy distillation consequently relies on sparse approximations of the teacher distribution\. One line of work uses deterministic truncation:Penget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)apply a top\-pp\-then\-top\-kktruncation strategy before renormalizing the retained distribution\. LFM2 introduces a top\-kkteacher support with a decoupled objective that separates a binary KL term, which matches the total probability mass assigned to the teacher top\-kkset, from a conditional KL term, which matches relative probabilities within that set\(Aminiet al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib12)\)\. A complementary line of work samples sparse teacher targets rather than selecting the deterministic top\-kkset\. Gemma 3 samples 256 logits per token weighted by teacher probabilities, assigns zero probability to non\-sampled entries, and renormalizes the resulting distribution\(Gemma Teamet al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib4)\)\.Goyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\)also study sparse\-label distillation in whichkklogits are sampled from the teacher distribution and renormalized\.Anshumannet al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib11)\)offer a more principled critique of deterministic top\-kkdistillation, arguing that it yields a biased estimate of the teacher distribution by discarding tail information, and propose random sampling with importance weighting\.Dasguptaet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib13)\)further argue that standard KL is dominated by the teacher’s highest\-probability modes, and propose a tail\-aware divergence that decouples top\-kkand tail contributions\.

Temperature is another central factor governing the shape of the distillation target\. In the classical formulation ofHintonet al\.\([2015](https://arxiv.org/html/2607.16246#bib.bib1)\), the same temperature is applied to both teacher and student logits, and the soft\-target loss is multiplied byτ2\\tau^\{2\}to compensate for the temperature\-dependent gradient scale\. In LLM pre\-training distillation, however, temperature is typically employed as part of teacher\-target construction: teacher logits are rescaled, then truncated or sampled, renormalized, and stored as sparse soft labels\.Goyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\)apply temperature to teacher logits when constructing sparse labels, andPenget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)systematically study both static and adaptive teacher\-temperature variants based on distribution sharpness\. Together, these works indicate that sparse support size and temperature are two coupled but distinct degrees of freedom in the design of off\-policy distillation targets\.

### 2\.3Training Objective Routing

Most distillation methods apply the training objective uniformly across all token positions\. The standard formulation interpolates the hard\-label language\-modeling objective and the distillation objective as

L=\(1−α\)​LLM\+α​LKD,L=\(1\-\\alpha\)L\_\{\\mathrm\{LM\}\}\+\\alpha L\_\{\\mathrm\{KD\}\},whereα\\alphais applied uniformly at every token position\.

Prior work has primarily adjusted the global mixture ratio or the training\-stage schedule\.Penget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)explore scheduled weighting between LM and KD losses in pre\-training distillation, treating the balance between hard\-label and teacher supervision as a time\-dependent design choice\. TAID achieves a form of temporal routing by replacing direct teacher matching with an adaptive intermediate distribution that evolves over the course of training\(Shinget al\.,[2025](https://arxiv.org/html/2607.16246#bib.bib15)\)\. These approaches move beyond a fixed global objective but still assign the same objective form to every token within a training step or stage\.

A finer\-grained line of work performs token\-conditional routing\.Goyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\)observe that distillation can harm in\-context learning by softening low\-entropy, near\-deterministic token mappings, and propose entropy\-based routing that removes the distillation term from the lowest\-entropy positions while retaining standard hard\-label supervision there\.Penget al\.\([2025](https://arxiv.org/html/2607.16246#bib.bib8)\)also examine adaptive temperature rules based on distributional sharpness—measured by standard deviation or entropy—thereby treating the softness of the distillation target as a token\-dependent quantity\.Zhanget al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib9)\)use teacher entropy as a token\-level difficulty signal and jointly adapt three components of distillation: the KL weight, the temperature, and the distillation path; their method emphasizes low\-entropy tokens early in training and gradually shifts supervision toward high\-entropy tokens, while also assigning token\-specific temperatures and applying deeper feature and attention distillation to difficult tokens\.Xieet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib10)\)instead measure token difficulty by the Hellinger distance between teacher and student distributions: their LATF module dynamically selects tokens for the distillation loss, while their IDTS module assigns lower temperatures to difficult tokens and higher temperatures to easier ones\.Huanget al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib16)\)further reframe distillation as a token\-acceptance problem in which the student proposes tokens, the teacher verifies them through top\-kkor Spec\-kkacceptance, and only accepted tokens receive the full distillation loss while rejected tokens are masked or down\-weighted\. Collectively, these studies suggest that objective routing is becoming a central degree of freedom in distillation design, alongside sparse support construction and temperature scaling\. However, the majority of fine\-grained routing methods have been developed and evaluated in post\-training or instruction\-distillation settings; fine\-grained objective routing during large\-scale off\-policy pre\-training remains comparatively underexplored, leaving open how token\- or subset\-level objective choices should be made when the training corpus, teacher targets, and downstream capabilities are all heterogeneous\.

## 3Preliminaries

Consider a training sample represented as a token sequence:

x=\(x1,x2,…,xn\)\.x=\(x\_\{1\},x\_\{2\},\\dots,x\_\{n\}\)\.\(1\)Under autoregressive language modeling, the model predicts each target tokenxrx\_\{r\}conditioned on the preceding contextx<rx\_\{<r\}\. We consider a large teacher modelfTf\_\{T\}and a smaller student modelfSf\_\{S\}, both evaluated on the same sequence\. ForM∈\{T,S\}M\\in\\\{T,S\\\}, modelfMf\_\{M\}produces a logit vector over the vocabulary𝒱\\mathcal\{V\}at each positionrr:

zr\(M\)=fM​\(x<r\)∈ℝ\|𝒱\|,M∈\{T,S\}\.z\_\{r\}^\{\(M\)\}=f\_\{M\}\(x\_\{<r\}\)\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\},\\quad M\\in\\\{T,S\\\}\.\(2\)Here,zr,i\(M\)z\_\{r,i\}^\{\(M\)\}denotes the logit assigned by modelMMto tokeni∈𝒱i\\in\\mathcal\{V\}at positionrr\.

### 3\.1Top\-kkTruncation and Temperature\-Scaled Teacher Probabilities

Storing and transferring full teacher logit vectors incurs substantial overhead due to the large vocabulary size\. To mitigate this, we apply top\-kktruncation to the teacher outputs\. Specifically, at each positionrr, we define the teacher top\-kksupport as

Kr\(T\)=TopK⁡\(zr\(T\),k\),K\_\{r\}^\{\(T\)\}=\\operatorname\{TopK\}\(z\_\{r\}^\{\(T\)\},k\),\(3\)whereKr\(T\)⊆𝒱K\_\{r\}^\{\(T\)\}\\subseteq\\mathcal\{V\}is the index set of thekklargest teacher logits at positionrr\.

The teacher logits are then restricted to this support:

z^r,i\(T\)=\{zr,i\(T\),i∈Kr\(T\),−∞,i∉Kr\(T\)\.\\hat\{z\}\_\{r,i\}^\{\(T\)\}=\\begin\{cases\}z\_\{r,i\}^\{\(T\)\},&i\\in K\_\{r\}^\{\(T\)\},\\\\\[4\.0pt\] \-\\infty,&i\\notin K\_\{r\}^\{\(T\)\}\.\\end\{cases\}\(4\)
Following truncation, we optionally apply a temperatureτ\>0\\tau\>0to control the sharpness of the teacher distribution\. The resulting teacher probability is

pr,i\(T,k,τ\)=\{exp⁡\(zr,i\(T\)/τ\)∑j∈Kr\(T\)exp⁡\(zr,j\(T\)/τ\),i∈Kr\(T\),0,i∉Kr\(T\)\.p\_\{r,i\}^\{\(T,k,\\tau\)\}=\\begin\{cases\}\\displaystyle\\frac\{\\exp\(z\_\{r,i\}^\{\(T\)\}/\\tau\)\}\{\\sum\_\{j\\in K\_\{r\}^\{\(T\)\}\}\\exp\(z\_\{r,j\}^\{\(T\)\}/\\tau\)\},&i\\in K\_\{r\}^\{\(T\)\},\\\\\[12\.0pt\] 0,&i\\notin K\_\{r\}^\{\(T\)\}\.\\end\{cases\}\(5\)
By contrast, student logits are generated and consumed on\-the\-fly during training, so they impose negligible storage or communication overhead\. To retain as much optimization signal as possible, we compute the student distribution over the full vocabulary without top\-kktruncation\. Consistent with standard practice in off\-policy distillation, we also omit temperature scaling for the student\. The student probability is thus defined as

pr,i\(S\)=exp⁡\(zr,i\(S\)\)∑j∈𝒱exp⁡\(zr,j\(S\)\)\.p\_\{r,i\}^\{\(S\)\}=\\frac\{\\exp\(z\_\{r,i\}^\{\(S\)\}\)\}\{\\sum\_\{j\\in\\mathcal\{V\}\}\\exp\(z\_\{r,j\}^\{\(S\)\}\)\}\.\(6\)

### 3\.2Training Objectives

To align the student distribution with the teacher’s truncated soft targets, we employ the forward KL divergence\. At the sequence level, the knowledge distillation loss is defined as

LKD=1n∑r=1nDKL\(pr\(T,k,τ\)∥pr\(S\)\)=1n∑r=1n∑i∈Kr\(T\)pr,i\(T,k,τ\)logpr,i\(T,k,τ\)pr,i\(S\)\.L\_\{\\mathrm\{KD\}\}=\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{n\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{r\}^\{\(T,k,\\tau\)\}\\;\\middle\\\|\\;p\_\{r\}^\{\(S\)\}\\right\)=\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{n\}\\sum\_\{i\\in K\_\{r\}^\{\(T\)\}\}p\_\{r,i\}^\{\(T,k,\\tau\)\}\\log\\frac\{p\_\{r,i\}^\{\(T,k,\\tau\)\}\}\{p\_\{r,i\}^\{\(S\)\}\}\.\(7\)The summation is restricted toKr\(T\)K\_\{r\}^\{\(T\)\}becausepr,i\(T,k,τ\)=0p\_\{r,i\}^\{\(T,k,\\tau\)\}=0for alli∉Kr\(T\)i\\notin K\_\{r\}^\{\(T\)\}\. The student probabilitypr,i\(S\)p\_\{r,i\}^\{\(S\)\}in the denominator is always computed over the full vocabulary\.

Apart from the distillation objective, the student can also be trained with the standard language modeling \(LM\) loss on ground\-truth tokens:

LLM=−1n​∑r=1nlog⁡pr,xr\(S\)\.L\_\{\\mathrm\{LM\}\}=\-\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{n\}\\log p\_\{r,x\_\{r\}\}^\{\(S\)\}\.\(8\)

## 4Experimental Setup

This section describes the experimental configuration shared across all analyses, including the model architecture, training data, and evaluation protocol\.

### 4\.1Model Setup

#### 4\.1\.1Pruning–Distillation Scaling Ladder

Conducting controlled studies directly on a large\-scale model is computationally prohibitive\. At the same time, using an independently trained small model introduces potential confounds, as its behavior may differ from that of the large model for reasons unrelated to the training objective under investigation\.

To reduce experimental cost while preserving a principled connection to the large\-scale model, we construct the student model as the final rung of a multi\-stage pruning–distillation scaling ladder\. Specifically, we begin with DeepSeek V3 Base, denotedM0M\_\{0\}, which has 671B total parameters and 37B activated parameters\. We pruneM0M\_\{0\}to 128B total parameters and 19B activated parameters, then apply distillation\-based continued pre\-training on 1T tokens to recover and stabilize performance; the resulting model is denotedM1M\_\{1\}\. This procedure is applied recursively:M1M\_\{1\}is pruned to 59B total parameters and 10B activated parameters, followed by 200B tokens of distillation\-based continued pre\-training to obtainM2M\_\{2\};M2M\_\{2\}is further pruned to 24B total parameters and 4\.5B activated parameters, followed by an additional 200B tokens of distillation\-based continued pre\-training to obtainM3M\_\{3\}\. The full configuration is summarized in Table[1](https://arxiv.org/html/2607.16246#S4.T1)\.

This construction offers two practical advantages for controlled experimentation\. First, it substantially reduces training and evaluation cost, sinceM3M\_\{3\}is far less expensive to use thanM0M\_\{0\}\. Second, becauseM3M\_\{3\}is derived fromM0M\_\{0\}through a controlled sequence of pruning and distillation steps rather than trained independently from scratch, it retains important behavioral priors of the original large\-scale model\. This lineage makes observations onM3M\_\{3\}more representative of the large\-model regime than those obtained from a randomly initialized model of comparable size\. Taken together, the pruning–distillation pipeline serves as a cost\-efficient scaling ladder for studying training objectives under controlled conditions\.

Table 1:Summary of the pruning–distillation scaling ladder\. “Parent Model” denotes the model from which the current rung is pruned\. “Distillation Tokens” denotes the number of tokens used for distillation\-based continued pre\-training after pruning\.
#### 4\.1\.2Teacher and Student Models

Unless otherwise specified, all subsequent experiments useM3M\_\{3\}as the student model\. We use DeepSeek V3\.1 Base as the teacher model, as it outperforms DeepSeek V3 Base according to our internal evaluations, making it the stronger teacher in our teacher–student training setup\. The architecture configurations of both models are reported in Table[2](https://arxiv.org/html/2607.16246#S4.T2)\.

Table 2:Architecture configurations of the teacher and student models\.

### 4\.2Training Data

All training data used in both scaling\-ladder construction and subsequent experiments are collected internally\. The corpus combines open\-source and internally synthetic data, processed through a unified internal data pipeline\.

During scaling\-ladder construction, we adopt a fixed data mixture across all stages, consisting of 50% general\-domain data, 30% mathematics data, and 20% code data\. After the multi\-stage pruning and distillation procedure summarized in Table[1](https://arxiv.org/html/2607.16246#S4.T1), we evaluateM3M\_\{3\}following the protocol described in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3)and report the corresponding results in Table[4](https://arxiv.org/html/2607.16246#S5.T4)and Table[5](https://arxiv.org/html/2607.16246#S5.T5)\. The results indicate thatM3M\_\{3\}achieves satisfactory performance as a pre\-trained base model\. Therefore in subsequent experiments, we increase the proportions of mathematics and code data relative to general\-domain content\. Concretely, the adjusted mixture consists of 33% general\-domain data, 40% mathematics data, and 27% code data\.

### 4\.3Evaluation Protocol

#### 4\.3\.1Benchmark\-Level Evaluation

Following common practice, we evaluate models on a suite of 13 widely used benchmarks covering multiple capability dimensions assessed during pre\-training, including factual knowledge, commonsense reasoning, code generation, and mathematical reasoning\. Table[3](https://arxiv.org/html/2607.16246#S4.T3)summarizes the benchmark suite, specifying the number of few\-shot examples and the evaluation metric for each benchmark\. To ensure robust performance tracking, we execute three evaluation runs for each benchmark and report the mean score across all trials\.

Table 3:Summary of the benchmark suite used for evaluating pre\-trained base models\. The table reports the benchmark name, the number of few\-shot examples, and the evaluation metric\.
#### 4\.3\.2Pass@KKEvaluation on Selected Benchmarks

Single\-sample benchmark metrics may not fully characterize a base model’s capability, particularly for tasks where multiple generations can reveal whether the model places non\-negligible probability mass on correct\(Goyalet al\.,[2026](https://arxiv.org/html/2607.16246#bib.bib2)\)\. To complement benchmark\-level evaluation, we conduct zero\-shot Pass@KKevaluation on selected benchmarks—MATH\-500, AIME 2024, AIME 2025, and HumanEval—withK∈\{1,16,64,128\}K\\in\\\{1,16,64,128\\\}\. The generation hyperparameters are fixed at top\-k=50k=50, top\-p=0\.95p=0\.95, and temperatureT=1\.0T=1\.0\.

## 5Motivating Observation

### 5\.1Experimental Design

We begin with a controlled diagnostic experiment designed to isolate the effect of the training objective on model capabilities\. The goal is not to maximize final model quality, but rather to characterize the differential effects of hard next\-token supervision and soft teacher\-distribution supervision under otherwise identical training conditions\.

Following the experimental setup described in Section[4](https://arxiv.org/html/2607.16246#S4), both variants are initialized from the same student checkpoint and trained on the same 100B\-token corpus\. All training hyperparameters are held fixed: a global batch size of 5760, a sequence length of 4096, and a constant learning rate of5×10−55\\times 10^\{\-5\}\. The sole difference between the two runs is the training objective\. The*LM*variant is optimized with the standard language modeling lossLLML\_\{\\mathrm\{LM\}\}\. The*KD*variant is optimized with the knowledge distillation lossLKDL\_\{\\mathrm\{KD\}\}, using a teacher distribution constructed with top\-kktruncation \(k=256k=256\) and temperatureτ=1\\tau=1\.

### 5\.2Empirical Observations

We evaluate both variants following the protocol described in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3)\. Table[4](https://arxiv.org/html/2607.16246#S5.T4)reports results on the standard benchmark suite\. Table[5](https://arxiv.org/html/2607.16246#S5.T5)reports Pass@KKresults\.

Table 4:Benchmark\-level comparison between the LM and KD variants\.M3M\_\{3\}denotes the warm\-start model used to initialize both LM and KD training\.Table 5:Pass@KKcomparison on mathematical reasoning and code\-generation benchmarks\.M3M\_\{3\}denotes the warm\-start model used to initialize both LM and KD training\.The results show that neither objective is uniformly superior\. Instead, LM and KD exhibit consistent, benchmark\-dependent differences\. On benchmarks requiring complex reasoning, mathematical problem solving, or exam\-style knowledge assessment, LM consistently outperforms KD\. The most pronounced benchmark\-level gaps appear on MMLU\-Pro, where LM exceeds KD by 4\.24 points, and on MATH \(Minerva\), where the margin is 2\.74 points\. The Pass@KKevaluation further reinforces this pattern\. On MATH\-500, LM substantially outperforms KD across all sampling budgets, with gaps of 9\.87 points at Pass@1 and 29\.86 points at Pass@16\. Consistent trends are observed on AIME 2024 and AIME 2025; on AIME 2025 in particular, the KD variant achieves a zero pass rate at every reported sampling budget, while LM performance improves steadily with additional samples\. HumanEval exhibits the same directional pattern, though the gap narrows asKKincreases\. Taken together, these findings indicate that LM training yields a stronger capability profile than KD for high\-difficulty reasoning and code\-generation tasks\.

In contrast, KD outperforms LM on a distinct set of benchmarks more closely associated with commonsense plausibility, factual retrieval, local reading comprehension, and structured program synthesis\. Specifically, KD improves over LM by 1\.47 points on PIQA, 0\.84 points on TriviaQA, 1\.78 points on DROP, and 1\.80 points on MBPP\. Although these margins are smaller than the largest LM\-favored gaps, they are consistent in direction, indicating that KD is not uniformly inferior to LM\.

Overall, these results reveal a systematic, non\-uniform trade\-off between the two training objectives\. LM proves more favorable on high\-difficulty reasoning and mathematical benchmarks, while KD proves more favorable on knowledge\-intensive, commonsense, reading comprehension, and structured code\-generation tasks\.

### 5\.3Research Questions

The observations above motivate a more fine\-grained analysis of the optimization signals induced by the two training objectives\. Because both variants share the same initialization, training corpus, model architecture, and hyperparameter configuration, their systematically divergent benchmark behavior can be attributed solely to the choice of training objective\. This gives rise to the first research question \(RQ\):

##### RQ1\.

How does top\-kk\-truncated, temperature\-scaled off\-policy distillation reshape the token\-level optimization landscape relative to standard LM training? Specifically, how do the truncation parameterkkand temperatureτ\\taugovern the resulting token\-level supervision signal and its downstream effects on benchmark performance?

The capability\-dependent performance differences observed above indicate that LM and KD strengthen distinct capability dimensions\. Committing to a single global training objective therefore amounts to fixing a particular trade\-off among these dimensions—a choice that need not be optimal for all tasks of interest\. This motivates the second research question:

##### RQ2\.

Can training objectives be routed adaptively across the training distribution—from coarse data subsets down to individual tokens—to differentially strengthen distinct capability dimensions, rather than committing to a single global trade\-off?

## 6RQ1: How Truncation Size and Temperature Shape Token\-Level Supervision in Off\-Policy?

This section addresses RQ1\. We approach this question in four steps\. We begin with a gradient\-level comparison ofLKDL\_\{\\mathrm\{KD\}\}andLLML\_\{\\mathrm\{LM\}\}that establishes the structural difference at each token position\. We then introduce a suite of diagnostic metrics to quantify key dimensions of the resulting supervision signal\. Controlled sweeps overkkat fixedτ\\tauand overτ\\tauat fixedkkconnect these token\-level diagnostics to benchmark performance under each configuration\. A concluding synthesis integrates the findings into a direct answer to the research question\.

### 6\.1Gradient\-Level Comparison

We begin by comparing the optimization signals induced byLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}at the level of student logits\. Because both objectives are averaged over sequence positions, an analysis at a single positionrrsuffices; the sequence\-level gradients differ only by a scalar factor of1/n1/n\. Throughout this subsection we omit the position indexrrwhen there is no ambiguity\.

Following the definitions in Section[3](https://arxiv.org/html/2607.16246#S3),LLML\_\{\\mathrm\{LM\}\}uses the observed tokenxxas a one\-hot target, whereasLKDL\_\{\\mathrm\{KD\}\}uses the top\-kk\-truncated, temperature\-scaled teacher distributionp\(T,k,τ\)p^\{\(T,k,\\tau\)\}\. The partial derivatives with respect to the student logitzi\(S\)z\_\{i\}^\{\(S\)\}are

∂LLM∂zi\(S\)=pi\(S\)−𝟏​\[i=x\],\\frac\{\\partial L\_\{\\mathrm\{LM\}\}\}\{\\partial z\_\{i\}^\{\(S\)\}\}=p\_\{i\}^\{\(S\)\}\-\\mathbf\{1\}\[i=x\],\(9\)and

∂LKD∂zi\(S\)=pi\(S\)−pi\(T,k,τ\)\.\\frac\{\\partial L\_\{\\mathrm\{KD\}\}\}\{\\partial z\_\{i\}^\{\(S\)\}\}=p\_\{i\}^\{\(S\)\}\-p\_\{i\}^\{\(T,k,\\tau\)\}\.\(10\)Subtracting \([9](https://arxiv.org/html/2607.16246#S6.E9)\) from \([10](https://arxiv.org/html/2607.16246#S6.E10)\), the gradient gap between the two objectives is

Δ​gi≜∂LKD∂zi\(S\)−∂LLM∂zi\(S\)=𝟏​\[i=x\]−pi\(T,k,τ\)\.\\Delta g\_\{i\}\\triangleq\\frac\{\\partial L\_\{\\mathrm\{KD\}\}\}\{\\partial z\_\{i\}^\{\(S\)\}\}\-\\frac\{\\partial L\_\{\\mathrm\{LM\}\}\}\{\\partial z\_\{i\}^\{\(S\)\}\}=\\mathbf\{1\}\[i=x\]\-p\_\{i\}^\{\(T,k,\\tau\)\}\.\(11\)The student distributionp\(S\)p^\{\(S\)\}cancels in the difference, soΔ​gi\\Delta g\_\{i\}depends only on the discrepancy between the one\-hot data target and the truncated teacher target\.

The structure ofΔ​gi\\Delta g\_\{i\}is governed entirely by whether the observed token falls within the teacher top\-kksupportK\(T\)K^\{\(T\)\}; we examine each case in turn\.

##### Case I:x∈K\(T\)x\\in K^\{\(T\)\}\.

When the observed token is covered by the teacher support, the gradient gap is

Δ​gi=\{−pi\(T,k,τ\),i∈K\(T\),i≠x,1−px\(T,k,τ\),i=x,0,i∉K\(T\)\.\\Delta g\_\{i\}=\\begin\{cases\}\-p\_\{i\}^\{\(T,k,\\tau\)\},&i\\in K^\{\(T\)\},\\ i\\neq x,\\\\\[4\.0pt\] 1\-p\_\{x\}^\{\(T,k,\\tau\)\},&i=x,\\\\\[4\.0pt\] 0,&i\\notin K^\{\(T\)\}\.\\end\{cases\}\(12\)
For teacher\-supported tokensi∈K\(T\)∖\{x\}i\\in K^\{\(T\)\}\\setminus\\\{x\\\}, the gapΔ​gi=−pi\(T,k,τ\)<0\\Delta g\_\{i\}=\-p\_\{i\}^\{\(T,k,\\tau\)\}<0implies that the KD gradient is algebraically smaller than the LM gradient at these positions\. Under gradient descent, this translates into a less suppressive update on the corresponding logits—or even a logit increase whenpi\(T,k,τ\)\>pi\(S\)p\_\{i\}^\{\(T,k,\\tau\)\}\>p\_\{i\}^\{\(S\)\}\. These tokens are therefore treated as teacher\-supported alternatives rather than uniformly suppressed examples\.

For the observed token,Δ​gx=1−px\(T,k,τ\)\>0\\Delta g\_\{x\}=1\-p\_\{x\}^\{\(T,k,\\tau\)\}\>0wheneverpx\(T,k,τ\)<1p\_\{x\}^\{\(T,k,\\tau\)\}<1, indicating thatLKDL\_\{\\mathrm\{KD\}\}applies weaker upward pressure onzx\(S\)z\_\{x\}^\{\(S\)\}thanLLML\_\{\\mathrm\{LM\}\}\. In other words, even whenx∈K\(T\)x\\in K^\{\(T\)\},LKDL\_\{\\mathrm\{KD\}\}dilutes token\-level supervision of the observed token by redistributing part of the target mass to other teacher\-supported alternatives\.

Tokens outside the teacher support are assigned zero target mass by both objectives \(Δ​gi=0\\Delta g\_\{i\}=0\); the two losses thus impose identical logit\-level pressure on these tokens\.

##### Case II:x∉K\(T\)x\\notin K^\{\(T\)\}\.

When the observed token falls outside the teacher support,px\(T,k,τ\)=0p\_\{x\}^\{\(T,k,\\tau\)\}=0, and the gradient gap becomes

Δ​gi=\{−pi\(T,k,τ\),i∈K\(T\),1,i=x,0,i∉K\(T\),i≠x\.\\Delta g\_\{i\}=\\begin\{cases\}\-p\_\{i\}^\{\(T,k,\\tau\)\},&i\\in K^\{\(T\)\},\\\\\[4\.0pt\] 1,&i=x,\\\\\[4\.0pt\] 0,&i\\notin K^\{\(T\)\},\\ i\\neq x\.\\end\{cases\}\(13\)
All teacher\-supported tokens are now non\-observed alternatives\. As in Case I,LKDL\_\{\\mathrm\{KD\}\}preserves or increases probability mass on these tokens relative toLLML\_\{\\mathrm\{LM\}\}\.

The sharpest discrepancy concerns the observed token itself\. Sincex∉K\(T\)x\\notin K^\{\(T\)\},LKDL\_\{\\mathrm\{KD\}\}assigns zero target probability toxx, yielding

∂LKD∂zx\(S\)=px\(S\),\\frac\{\\partial L\_\{\\mathrm\{KD\}\}\}\{\\partial z\_\{x\}^\{\(S\)\}\}=p\_\{x\}^\{\(S\)\},\(14\)whereasLLML\_\{\\mathrm\{LM\}\}gives

∂LLM∂zx\(S\)=px\(S\)−1\.\\frac\{\\partial L\_\{\\mathrm\{LM\}\}\}\{\\partial z\_\{x\}^\{\(S\)\}\}=p\_\{x\}^\{\(S\)\}\-1\.\(15\)Under gradient descent,LLML\_\{\\mathrm\{LM\}\}increaseszx\(S\)z\_\{x\}^\{\(S\)\}, whereasLKDL\_\{\\mathrm\{KD\}\}decreases it\. This sign reversal represents the most severe conflict between the two objectives: the observed token is a positive target under LM but receives no target mass under KD\.

For tokens in neitherK\(T\)K^\{\(T\)\}nor\{x\}\\\{x\\\}, the gradient gap is again zero\.

This case analysis reveals the fundamental divergence betweenLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}\. The former concentrates all target mass on the observed token; the latter reallocates it to the teacher top\-kksupport\. Whenx∈K\(T\)x\\in K^\{\(T\)\},LKDL\_\{\\mathrm\{KD\}\}weakens the exclusive supervision of the observed token and treats teacher\-supported alternatives as valid targets\. Whenx∉K\(T\)x\\notin K^\{\(T\)\},LKDL\_\{\\mathrm\{KD\}\}removes the positive token\-level signal entirely and directly conflicts withLLML\_\{\\mathrm\{LM\}\}on the observed token\. These structural differences motivate the diagnostic metrics introduced in the following section\.

### 6\.2Diagnostic Metrics for Token\-Level Supervision

The gradient decomposition in Section[6\.1](https://arxiv.org/html/2607.16246#S6.SS1)shows that the influence ofLKDL\_\{\\mathrm\{KD\}\}on token\-level supervision can be characterized along three complementary dimensions: whether the observed token falls within the teacher’s top\-kksupport, how much target mass the teacher distribution assigns to the observed token, and how the probability mass is distributed among the teacher\-supported set\. We introduce a suite of diagnostic metrics to quantify each dimension\.

##### Observed\-token coverage\.

The first metric records whetherLKDL\_\{\\mathrm\{KD\}\}assigns any positive target mass to the observed token:

Coverage​@​k=𝟏​\[xr∈Kr\(T\)\]\.\\mathrm\{Coverage\}@k=\\mathbf\{1\}\\\!\\left\[x\_\{r\}\\in K\_\{r\}^\{\(T\)\}\\right\]\.\(16\)This metric directly distinguishes the two cases analyzed in Eqs\. \([12](https://arxiv.org/html/2607.16246#S6.E12)\)–\([13](https://arxiv.org/html/2607.16246#S6.E13)\)\. Because support membership depends solely on the top\-kktoken ranking, Coverage@kkis controlled bykkand is invariant to temperature scaling, provided that temperature does not alter the teacher’s token ranking\.

##### Observed\-token rank\.

To obtain a more granular characterization of the observed token’s position within the teacher distribution, we measure its rank under the teacher’s next\-token logits\. Letzr\(T\)​\(v\)z\_\{r\}^\{\(T\)\}\(v\)denote the teacher logit for tokenvvat positionrr\. The observed\-token rank is

OTRank=1\+∑v∈𝒱𝟏​\[zr\(T\)​\(v\)\>zr\(T\)​\(xr\)\],\\mathrm\{OTRank\}=1\+\\sum\_\{v\\in\\mathcal\{V\}\}\\mathbf\{1\}\\\!\\left\[z\_\{r\}^\{\(T\)\}\(v\)\>z\_\{r\}^\{\(T\)\}\(x\_\{r\}\)\\right\],\(17\)A lower value indicates that the observed token ranks closer to the top of the teacher distribution\. Like coverage, OTRank is invariant to temperature scaling as long as the teacher’s token ordering is preserved\.

##### Observed\-token mass\.

Coverage@kkindicates only whether the observed token is included in the teacher support; it does not quantify the strength of the resulting KD signal\. We therefore define the observed\-token mass as

OTMass​\(k,τ\)=pr,xr\(T,k,τ\)\.\\mathrm\{OTMass\}\(k,\\tau\)=p\_\{r,x\_\{r\}\}^\{\(T,k,\\tau\)\}\.\(18\)Becausepr,xr\(T,k,τ\)=0p\_\{r,x\_\{r\}\}^\{\(T,k,\\tau\)\}=0wheneverxr∉Kr\(T\)x\_\{r\}\\notin K\_\{r\}^\{\(T\)\}, this metric jointly reflects support coverage and within\-support mass allocation\. A lower value indicates weaker observed\-token reinforcement relative toLLML\_\{\\mathrm\{LM\}\}, which always assigns a target mass of one toxrx\_\{r\}\.

To disentangle the contribution of support coverage from within\-support mass allocation, we also report the conditional observed\-token mass:

CondOTMass​\(k,τ\)=pr,xr\(T,k,τ\),conditioned on​xr∈Kr\(T\)\.\\mathrm\{CondOTMass\}\(k,\\tau\)=p\_\{r,x\_\{r\}\}^\{\(T,k,\\tau\)\},\\qquad\\text\{conditioned on \}x\_\{r\}\\in K\_\{r\}^\{\(T\)\}\.\(19\)Unlike coverage and rank, both mass metrics are sensitive tokkandτ\\tau, since both parameters affect the normalized teacher probabilitiespr,i\(T,k,τ\)p\_\{r,i\}^\{\(T,k,\\tau\)\}\.

##### Teacher target distribution statistics\.

Let

pr,\(1\)\(T,k,τ\)≥pr,\(2\)\(T,k,τ\)≥⋯≥pr,\(k\)\(T,k,τ\)p\_\{r,\(1\)\}^\{\(T,k,\\tau\)\}\\geq p\_\{r,\(2\)\}^\{\(T,k,\\tau\)\}\\geq\\dots\\geq p\_\{r,\(k\)\}^\{\(T,k,\\tau\)\}denote the order statistics of the truncated teacher distribution at positionrr\. We summarize its concentration and softness with the following statistics\.

The top\-ranked teacher probability is

Top1Prob​\(k,τ\)=pr,\(1\)\(T,k,τ\)\.\\mathrm\{Top1Prob\}\(k,\\tau\)=p\_\{r,\(1\)\}^\{\(T,k,\\tau\)\}\.\(20\)A high value indicates that the KD target is concentrated on the teacher’s most probable token, approaching hard pseudo\-labeling; a low value indicates that substantial mass is distributed over lower\-ranked alternatives\.

The entropy of the truncated teacher distribution is

Entropy​\(k,τ\)=−∑i∈Kr\(T\)pr,i\(T,k,τ\)​log⁡pr,i\(T,k,τ\)\.\\mathrm\{Entropy\}\(k,\\tau\)=\-\\sum\_\{i\\in K\_\{r\}^\{\(T\)\}\}p\_\{r,i\}^\{\(T,k,\\tau\)\}\\log p\_\{r,i\}^\{\(T,k,\\tau\)\}\.\(21\)When comparing results across different values ofkk, we report the normalized entropy:

NormEntropy​\(k,τ\)=−∑i∈Kr\(T\)pr,i\(T,k,τ\)​log⁡pr,i\(T,k,τ\)log⁡k\.\\mathrm\{NormEntropy\}\(k,\\tau\)=\\frac\{\-\\sum\_\{i\\in K\_\{r\}^\{\(T\)\}\}p\_\{r,i\}^\{\(T,k,\\tau\)\}\\log p\_\{r,i\}^\{\(T,k,\\tau\)\}\}\{\\log k\}\.\(22\)These metrics characterize the extent to which the KD target spreads mass over teacher\-supported alternatives\. Under a fixed top\-kksupport, increasing temperature generally raises entropy by redistributing probability mass from the highest\-ranked tokens toward lower\-ranked alternatives\.

Finally, we measure the pre\-truncation top\-kkprobability mass, which quantifies how much of the original teacher distribution is retained by the top\-kksupport prior to renormalization\. Let

p~r,i\(T,τ\)=exp⁡\(zr,i\(T\)/τ\)∑j∈𝒱exp⁡\(zr,j\(T\)/τ\)\\widetilde\{p\}\_\{r,i\}^\{\(T,\\tau\)\}=\\frac\{\\exp\(z\_\{r,i\}^\{\(T\)\}/\\tau\)\}\{\\sum\_\{j\\in\\mathcal\{V\}\}\\exp\(z\_\{r,j\}^\{\(T\)\}/\\tau\)\}\(23\)denote the full\-vocabulary teacher distribution before truncation\. The pre\-truncation top\-kkmass is then

RawTopKMass​\(k,τ\)=∑i∈Kr\(T\)p~r,i\(T,τ\)\.\\mathrm\{RawTopKMass\}\(k,\\tau\)=\\sum\_\{i\\in K\_\{r\}^\{\(T\)\}\}\\widetilde\{p\}\_\{r,i\}^\{\(T,\\tau\)\}\.\(24\)This quantity is distinct from the total mass ofp\(T,k,τ\)p^\{\(T,k,\\tau\)\}overKr\(T\)K\_\{r\}^\{\(T\)\}, which is identically one by construction; it instead reflects the fraction of teacher probability that truncation discards from the full vocabulary\.

In the controlled experiments that follow, we use these metrics to characterize how varyingkkat fixedτ\\tauand varyingτ\\tauat fixedkkreshape the token\-level supervision signal, and to connect the resulting shifts to downstream evaluation behavior\.

### 6\.3Effect of Top\-kk

#### 6\.3\.1Experimental Setup

We examine how the top\-kktruncation parameter affects off\-policy distillation\. We vary the teacher support size overk∈\{1,4,16,64,256\}k\\in\\\{1,4,16,64,256\\\}while fixing the distillation temperature atτ=1\\tau=1\. All other training hyperparameters follow the configuration described in Section[5\.1](https://arxiv.org/html/2607.16246#S5.SS1)\.

We evaluate each setting using the protocol described in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3)\. Table[6](https://arxiv.org/html/2607.16246#S6.T6)reports results on the standard benchmark suite\. Table[7](https://arxiv.org/html/2607.16246#S6.T7)further reports Pass@KKresults, whereKKdenotes the number of sampled solutions at evaluation time and is distinct from the training\-side truncation parameterkk\.

Table 6:Benchmark results across teacher top\-kktruncation settings \(τ=1\\tau=1\)\.Table 7:Pass@KKresults across teacher top\-kktruncation settings on mathematical reasoning and code\-generation benchmarks\.
#### 6\.3\.2Analysis

Scores on many benchmarks vary only marginally across top\-kksettings\. To focus the analysis on benchmarks with non\-trivial sensitivity, we compute the relative score range

\|maxk⁡sk−mink⁡sk\|avgk​\(sk\),\\frac\{\|\\max\_\{k\}s\_\{k\}\-\\min\_\{k\}s\_\{k\}\|\}\{\\mathrm\{avg\}\_\{k\}\(s\_\{k\}\)\},wheresks\_\{k\}denotes the score under top\-kkdistillation, and use this ratio to identify benchmarks with meaningful top\-kksensitivity, setting aside those whose variation is likely dominated by measurement noise\.

##### Observed benchmark patterns\.

The standard benchmark suite exhibits heterogeneous responses to teacher support size\. HellaSwag and PIQA clearly benefit from expanding the support beyond Top\-1, and RACE shows a milder trend in the same direction\. In contrast, MMLU\-Pro and MATH \(Minerva\) favor intermediate support sizes—typically Top\-4 or Top\-16—and degrade as the support grows to Top\-256\. DROP exhibits yet another pattern: its best score occurs under Top\-1, suggesting that this benchmark may benefit more from a sharper target distribution than from additional teacher\-supported alternatives\. Overall, these results indicate that the effect of top\-kktruncation is non\-monotonic and task\-dependent; no single support size dominates across the full benchmark suite\.

Pass@KKevaluations show stronger top\-kksensitivity than the standard benchmark suite, particularly on mathematical reasoning tasks\. On MATH\-500 and AIME, Top\-4 and Top\-16 outperform both the narrow Top\-1 setting and the broader Top\-64 and Top\-256 settings\. HumanEval exhibits a different pattern: it favors Top\-1 at Pass@1, but the gap across top\-kksettings narrows substantially as the sampling budget increases\. This suggests that top\-kkhas a more pronounced influence on difficult mathematical reasoning tasks that require large sampling budgets, whereas code generation becomes less sensitive once a sufficient number of candidate solutions are drawn\.

##### Diagnostic statistics\.

To analyze these patterns, we randomly sample 1B tokens from the training corpus and report the position\-averaged diagnostic statistics defined in Section[6\.2](https://arxiv.org/html/2607.16246#S6.SS2)\. Figure[1](https://arxiv.org/html/2607.16246#S6.F1)summarizes the resulting token\-level measurements\.

![Refer to caption](https://arxiv.org/html/2607.16246v1/x1.png)Figure 1:Diagnostic statistics for top\-kktruncated teacher distributions atτ=1\\tau=1\.The diagnostics reveal a systematic trade\-off induced by increasing the teacher support size\. Observed\-token coverage increases monotonically withkk: Coverage@kkrises from0\.71570\.7157at Top\-1 to0\.86400\.8640,0\.93490\.9349,0\.97040\.9704, and0\.98760\.9876at Top\-4, Top\-16, Top\-64, and Top\-256, respectively\. Larger support sizes therefore reduce the frequency with which the observed data token falls outside the teacher support\.

This coverage gain is accompanied by weaker observed\-token reinforcement\. OTMass decreases from0\.71570\.7157at Top\-1 to0\.63590\.6359at Top\-256, while CondOTMass falls from1\.00001\.0000to0\.64390\.6439\. Top1Prob declines from1\.00001\.0000to0\.72060\.7206, and the entropy of the truncated distribution rises from0\.52990\.5299at Top\-4 to1\.11661\.1166at Top\-256\. Because raw entropy partly depends on support size, we interpret it jointly with Top1Prob, OTMass, CondOTMass, and normalized entropy\. Taken together, these statistics indicate that larger support sizes produce a less concentrated KD target: more observed tokens are covered, but each observed token receives less probability mass, and the remaining mass is spread more diffusely over teacher\-supported alternatives\.

Observed\-token rank statistics clarify why the marginal coverage benefit of increasingkkdiminishes rapidly\. The median observed\-token rank is close to11, the 75th percentile falls below22, and the 90th percentile is only7\.777\.77, while the 99th percentile reaches256\.70256\.70\. Consequently, the vast majority of observed tokens already lie within a small or intermediate support such as Top\-4 or Top\-16\. Increasingkkbeyond this range primarily extends the support to long\-tail positions and introduces additional low\-probability teacher alternatives without meaningfully improving coverage for most tokens\.

##### Core mechanism\.

The diagnostics suggest that the top\-kkparameter governs a fundamental coverage–sharpness trade\-off\. Increasingkkraises observed\-token coverage, reducing support mismatch, but simultaneously lowers observed\-token mass and raises entropy, producing a progressively softer target distribution over teacher\-supported alternatives\. The coverage gain can be beneficial when Top\-1 is too restrictive and frequently excludes observed tokens from the teacher support\. The resulting target softening, however, can be detrimental when the target capability requires precise token\-level reinforcement, as the KD objective becomes increasingly diffuse over teacher\-preferred alternatives\.

Because the diagnostic statistics are computed on a random sample from the training corpus rather than on benchmark\-specific distributions, they characterize the global optimization signal induced by top\-kktruncation\. The benchmark\-level interpretations that follow should therefore be understood as hypotheses consistent with the diagnostics rather than as direct causal attributions\.

##### Analysis of standard benchmarks\.

The aggregate score on the standard benchmark suite is relatively insensitive to top\-kk, which is consistent with the diagnostic trade\-off\. Expanding the support beyond Top\-1 improves observed\-token coverage but also reduces CondOTMass and softens the target distribution\. Across a diverse benchmark suite, these two effects tend to partially offset each other: some tasks may benefit from reduced support mismatch, while others may be adversely affected by weaker observed\-token reinforcement\. As a result, no single top\-kkvalue dominates uniformly across the standard benchmarks\.

HellaSwag and PIQA—with RACE as a milder case—are representative benchmarks that benefit from larger support sizes\. Under Top\-1, Coverage@kkis0\.71570\.7157, meaning a substantial fraction of observed tokens receive zero KD target mass when they fall outside the teacher’s top prediction\. Expanding the support to Top\-4 or Top\-16 raises coverage sharply to0\.86400\.8640and0\.93490\.9349, respectively\. For commonsense plausibility and reading\-style multiple\-choice tasks, the additional teacher\-supported alternatives may preserve semantically relevant uncertainty rather than acting as noise\. In these cases, the coverage gain and richer soft supervision appear to outweigh the dilution of observed\-token mass\.

MMLU\-Pro and MATH \(Minerva\) exhibit a different pattern\. Both benefit from moving beyond Top\-1 but degrade as the support becomes too broad, consistent with an intermediate optimum in the coverage–sharpness trade\-off\. Top\-4 and Top\-16 substantially reduce support mismatch while maintaining a relatively concentrated target distribution\. Expanding to Top\-64 or Top\-256 yields only modest additional coverage while further reducing CondOTMass and increasing entropy\. For knowledge\-intensive and mathematical reasoning tasks, this additional softening may weaken the precise token\-level optimization signal required for stable reasoning chains and exact answer formation\.

DROP provides a useful counterexample\. Its preference for Top\-1 suggests that certain reading\-comprehension and numerical QA settings may benefit more from a sharp pseudo\-label\-like target than from a broader teacher distribution\. A plausible explanation is that DROP\-style tasks—which often involve localized evidence selection, span matching, or numerical manipulation—are sensitive to the dilution of exact\-token reinforcement by additional soft supervision\. This observation reinforces the conclusion that larger teacher support is not universally beneficial\.

##### Analysis of Pass@KKresults\.

The Pass@KKresults, particularly on mathematical reasoning benchmarks, are more sensitive to top\-kkthan the standard benchmark suite\. A plausible explanation is that long\-chain reasoning amplifies both token\-level support mismatch and the effect of target softening\. Under Top\-1, many intermediate reasoning tokens may fall outside the teacher support, and local supervision gaps can accumulate over long solution trajectories\. Expanding the support to Top\-4 or Top\-16 substantially improves observed\-token coverage, thereby reducing this mismatch\.

However, excessively large supports are also suboptimal\. From Top\-16 to Top\-256, Coverage@kkincreases only from0\.93490\.9349to0\.98760\.9876, while CondOTMass decreases from0\.68950\.6895to0\.64390\.6439and entropy continues to rise\. The additional support thus primarily introduces low\-probability alternatives rather than meaningfully improving coverage for most observed tokens\. For MATH\-500 and AIME, where successful generation requires coherent and precise reasoning trajectories, a progressively softer target may reduce the probability of producing valid solution paths\. This interpretation is consistent with the finding that Top\-4 and Top\-16 outperform both Top\-1 and Top\-64/Top\-256 on these mathematical Pass@KKevaluations\.

HumanEval behaves differently from the mathematical benchmarks\. Its Pass@1 score favors Top\-1, suggesting that sharp supervision may aid single\-sample code generation, where syntactic correctness, API usage, and formatting precision are critical\. As the sampling budget increases, the performance gap across top\-kksettings narrows\. Under larger sampling budgets, success depends less on the single highest\-probability generation and more on whether the model can produce at least one correct candidate, which may partially offset the advantage of the sharper Top\-1 target\.

##### Why Top\-64 and Top\-256 perform similarly\.

The small performance difference between Top\-64 and Top\-256 follows directly from the diagnostics\. By Top\-64, Coverage@kkhas already reached0\.97040\.9704and RawTopKMass is0\.97170\.9717\. Increasing the support to Top\-256 raises these figures only to0\.98760\.9876and0\.98850\.9885, respectively, with correspondingly modest changes in OTMass, CondOTMass, and Top1Prob\. Top\-256 therefore adds predominantly low\-probability tail tokens, yielding only a marginal shift in the effective optimization signal for most positions\. These considerations help explain why Top\-64 and Top\-256 often yield similar performance on the standard benchmark suite, while leaving room for larger divergences in settings that are particularly sensitive to target softening, such as low\-budget mathematical Pass@KKevaluations\.

### 6\.4Effect of Temperature

#### 6\.4\.1Experimental Setup

We next study how distillation temperature affects off\-policy distillation\. Based on the findings in Section[6\.3](https://arxiv.org/html/2607.16246#S6.SS3), we fix the teacher support size atk=256k=256and vary the temperature overτ∈\{0\.25,0\.5,1\.0\}\\tau\\in\\\{0\.25,0\.5,1\.0\\\}\. This broad\-support regime ensures that the teacher support covers most observed tokens and retains most of the raw teacher probability mass, so the temperature sweep isolates within\-support probability allocation rather than support coverage effects\.

We also include the limiting caseτ→0\\tau\\to 0as a hard pseudo\-labeling baseline\. In this limit, the top\-256256teacher distribution collapses to a one\-hot target on the teacher’s top\-1 token and is therefore equivalent to top\-11distillation in terms of the resulting target distribution, absent ties in the teacher logits\. All other training hyperparameters follow the configuration described in Section[5\.1](https://arxiv.org/html/2607.16246#S5.SS1)\.

We evaluate each temperature setting using the protocol described in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3)\. Table[8](https://arxiv.org/html/2607.16246#S6.T8)reports results on the standard benchmark suite\. Table[9](https://arxiv.org/html/2607.16246#S6.T9)further reports Pass@KKresults\.

Table 8:Benchmark results under different distillation temperatures\. The limiting caseτ→0\\tau\\to 0collapses the teacher target to a one\-hot distribution on the teacher’s top\-1 token, equivalent to top\-11distillation in terms of the resulting target distribution\.Table 9:Pass@KKresults under different distillation temperatures on mathematical reasoning and code\-generation benchmarks\.
#### 6\.4\.2Analysis

##### Observed benchmark patterns\.

The standard benchmark suite exhibits heterogeneous responses to temperature\. HellaSwag shows the clearest preference for higher temperature, with scores increasing from74\.2874\.28underτ→0\\tau\\to 0to79\.1679\.16underτ=1\.0\\tau=1\.0\. PIQA follows the same direction, rising from78\.6778\.67to80\.6380\.63\. DROP exhibits the opposite pattern, achieving its best score underτ→0\\tau\\to 0and its lowest underτ=1\.0\\tau=1\.0\. MMLU\-Pro and MATH \(Minerva\) favor low\-to\-moderate temperatures: MMLU\-Pro peaks atτ=0\.25\\tau=0\.25, while MATH \(Minerva\) achieves its highest scores atτ=0\.25\\tau=0\.25andτ=0\.5\\tau=0\.5\. MBPP exhibits a less interpretable pattern—bothτ→0\\tau\\to 0andτ=1\.0\\tau=1\.0tie for the best score while intermediate temperatures underperform—and we therefore set it aside in the analysis below\.

Pass@KKevaluations show stronger temperature sensitivity than the standard benchmark suite, particularly on mathematical reasoning tasks\. For MATH\-500, Pass@16, Pass@64, and Pass@128 all favor the intermediate settingτ=0\.5\\tau=0\.5\. AIME exhibits a similar pattern; on AIME 2025, Pass@128 peaks atτ=0\.5\\tau=0\.5while bothτ→0\\tau\\to 0andτ=1\.0\\tau=1\.0perform substantially worse\. HumanEval follows a different pattern: Pass@1 strongly favorsτ→0\\tau\\to 0, decreasing from37\.2237\.22to22\.7922\.79asτ\\tauincreases to1\.01\.0, while Pass@64 and Pass@128 are nearly insensitive to temperature\.

##### Diagnostic statistics\.

Figure[2](https://arxiv.org/html/2607.16246#S6.F2)reports the diagnostic statistics defined in Section[6\.2](https://arxiv.org/html/2607.16246#S6.SS2), computed on a random 1B\-token sample from the training corpus\.

![Refer to caption](https://arxiv.org/html/2607.16246v1/x2.png)Figure 2:Diagnostic statistics for top\-256256truncated teacher distributions under different distillation temperatures\.Because temperature scaling preserves the teacher’s token ranking, the top\-256256support, Coverage@kk, and observed\-token rank are all invariant across this sweep\. Temperature therefore affects only within\-support probability allocation\. Asτ\\tauincreases from0to1\.01\.0, OTMass decreases from0\.71570\.7157to0\.63590\.6359, and CondOTMass falls more sharply from1\.00001\.0000to0\.64390\.6439\. Top1Prob similarly declines from1\.00001\.0000to0\.72060\.7206, while the entropy of the truncated teacher distribution rises from0\.19090\.1909atτ=0\.25\\tau=0\.25to1\.11661\.1166atτ=1\.0\\tau=1\.0\. These trends confirm that higher temperature redistributes probability mass from the top\-ranked token toward lower\-ranked teacher\-supported alternatives\.

##### Core mechanism\.

Under fixed top\-kktruncation, temperature controls within\-support probability allocation\. At very lowτ\\tau, the teacher target approaches a one\-hot pseudo\-label on the teacher’s top\-1 token, providing strong direct reinforcement of the highest\-ranked prediction while suppressing teacher\-supported alternatives\. Asτ\\tauincreases, the target distribution softens: mass flows from the top\-ranked token toward lower\-ranked alternatives within the support, reducing the average probability assigned to the observed token\.

This induces a trade\-off between two complementary training signals: direct observed\-token reinforcement and teacher\-supported alternative supervision\. Lower temperature emphasizes sharp token\-level supervision, resembling pseudo\-label training\. Higher temperature exposes the student to a broader set of teacher\-preferred alternatives but simultaneously weakens direct reinforcement of the observed token\. As in Section[6\.3](https://arxiv.org/html/2607.16246#S6.SS3), the diagnostic statistics characterize the global optimization signal; benchmark\-level interpretations below should be understood as hypotheses consistent with the diagnostics rather than as direct causal attributions\.

##### Analysis of standard benchmarks\.

HellaSwag’s improvement under higher temperature is consistent with the interpretation that commonsense plausibility tasks benefit from richer soft supervision\. As a semantic discrimination benchmark, HellaSwag may contain positions where non\-top\-1 teacher\-supported tokens encode locally plausible continuations or distributional preferences rather than noise\. As temperature increases and these alternatives receive greater probability mass, the resulting training signal provides richer guidance than a near\-one\-hot target\. PIQA exhibits a similar, albeit weaker, trend\.

DROP exhibits the opposite behavior, performing best under the sharpest target and degrading atτ=1\.0\\tau=1\.0\. Reading\-comprehension and numerical QA tasks often require precise evidence tokens, span\-level grounding, or numerical specificity, concentrating relevant supervision on a single correct token or phrase\. High\-temperature teacher targets can dilute this signal by distributing more mass to within\-support alternatives, consistent with the observed decreases in OTMass and CondOTMass\.

MMLU\-Pro and MATH \(Minerva\) favor intermediate temperatures\. We hypothesize that these benchmarks benefit from a balance between target sharpness and limited alternative supervision: very low temperature over\-concentrates the target on the teacher’s top\-1 prediction and suppresses useful alternative reasoning tokens, while high temperature weakens precise reasoning\- and answer\-token supervision\. Intermediate temperatures \(τ=0\.25\\tau=0\.25–0\.50\.5\) preserve a relatively concentrated target while allowing some non\-top\-1 teacher preferences to contribute to training\.

##### Analysis of Pass@KKresults\.

The Pass@KKresults reveal stronger temperature sensitivity for mathematical generation than the standard benchmark suite\. For MATH\-500 and AIME, intermediate temperatures—particularlyτ=0\.5\\tau=0\.5—are consistently favorable under larger sampling budgets\. A plausible interpretation is that successful mathematical generation benefits from learning a distribution over multiple reasoning trajectories: a near\-one\-hot target underτ→0\\tau\\to 0may provide strong local supervision but underrepresent alternative reasoning paths that become useful under repeated sampling, whileτ=1\.0\\tau=1\.0may be too diffuse to maintain precise reasoning\-step supervision\. The intermediate settingτ=0\.5\\tau=0\.5appears to balance these competing requirements\.

HumanEval follows a different pattern\. Its Pass@1 score clearly favorsτ→0\\tau\\to 0, consistent with code generation’s sensitivity to local syntax, API conventions, indentation, and variable naming, where a high\-temperature teacher target may assign non\-negligible mass to locally plausible but functionally incorrect alternatives\. This advantage diminishes substantially at higher sampling budgets: Pass@64 and Pass@128 on HumanEval are nearly insensitive to temperature\. This distinguishes HumanEval from the mathematical benchmarks, for which intermediate temperature remains advantageous even under large sampling budgets\.

### 6\.5Summary of Findings

The preceding analysis establishes that top\-kk\-truncated, temperature\-scaled off\-policy distillation reshapes the token\-level optimization landscape by replacing the one\-hot supervision of standard LM training with a distributional teacher target defined over a restricted support\. At the gradient level, the discrepancy betweenLKDL\_\{\\mathrm\{KD\}\}andLLML\_\{\\mathrm\{LM\}\}depends solely on the difference between the observed data token and the truncated teacher distribution\. When the observed token lies within the top\-kkteacher support,LKDL\_\{\\mathrm\{KD\}\}retains positive target mass for that token, but the resulting upward gradient pressure is weaker than underLLML\_\{\\mathrm\{LM\}\}because part of the probability mass is reallocated to teacher\-supported alternatives\. These alternatives are no longer treated as uniformly negative examples; they instead receive positive or less negative supervision\. When the observed token falls outside the teacher support, the conflict is more severe:LLML\_\{\\mathrm\{LM\}\}increases the corresponding logit, whereasLKDL\_\{\\mathrm\{KD\}\}assigns it zero target mass and suppresses it under gradient descent\. Relative to standard LM training,LKDL\_\{\\mathrm\{KD\}\}thus trades exclusive observed\-token reinforcement for teacher\-supported alternative supervision, with the nature and severity of this trade\-off determined by the support structure and its probability allocation\.

The truncation sizekkand temperatureτ\\tauregulate this trade\-off through different mechanisms\. The top\-kkparameter primarily governs a coverage–sharpness trade\-off\. Increasingkkraises the probability that the observed token falls within the teacher support, reducing support mismatch and thereby avoiding positions whereLKDL\_\{\\mathrm\{KD\}\}assigns zero target mass to the data token\. Coverage@kkand the observed\-token rank distribution directly quantify this effect\. However, expanding the support simultaneously dilutes the teacher target: OTMass and CondOTMass decrease, Top1Prob declines, and entropy rises, indicating that probability mass is distributed over a progressively broader set of teacher\-supported alternatives\. Temperature, by contrast, does not alter support membership or observed\-token rank at fixedkk, since rescaling logits preserves the teacher’s token ordering\. Instead,τ\\taucontrols within\-support probability allocation: lower temperature concentrates mass toward the top\-ranked token, making the teacher target approach a hard pseudo\-label, while higher temperature redistributes mass toward lower\-ranked alternatives within the same support\. Raisingτ\\tautherefore weakens direct observed\-token reinforcement at positions where the observed token is among the teacher’s highest\-ranked predictions, while strengthening the supervision carried by non\-top\-1 teacher\-supported alternatives\.

These two hyperparameters modulate two complementary factors:*direct observed\-token reinforcement*and*teacher\-supported alternative supervision*\. Direct observed\-token reinforcement is strongest when the teacher target is concentrated and the observed token receives high probability mass—as under Top\-1 distillation, or under very low temperature when the observed token coincides with the teacher’s top\-ranked prediction\. Teacher\-supported alternative supervision is strongest when the support is broad or the temperature is high, exposing the student to a wider set of teacher\-preferred continuations\. Neither factor is universally beneficial\. Strong observed\-token reinforcement preserves precise data\-token learning but may discard useful teacher uncertainty and suppress valid alternative continuations\. Conversely, rich alternative supervision can expose the student to semantically or procedurally relevant teacher preferences, but may weaken exact\-token supervision in tasks that require precise grounding, stable reasoning chains, or exact answer formation\.

The benchmark results are consistent with this account and reveal considerable task heterogeneity\. Commonsense plausibility tasks such as HellaSwag and PIQA benefit from broader or softer teacher supervision: scores improve when expanding beyond Top\-1 and tend to increase with temperature\. This pattern suggests that teacher\-supported alternatives encode meaningful semantic uncertainty for these tasks rather than acting as noise\. In contrast, DROP favors sharper targets, performing best under Top\-1 or very low temperature, consistent with its requirement for localized evidence selection, span grounding, and numerical specificity—settings in which spreading probability mass over alternatives dilutes the exact\-token signal\. MMLU\-Pro and MATH \(Minerva\) favor intermediate configurations—Top\-4 or Top\-16 at low\-to\-moderate temperatures—indicating that these tasks benefit from reduced support mismatch while still maintaining a sufficiently concentrated target\.

The Pass@KKresults reinforce and extend this picture\. Mathematical generation benchmarks such as MATH\-500 and AIME exhibit strong sensitivity to the coverage–sharpness balance: intermediate support sizes \(Top\-4 or Top\-16\) and intermediate temperatures \(e\.g\.,τ=0\.5\\tau=0\.5\) consistently outperform both highly concentrated and highly diffuse targets\. This suggests that mathematical reasoning benefits from limited alternative supervision—plausibly because multiple valid reasoning trajectories are useful under large sampling budgets—while still requiring sufficient target concentration to sustain coherent derivations and precise answer tokens\. HumanEval shows a different pattern: Pass@1 clearly favors sharper supervision, consistent with the sensitivity of code generation to local syntactic correctness, API usage, and formatting conventions, whereas sensitivity tokkandτ\\taudiminishes substantially at larger sampling budgets\. Taken together, these findings indicate that top\-kkand temperature do not act as simple quality multipliers for knowledge distillation; rather, they position the token\-level optimization signal along a spectrum between direct observed\-token reinforcement and teacher\-supported alternative supervision, and different tasks favor different positions along this spectrum for optimal knowledge transfer\.

## 7RQ2: Can Training Objectives Be Routed Adaptively Across the Training Distribution?

We approach RQ2 through a sequence of progressively finer\-grained analyses\. The complementary performance profiles ofLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}across benchmarks motivate a natural first step: assigning different training objectives to different data subsets according to coarse domain labels\. To understand why domain\-level routing may serve as an effective proxy for objective routing, we analyze the diagnostic metrics Coverage@kk, CondOTMass, and Entropy stratified by domain, linking the coarse domain partition to the underlying token\-level supervision statistics\. Building on these findings, we evaluate finer\-grained token\-level routing policies that use OTMass and Entropy directly as per\-token routing criteria\. The section concludes with a direct answer to RQ2 that integrates the subset\-level and token\-level results, with particular attention to how routing\-signal quality determines the practical effectiveness of adaptive objective routing\.

### 7\.1Domain\-Based Subset\-Level Objective Routing

Table 10:Benchmark\-level comparison under different training\-objective routing strategies\.Table 11:Pass@KKcomparison under different training\-objective routing strategies\.The findings from RQ1 establish thatLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}exhibit complementary advantage regions across benchmarks and task types\.LLML\_\{\\mathrm\{LM\}\}is more favorable for several mathematics\- and code\-related evaluations\.LKDL\_\{\\mathrm\{KD\}\}, by contrast, performs better on several general\-domain benchmarks—including knowledge\-oriented, commonsense, and reading\-comprehension evaluations\. These complementary patterns motivate a natural question: rather than applying a single objective uniformly across all training data, can objective routing be guided by coarse data\-domain labels?

We evaluate a domain\-based subset\-level routing policy\. Following the data mixture described in Section[4\.2](https://arxiv.org/html/2607.16246#S4.SS2), we retain the same overall training composition as the single\-objective baselines but assign different objectives to different domains\. Specifically, we applyLLML\_\{\\mathrm\{LM\}\}to mathematics and code data, andLKDL\_\{\\mathrm\{KD\}\}with top\-256256truncation and temperatureτ=1\\tau=1to the remaining general\-domain data\. Data packing is performed separately within each domain, ensuring that no packed training sequence spans multiple domains\. This design makes the objective assignment unambiguous at the sequence level and avoids mixingLLML\_\{\\mathrm\{LM\}\}\- andLKDL\_\{\\mathrm\{KD\}\}\-supervised tokens within the same packed sequence\. All other training hyperparameters follow the configuration in Section[5\.1](https://arxiv.org/html/2607.16246#S5.SS1)\. We evaluate the resulting model using the protocol described in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3), and refer to this configuration asDomainthroughout the following discussion\.

Tables[10](https://arxiv.org/html/2607.16246#S7.T10)and[11](https://arxiv.org/html/2607.16246#S7.T11)report the standard benchmark and Pass@KKresults, respectively\. Across both evaluation settings, domain\-level routing consistently improves aggregate performance relative to the single\-objective baselines and often matches or exceeds the stronger of the two on individual benchmarks\. On the standard benchmark suite,Domainachieves the highest average score among the three variants\. On the Pass@KKevaluation, where the gap betweenLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}is more pronounced,Domainsubstantially outperformsLKDL\_\{\\mathrm\{KD\}\}and is broadly competitive withLLML\_\{\\mathrm\{LM\}\}\. These results suggest that coarse domain\-level routing can selectively retain the advantages of each objective rather than producing outcomes intermediate between the two single\-objective baselines\.

##### Recovering theLLML\_\{\\mathrm\{LM\}\}advantage whereLKDL\_\{\\mathrm\{KD\}\}underperforms\.

The first clear pattern is thatDomainlargely recovers theLLML\_\{\\mathrm\{LM\}\}advantage on tasks whereLKDL\_\{\\mathrm\{KD\}\}is weak, particularly on high\-difficulty reasoning, mathematics, and sampling\-intensive evaluations\. On MMLU\-Pro,LKDL\_\{\\mathrm\{KD\}\}falls noticeably behindLLML\_\{\\mathrm\{LM\}\}, whileDomainremains close toLLML\_\{\\mathrm\{LM\}\}\. A similar pattern appears on MATH \(Minerva\), whereDomainslightly exceedsLLML\_\{\\mathrm\{LM\}\}and clearly outperformsLKDL\_\{\\mathrm\{KD\}\}\. The trend is more pronounced in the Pass@KKresults\. On MATH\-500,LKDL\_\{\\mathrm\{KD\}\}is consistently weaker thanLLML\_\{\\mathrm\{LM\}\}across all sampling budgets, whereasDomainremains close toLLML\_\{\\mathrm\{LM\}\}, especially at largerKK\. On AIME 2024 and AIME 2025,LKDL\_\{\\mathrm\{KD\}\}performs poorly, whileDomainrecovers strong mathematical Pass@KKperformance and exceedsLLML\_\{\\mathrm\{LM\}\}in several settings\. These results are consistent with the intended effect of the routing policy: applyingLLML\_\{\\mathrm\{LM\}\}to mathematics and code subsets mitigates the degradation observed under distillation\-only training on tasks requiring precise token\-level supervision\.

##### Retaining theLKDL\_\{\\mathrm\{KD\}\}advantage whereLKDL\_\{\\mathrm\{KD\}\}is favorable\.

The second pattern is thatDomainretains the benefit ofLKDL\_\{\\mathrm\{KD\}\}on several general\-domain benchmarks\. DROP is the clearest case:LKDL\_\{\\mathrm\{KD\}\}outperformsLLML\_\{\\mathrm\{LM\}\}, andDomainmatches theLKDL\_\{\\mathrm\{KD\}\}score, indicating that the routing strategy preserves the distillation advantage on this benchmark\. PIQA and TriviaQA show a weaker but consistent version of the same trend, whereDomainremains close toLKDL\_\{\\mathrm\{KD\}\}rather than reverting toLLML\_\{\\mathrm\{LM\}\}\-only behavior\.

##### Cases whereDomainexceeds both single\-objective baselines\.

In several cases,Domainexceeds bothLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}, indicating that the routed model is not always bounded by the two single\-objective baselines\. On MBPP,Domainachieves a clear improvement over both variants, with a smaller gain also observed on BBH\. In the Pass@KKevaluation,Domainoutperforms both baselines on AIME 2024 and AIME 2025 across several sampling budgets\. These results are compatible with the view thatLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}provide complementary supervision signals that, when applied selectively, can produce outcomes stronger than either objective alone\.

##### A remaining exception\.

RACE is the main exception to the overall trend:Domainunderperforms bothLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}on this benchmark, despite the generally favorable behavior of domain\-level routing across the rest of the evaluation suite\. We currently have no satisfactory explanation for this isolated reversal and leave its investigation to future work\.

### 7\.2Diagnostic Metrics Across Domains

![Refer to caption](https://arxiv.org/html/2607.16246v1/x3.png)Figure 3:Domain\-stratified diagnostic metrics under top\-256256distillation with temperatureτ=1\\tau=1\.The domain\-routing results demonstrate that coarse domain labels can serve as a useful prior for objective routing\. However, a domain label is inherently a qualitative, high\-level annotation: it does not directly characterize the relationship between the observed training token and the teacher target distribution\. To examine whether domain labels correlate with more fundamental token\-level signals, we follow the analysis protocol of Section[6\.3](https://arxiv.org/html/2607.16246#S6.SS3)\. Specifically, we sample 1B tokens from the training corpus and compute the diagnostic metrics defined in Section[6\.2](https://arxiv.org/html/2607.16246#S6.SS2)separately for each domain, fixing the teacher target to top\-256256truncation with temperatureτ=1\\tau=1\.

We focus on three metrics: Coverage@kk, CondOTMass, and Entropy\. Together, these capture complementary aspects of the teacher target distribution\. Coverage@kkmeasures whether the observed token falls within the teacher top\-kksupport\. CondOTMass measures the probability mass assigned to the observed token by the truncated teacher distribution, conditioned on the observed token being covered\. Entropy measures how teacher probability mass is distributed among supported alternatives\. The domain\-stratified statistics are shown in Figure[3](https://arxiv.org/html/2607.16246#S7.F3)\.

Coverage@kkis high across all three domains and is therefore unlikely to be the primary source of domain\-level differences under the top\-256256setting\. General\-domain data has Coverage@kkof0\.97970\.9797, while mathematics and code reach0\.98880\.9888and0\.99450\.9945, respectively\. This indicates that, when the teacher support is sufficiently large, the main distinction across domains lies not in whether the observed token is included in the support\.

CondOTMass exhibits a more pronounced domain\-level separation\. The value increases from0\.49580\.4958for general\-domain data to0\.65560\.6556for mathematics and0\.82580\.8258for code\. This indicates that, conditioned on coverage, the observed token receives substantially higher teacher probability mass in mathematics and code than in the general domain\. CondOTMass is directly related to the within\-support component of theLLML\_\{\\mathrm\{LM\}\}–LKDL\_\{\\mathrm\{KD\}\}supervision gap described in Eq\. \([11](https://arxiv.org/html/2607.16246#S6.E11)\):LLML\_\{\\mathrm\{LM\}\}assigns all target mass to the observed token, whereasLKDL\_\{\\mathrm\{KD\}\}assigns onlypx\(T,k,τ\)p\_\{x\}^\{\(T,k,\\tau\)\}and distributes the remainder over teacher\-supported alternatives\. Since Coverage@kkis consistently high across all domains under top\-256256, domain\-level variation in CondOTMass provides a useful proxy for the strength of the observed\-token supervision signal\. Notably, CondOTMass depends jointly on the observed data token and the teacher distribution, making it a measure of teacher–data alignment rather than a property of the teacher alone\.

Entropy shows the opposite trend\. General\-domain data has the highest entropy \(1\.65531\.6553\), while mathematics and code have lower values of1\.05791\.0579and0\.49320\.4932, respectively, indicating that the teacher target distribution is more diffuse on general\-domain data and more concentrated on mathematics and code\. Unlike CondOTMass, however, Entropy does not directly isolate the observed\-token component of theLLML\_\{\\mathrm\{LM\}\}–LKDL\_\{\\mathrm\{KD\}\}gradient gap in Eq\. \([11](https://arxiv.org/html/2607.16246#S6.E11)\): it characterizes how teacher probability mass is spread over supported alternatives without reference to whether the observed token is covered or assigned high mass\. Entropy therefore reflects teacher\-side distributional concentration in isolation, rather than alignment between the teacher target and the observed token\.

Taken together, these observations suggest that the informativeness of domain labels is consistent with their correlation with more fundamental token\-level statistics\. Mathematics and code data tend to exhibit higher CondOTMass and lower Entropy, whereas general\-domain data tends to exhibit lower CondOTMass and higher Entropy\. Nonetheless, domain labels remain a coarse proxy for these underlying signals\. To evaluate more direct routing criteria, we next examine routing policies defined directly in terms of the diagnostic metrics, using OTMass and Entropy as two representative signals\.111The domain\-level analysis in Section[7\.2](https://arxiv.org/html/2607.16246#S7.SS2)decomposes OTMass into Coverage@kkand CondOTMass to separately characterize coverage and within\-support mass variation across domains\. For token\-level routing, however, CondOTMass is undefined at positions wherexr∉Kr\(T\)x\_\{r\}\\notin K\_\{r\}^\{\(T\)\}, precluding its direct use as a per\-token routing signal\. OTMass is well\-defined at every position—it equals zero whenxr∉Kr\(T\)x\_\{r\}\\notin K\_\{r\}^\{\(T\)\}\(Eq\. \([18](https://arxiv.org/html/2607.16246#S6.E18)\)\)—and jointly captures both coverage and within\-support mass allocation, making it the appropriate signal for per\-token routing decisions\.

### 7\.3Token\-Level Objective Routing by OTMass and Teacher Entropy

We evaluate two token\-level routing policies that use OTMass and teacher entropy, respectively, as per\-token routing signals\. Both policies are motivated in part by the objective\-selection rationale ofGoyalet al\.\([2026](https://arxiv.org/html/2607.16246#bib.bib2)\):LLML\_\{\\mathrm\{LM\}\}is assigned to tokens for which the teacher distribution is unlikely to provide meaningful supervision beyond the observed token, whileLKDL\_\{\\mathrm\{KD\}\}is assigned to tokens for which the teacher may supply useful distributional guidance over alternatives\.

For OTMass\-based routing,LKDL\_\{\\mathrm\{KD\}\}with top\-256256truncation and temperatureτ=1\\tau=1is applied to tokens with low OTMass, andLLML\_\{\\mathrm\{LM\}\}to the remaining tokens\. Based on the OTMass statistics in Section[7\.2](https://arxiv.org/html/2607.16246#S7.SS2), we select two thresholds corresponding approximately to the 30th and 70th quantiles of the OTMass distribution:TM∈\{0\.3678,0\.9876\}T\_\{M\}\\in\\\{0\.3678,0\.9876\\\}\. For entropy\-based routing,LLML\_\{\\mathrm\{LM\}\}is applied to tokens with low entropy andLKDL\_\{\\mathrm\{KD\}\}with top\-256256truncation and temperatureτ=1\\tau=1to tokens with high entropy\. Based on the entropy statistics in Section[7\.2](https://arxiv.org/html/2607.16246#S7.SS2), we select two thresholds corresponding approximately to the 30th and 50th quantiles:TE∈\{0\.1,0\.5\}T\_\{E\}\\in\\\{0\.1,0\.5\\\}\. All other training hyperparameters follow the configuration in Section[5\.1](https://arxiv.org/html/2607.16246#S5.SS1), and evaluation follows the protocol in Section[4\.3](https://arxiv.org/html/2607.16246#S4.SS3)\. The standard benchmark and Pass@K results are presented in Tables[12](https://arxiv.org/html/2607.16246#S7.T12),[13](https://arxiv.org/html/2607.16246#S7.T13),[14](https://arxiv.org/html/2607.16246#S7.T14)and[15](https://arxiv.org/html/2607.16246#S7.T15)\.

Table 12:Benchmark\-level comparison under different OTMass routing strategies\.Table 13:Pass@KKcomparison under different OTMass routing strategies\.Table 14:Benchmark\-level comparison under different entropy routing strategies\.Table 15:Pass@KKcomparison under different entropy routing strategies\.##### Token\-level routing does not provide a consistent overall gain overLLML\_\{\\mathrm\{LM\}\}orLKDL\_\{\\mathrm\{KD\}\}\.

The results for OTMass routing are mixed\. On DROP and TriviaQA, whereLKDL\_\{\\mathrm\{KD\}\}is stronger thanLLML\_\{\\mathrm\{LM\}\}, theTM=0\.3678T\_\{M\}=0\.3678variant exceeds both single\-objective baselines: it achieves54\.4554\.45on DROP \(vs\.53\.3753\.37forLKDL\_\{\\mathrm\{KD\}\}and51\.5951\.59forLLML\_\{\\mathrm\{LM\}\}\) and56\.2656\.26on TriviaQA \(vs\.55\.4955\.49and54\.6554\.65\)\. On BBH, the same variant also surpasses both baselines, reaching70\.5070\.50against69\.3369\.33forLKDL\_\{\\mathrm\{KD\}\}and69\.9069\.90forLLML\_\{\\mathrm\{LM\}\}\. These gains are consistent with the view that OTMass captures a useful per\-token routing signal, though benchmark\-level aggregates alone do not identify the specific token subsets responsible\.

These improvements are not systematic, however\. On benchmarks whereLLML\_\{\\mathrm\{LM\}\}is clearly stronger, OTMass routing only partially recovers theLLML\_\{\\mathrm\{LM\}\}advantage: on MMLU\-Pro, the two OTMass variants achieve41\.4241\.42and40\.7740\.77against43\.9843\.98forLLML\_\{\\mathrm\{LM\}\}; on MATH \(Minerva\), they achieve41\.9841\.98and41\.1041\.10against43\.3843\.38\. The limitation is most pronounced in pass@KKevaluation: on MATH\-500 Pass@128, the best OTMass variant reaches74\.0074\.00, a partial recovery fromLKDL\_\{\\mathrm\{KD\}\}\(63\.0063\.00\) but still well short ofLLML\_\{\\mathrm\{LM\}\}\(84\.4084\.40\)\. In aggregate, OTMass routing mitigates the degradation associated withLKDL\_\{\\mathrm\{KD\}\}but does not consistently restore the performance ofLLML\_\{\\mathrm\{LM\}\}on high\-difficulty reasoning and generation tasks\.

Entropy\-based routing shows a similar lack of consistent dominance\. TheTE=0\.5T\_\{E\}=0\.5variant produces competitive results on several benchmarks—nearly matchingLLML\_\{\\mathrm\{LM\}\}on MMLU\-Pro \(43\.8643\.86vs\.43\.9843\.98\) and marginally exceeding both baselines on GSM8K \(71\.3471\.34vs\.70\.4370\.43and70\.7470\.74\) and MBPP \(55\.0055\.00vs\.54\.8054\.80\)—and improves AIME 2024 Pass@128 from10\.0010\.00underLKDL\_\{\\mathrm\{KD\}\}to13\.3313\.33\. However, these gains do not form a consistent pattern, and entropy routing can produce marked regressions\. The most notable case is RACE:LLML\_\{\\mathrm\{LM\}\}obtains46\.8946\.89andLKDL\_\{\\mathrm\{KD\}\}obtains45\.7445\.74, whereas theTE=0\.5T\_\{E\}=0\.5variant drops to39\.3339\.33\. On BBH, both entropy variants \(68\.1968\.19and68\.7068\.70\) fall belowLLML\_\{\\mathrm\{LM\}\}\(69\.9069\.90\),LKDL\_\{\\mathrm\{KD\}\}\(69\.3369\.33\), and the OTMass routing results\. These observations suggest that teacher entropy alone is an insufficient and potentially unreliable signal for per\-token objective selection\.

##### OTMass routing is more stable than entropy routing\.

Although neither token\-level routing strategy consistently outperforms both single\-objective baselines, OTMass routing is more stable across the benchmarks considered here, particularly in avoiding large regressions\. On RACE, OTMass routing remains close toLKDL\_\{\\mathrm\{KD\}\}, with scores of45\.6545\.65and45\.8445\.84, whereas entropy routing withTE=0\.5T\_\{E\}=0\.5drops to39\.3339\.33\. On BBH, OTMass routing obtains70\.5070\.50and69\.5469\.54, compared with68\.1968\.19and68\.7068\.70for entropy routing\. On GSM8K, OTMass routing varies from69\.5269\.52to70\.8970\.89, against a wider spread of68\.3168\.31to71\.3471\.34for entropy routing\. In pass@KKevaluation, OTMass routing also provides a more predictable interpolation between the two single\-objective baselines than entropy routing does\.

This difference is consistent with the properties of the two metrics\. OTMass depends jointly on the teacher distribution and the observed training token: it measures the probability mass assigned by the teacher to the observed token under top\-kktruncation and is zero when the observed token falls outside the support\. It is thus closely connected to the observed\-token component of theLLML\_\{\\mathrm\{LM\}\}–LKDL\_\{\\mathrm\{KD\}\}gradient discrepancy described in Eq\. \([11](https://arxiv.org/html/2607.16246#S6.E11)\), and may be expected to capture teacher–data alignment more directly than teacher\-side statistics alone\. Entropy, by contrast, characterizes only the concentration of the teacher distribution, independently of the observed token\. A low\-entropy teacher distribution indicates that the teacher assigns high confidence to some token, but that token need not be the observed one\. As entropy does not condition on the observed training token, it cannot in general distinguish between a teacher that agrees with the data and one that confidently favors an alternative; this may contribute to entropy\-based routing being more sensitive to threshold choice and more susceptible to abrupt performance regressions\.

##### Domain\-level subset routing appears to outperform these token\-level routing strategies\.

Compared with both token\-level policies, domain\-based subset\-level routing tends to yield a more favorable and more consistent improvement pattern\. On the standard benchmark suite, domain routing recovers theLLML\_\{\\mathrm\{LM\}\}advantage on tasks whereLKDL\_\{\\mathrm\{KD\}\}underperforms while preserving theLKDL\_\{\\mathrm\{KD\}\}advantage where it is favorable\. On MMLU\-Pro, domain routing achieves43\.7243\.72—close toLLML\_\{\\mathrm\{LM\}\}\(43\.9843\.98\) and well above the best OTMass variant \(41\.4241\.42\)\. On MATH \(Minerva\), domain routing reaches43\.5843\.58, slightly exceedingLLML\_\{\\mathrm\{LM\}\}\(43\.3843\.38\) and outperforming all token\-level routing variants\. On MBPP, domain routing achieves58\.4058\.40, surpassingLLML\_\{\\mathrm\{LM\}\}\(53\.0053\.00\),LKDL\_\{\\mathrm\{KD\}\}\(54\.8054\.80\), and the best token\-level results\.

The advantage of domain routing is more pronounced in pass@KKevaluation\. On MATH\-500 Pass@128, domain routing reaches83\.2083\.20, far closer toLLML\_\{\\mathrm\{LM\}\}\(84\.4084\.40\) than the best OTMass routing result \(74\.0074\.00\)\. On AIME 2024 Pass@128, domain routing achieves33\.3333\.33, exceedingLLML\_\{\\mathrm\{LM\}\}\(23\.3323\.33\),LKDL\_\{\\mathrm\{KD\}\}\(10\.0010\.00\), and the best token\-level result \(13\.3313\.33\); a similar pattern holds on AIME 2025, where domain routing matchesLLML\_\{\\mathrm\{LM\}\}\(23\.3323\.33\) while the best token\-level variant reaches only16\.6716\.67\. On HumanEval at Pass@128, domain routing obtains92\.6892\.68, compared with92\.0792\.07for the best entropy routing variant and90\.8590\.85for the best OTMass variant\. These results indicate that, within the current experimental setup, coarse domain information provides a more effective basis for objective routing than the per\-token statistics examined here\.

One possible explanation for this gap is that token\-level routing may be more sensitive to context contamination introduced by the pre\-training data format\. In large\-scale pre\-training, multiple samples are typically packed into fixed\-length sequences, and training is often performed without cross\-sample attention masks for efficiency\. As a result, teacher logits at any position may reflect not only the local sample content but also adjacent packed samples, potentially perturbing fine\-grained per\-token quantities such as OTMass and entropy and introducing noise into the routing decisions\. Domain\-level routing, operating at a coarser granularity, is likely less sensitive to such local perturbations\. We leave a more systematic investigation of this packed\-context effect and its implications for token\-level objective routing to future work\.

### 7\.4Summary of Findings

The experimental results provide a qualified but substantive answer to RQ2\. Training objectives can be routed adaptively across the training distribution using statistics derived from the training data, and such adaptive routing offers a more targeted mechanism for managing capability trade\-offs than a fixed global objective\. The effectiveness of this approach, however, appears to depend largely on the routing signal: increasing routing granularity alone does not appear sufficient, and the routing criterion may need to be reasonably aligned with the capability differences induced by the two objectives\.

The clearest evidence for adaptive routing comes from domain\-based subset\-level routing\. AssigningLLML\_\{\\mathrm\{LM\}\}to mathematics and code subsets andLKDL\_\{\\mathrm\{KD\}\}to the remaining general\-domain data yields consistent aggregate improvements over both single\-objective baselines, and in several cases—including MBPP and the AIME benchmarks—the routed model exceeds bothLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}individually\. The routing strategy largely recovers the advantage ofLLML\_\{\\mathrm\{LM\}\}on tasks where distillation underperforms, particularly on high\-difficulty reasoning and sampling\-intensive evaluations, while preserving the advantage ofLKDL\_\{\\mathrm\{KD\}\}on knowledge\-oriented and commonsense benchmarks where teacher\-distribution supervision is beneficial\. These results indicate that coarse domain structure provides a reliable basis for objective routing, at least for broad capability dimensions such as mathematical reasoning, coding, and general\-domain language understanding, and suggest that selectively combining complementary supervision signals can produce outcomes stronger than either objective alone\.

Between the two token\-level signals, OTMass appears to be the more reliable routing criterion\. This is broadly consistent with its interpretation as a joint teacher–data alignment statistic: OTMass directly measures the teacher probability mass assigned to the observed training token under top\-kktruncation, and is thus closely connected to the observed\-token component of theLLML\_\{\\mathrm\{LM\}\}–LKDL\_\{\\mathrm\{KD\}\}gradient discrepancy characterized in Eq\. \([11](https://arxiv.org/html/2607.16246#S6.E11)\)\. Teacher entropy, by contrast, reflects only the concentration of the teacher distribution independently of the observed token; it cannot distinguish between a teacher that agrees with the data and one that confidently favors an alternative, which may make it a less principled and more threshold\-sensitive routing criterion\.

These findings point to several tentative conclusions about the design of effective routing signals\. First, signals that jointly reflect the teacher distribution and its alignment with the observed training token may be more informative for objective routing than teacher\-only uncertainty measures\. Second, at token\-level granularity, routing signals may be susceptible to noise introduced by the packed\-sequence training format, in which teacher logits at a given position can be influenced by adjacent samples packed into the same sequence; operating at coarser granularities may average over such local variability and yield more stable routing decisions\. Third, and most broadly, the advantage of domain\-level routing over the token\-level policies evaluated here suggests that the effectiveness of adaptive routing need not increase monotonically with routing granularity\. Identifying more principled routing criteria—including sequence\-level, capability\-aware, and learned routing signals—remains an important direction for future work\.

## 8Conclusion

##### Summary\.

This paper examines continued pre\-training via off\-policy distillation through the joint lens of token\-level supervision, KD parameterization, and data heterogeneity\. We begin by establishing thatLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}do not differ merely in degree but in kind: under otherwise identical training conditions, the two objectives induce systematically distinct capability profiles\.LLML\_\{\\mathrm\{LM\}\}is consistently stronger on high\-difficulty reasoning, mathematical problem solving, and knowledge\-intensive evaluations—with performance gaps that widen substantially under large\-budget Pass@KKevaluation—whileLKDL\_\{\\mathrm\{KD\}\}is more favorable on commonsense plausibility, factual retrieval, reading comprehension, and structured program synthesis\. To explain this divergence, we analyze the two objectives at the gradient level and identify a fundamental tension between two complementary supervision signals:*direct observed\-token reinforcement*, which is the exclusive mode ofLLML\_\{\\mathrm\{LM\}\}, and*teacher\-supported alternative supervision*, whichLKDL\_\{\\mathrm\{KD\}\}introduces by distributing target mass across the teacher top\-kksupport\. We introduce a suite of diagnostic metrics—Coverage@kk, OTRank, OTMass, CondOTMass, and teacher\-support entropy—to quantify this balance at each training position\. Controlled sweeps over the two principal KD degrees of freedom show that the support sizekkand distillation temperatureτ\\tauoperate through distinct mechanisms:kkgoverns a coverage–sharpness trade\-off in which expanding the support reduces observed\-token exclusion at the cost of a progressively more diffuse target, whileτ\\taucontrols within\-support probability allocation without altering support membership\. The resulting downstream effects are qualitatively distinct and task\-dependent, consistent with the view that different tasks favor different positions along the spectrum between the two complementary supervision signals\.

We then investigate whether objective routing can be made adaptive to the training data\. A domain\-based routing policy—applyingLLML\_\{\\mathrm\{LM\}\}to mathematics and code subsets andLKDL\_\{\\mathrm\{KD\}\}to general\-domain data—yields consistent aggregate improvements over both single\-objective baselines, recovering theLLML\_\{\\mathrm\{LM\}\}advantage on reasoning\-intensive and sampling\-intensive evaluations while preserving theLKDL\_\{\\mathrm\{KD\}\}advantage on knowledge\-oriented and commonsense benchmarks; in several cases, including MBPP and the AIME benchmarks, the routed model exceeds both baselines simultaneously\. Analysis of the diagnostic metrics stratified by domain reveals that this effectiveness does not appear to be incidental: mathematics and code data exhibit substantially higher OTMass and lower teacher\-support entropy than general\-domain data, indicating that domain labels correlate with more fundamental differences in teacher–data alignment\. Finer\-grained token\-level routing via OTMass and teacher entropy, however, does not consistently match the stronger single\-objective baseline and falls well short of domain\-level routing on high\-difficulty reasoning evaluations\. Between the two token\-level signals, OTMass appears to be the more reliable criterion, plausibly in that it jointly reflects the teacher distribution and its alignment with the observed training token, whereas teacher entropy characterizes only teacher\-side distributional concentration and cannot distinguish a teacher that agrees with the data from one that confidently predicts an alternative\. Taken together, these findings suggest that routing effectiveness need not increase monotonically with routing granularity, and that principled objective routing may instead call for signals grounded in teacher–data alignment rather than teacher\-side uncertainty alone\.

##### Limitations\.

This work has several limitations\. First, the gap between domain\-level and token\-level routing also remains incompletely explained\. Our experiments suggest that domain\-based subset routing tends to outperform per\-token policies based on OTMass or teacher entropy, suggesting that finer routing granularity is not necessarily advantageous in itself\. A plausible contributing factor is the sequence\-packing format common in large\-scale pre\-training: when multiple documents are packed into a single fixed\-length sequence without cross\-sample attention masking, teacher logits at a given position may reflect surrounding packed content rather than the local document context alone, perturbing token\-level diagnostic statistics and weakening their alignment with the intended routing criterion\. Whether this contamination fully accounts for the observed gap—and whether token\-level routing with cleaner document boundaries or attention masks can recover stronger performance—remains an open question\.

Second, our capability characterization is grounded in benchmark performance, which is an imperfect proxy for isolated model abilities\. Individual benchmarks typically conflate multiple factors—factual recall, multi\-step reasoning, reading comprehension, numerical manipulation, and decoding sensitivity—so the performance profiles reported here should be interpreted as empirical regularities at the benchmark level rather than as clean decompositions of latent capabilities\. Establishing sharper causal links between training\-objective choices and specific capability dimensions would require evaluation protocols with explicitly annotated skill factors and controlled data\-to\-capability mappings\.

Finally, due to computational and experimental budget constraints, all experiments are conducted under a continued pre\-training setup, in which the student is warm\-started from a checkpoint derived via a pruning–distillation scaling ladder, rather than trained from scratch; in addition, we do not evaluate our method across different model families\. Whether the qualitative patterns reported here—particularly the optimal ranges ofkkandτ\\tauand the relative advantage of domain\-level over token\-level routing—transfer to from\-scratch pre\-training regimes or to other model families remains an open question\.

##### Future Work\.

A natural extension of this work is to develop a principled framework for adaptive objective routing in off\-policy distillation, organized around three components: the routing signal, the routing objective, and the routing rule\. On the signal side, the central questions are what information should govern objective assignment and at what granularity routing decisions should be made\. Routing can operate at the level of domains, documents, sequences, spans, or individual tokens, and finer granularity trades stability for adaptivity while also increasing susceptibility to local noise—including the packed\-sequence contamination discussed in Section[7\.3](https://arxiv.org/html/2607.16246#S7.SS3)\. Candidate signals include domain labels, observed\-token probability mass, teacher–data alignment metrics, and gradient\-based diagnostics\.

On the objective side, the design space extends beyond the binary choice betweenLLML\_\{\\mathrm\{LM\}\}andLKDL\_\{\\mathrm\{KD\}\}explored here\. The KD family spans a continuous range of supervision signals parameterized by the top\-kksupport size, distillation temperature, LM–KD interpolation weight, and the choice of divergence measure, each of which shifts the balance between direct observed\-token reinforcement and teacher\-supported alternative supervision in a distinct way\. A more general routing framework could assign not only whether to apply distillation, but also which configuration within the KD family is most appropriate for a given data region—treating sparse KD as a structured family of supervision signals rather than a monolithic objective, and enabling more targeted capability shaping across heterogeneous training data\.

On the rule side, the threshold\-based policies evaluated in this work are interpretable and computationally inexpensive, but may be suboptimal when multiple signals interact or when capability trade\-offs are task\-specific in ways that fixed thresholds cannot capture\. More flexible alternatives, such as learned routing policies that map diagnostic features to objective assignments, raise further questions about router supervision, training stability, computational overhead, and generalization across model scales and data distributions\. Taken together, these directions point toward a broader reconception of off\-policy distillation: rather than a global objective to be tuned empirically, it is more productively framed as a structured supervision design problem whose central challenge is to characterize and exploit the interplay among data heterogeneity, training objective, and model capability\.

## References

- A\. Amini, A\. Banaszak, H\. Benoit, A\. Böök, T\. Dakhran, S\. Duong, A\. Eng, F\. Fernandes, M\. Härkönen, A\. Harrington,et al\.\(2025\)Lfm2 technical report\.arXiv preprint arXiv:2511\.23404\.Cited by:[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8)\.
- A\. Anshumann, M\. A\. Zaidi, A\. Kedia, J\. Ahn, T\. Kwon, K\. Lee, H\. Lee, and J\. Lee \(2025\)Sparse logit sampling: accelerating knowledge distillation in llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 18085–18108\.Cited by:[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8)\.
- S\. Dasgupta, T\. Cohn, and T\. Baldwin \(2026\)Don’t ignore the tail: decoupling top\-k probabilities for efficient language model distillation\.arXiv preprint arXiv:2602\.20816\.Cited by:[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8)\.
- Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, T\. Mesnard,et al\.\(2025\)Gemma 3 Technical Report\.arXiv preprint arXiv:2503\.19786\.External Links:[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§1](https://arxiv.org/html/2607.16246#S1.p4.2),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8)\.
- S\. Goyal, D\. Lopez\-Paz, and K\. Ahuja \(2026\)Distilled pretraining: a modern lens of data, in\-context learning and test\-time scaling\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PNm2dl7HcY)Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p4.2),[§1](https://arxiv.org/html/2607.16246#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p4.2),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p2.3),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p5.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p3.2),[§4\.3\.2](https://arxiv.org/html/2607.16246#S4.SS3.SSS2.p1.5),[§7\.3](https://arxiv.org/html/2607.16246#S7.SS3.p1.2)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p3.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p1.2),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p5.1)\.
- H\. Huang, J\. Song, Y\. Zhang, and P\. Ren \(2026\)SelecTKD: selective token\-weighted knowledge distillation for llms\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19287–19297\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p3.2)\.
- X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu \(2020\)Tinybert: distilling bert for natural language understanding\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 4163–4174\.Cited by:[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p1.1)\.
- A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan,et al\.\(2026\)Ministral 3\.arXiv preprint arXiv:2601\.08584\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§1](https://arxiv.org/html/2607.16246#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p2.3)\.
- S\. Muralidharan, S\. Turuvekere Sreenivas, R\. Joshi, M\. Chochowski, M\. Patwary, M\. Shoeybi, B\. Catanzaro, J\. Kautz, and P\. Molchanov \(2024\)Compact language models via pruning and knowledge distillation\.Advances in Neural Information Processing Systems37,pp\. 41076–41102\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§1](https://arxiv.org/html/2607.16246#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p2.3)\.
- H\. Peng, X\. Lv, Y\. Bai, Z\. Yao, J\. Zhang, L\. Hou, and J\. Li \(2025\)Pre\-training distillation for large language models: a design space exploration\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3603–3618\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p3.1),[§1](https://arxiv.org/html/2607.16246#S1.p4.2),[§1](https://arxiv.org/html/2607.16246#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p4.2),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p2.3),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p4.8),[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p5.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p3.2)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p1.1)\.
- M\. Shing, K\. Misaki, H\. Bao, S\. Yokoi, and T\. Akiba \(2025\)TAID: temporally adaptive interpolated distillation for efficient knowledge transfer in language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cqsw28DuMW)Cited by:[§2\.2](https://arxiv.org/html/2607.16246#S2.SS2.p3.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p2.1)\.
- Z\. Sun, H\. Yu, X\. Song, R\. Liu, Y\. Yang, and D\. Zhou \(2020\)Mobilebert: a compact task\-agnostic bert for resource\-limited devices\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 2158–2170\.Cited by:[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p1.1)\.
- S\. Tang, Z\. Wang, B\. Zheng, L\. Wang, R\. Men, S\. Zhang, X\. Yuan, Z\. Qiu, Z\. Shen, and D\. Liu \(2026\)SlimQwen: exploring the pruning and distillation in large moe model pre\-training\.arXiv preprint arXiv:2605\.08738\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p3.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p2.1)\.
- W\. Wang, H\. Bao, S\. Huang, L\. Dong, and F\. Wei \(2021\)Minilmv2: multi\-head self\-attention relation distillation for compressing pretrained transformers\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 2140–2151\.Cited by:[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p1.1)\.
- W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou \(2020\)Minilm: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.Advances in neural information processing systems33,pp\. 5776–5788\.Cited by:[§2\.1](https://arxiv.org/html/2607.16246#S2.SS1.p1.1)\.
- X\. Xie, Z\. Xue, J\. Wu, J\. Li, Y\. Wang, X\. Hu, Y\. Liu, and J\. Zhang \(2026\)Llm\-oriented token\-adaptive knowledge distillation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34070–34078\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p3.2)\.
- H\. Zhang, Z\. Zhang, G\. Wu, W\. Ning, J\. Guo, and X\. Cheng \(2026\)EGAD: entropy\-guided adaptive distillation for token\-level knowledge transfer\.arXiv preprint arXiv:2605\.01732\.Cited by:[§1](https://arxiv.org/html/2607.16246#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.16246#S2.SS3.p3.2)\.