X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation

# X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
Source: [https://arxiv.org/html/2605.21699](https://arxiv.org/html/2605.21699)
\\correspondingauthor

sharatht@nvidia\.com

Adithyakrishna Venkatesh HanasogeMingyu YangAli TaghibakhshiSaurav MuralidharanAshwath AithalPavlo Molchanov

###### Abstract

Abstract:Cross\-tokenizer knowledge distillation allows a student model to learn from teachers with incompatible vocabularies\. Prior work operates on hidden states or logits; the latter is preferred as a drop\-in replacement requiring no auxiliary components\. Logit\-based methods either use only the correct\-token probability, missing the full ’dark knowledge’ in the teacher’s distribution, or operate on the full output distribution, relying on strict token partitioning and/or unprincipled heuristic ranking\. We identify two key shortcomings of full\-distribution, logit\-based methods: \(*i*\) an*uncommon\-token failure*, where critical tokens fall into the unmatched subset \(e\.g\., Llama producing 1100 multi\-digit numerals under digit\-splitting Qwen supervision\) and are suppressed during training, reducing GSM8k from 12\.89 to 2\.56 compared to same\-tokenizer KD from a weaker teacher; and \(*ii*\)*over\-conservative matching*, where strict 1\-to\-1 matching excludes near\-equivalent tokens across surface forms\. These failures require distinct remedies: eliminating the partition when critical tokens are misaligned, and refining it when alignment is reliable\. We proposeX\-Token, an approach with two complementary loss formulations targeting these issues\.P\-KLremoves partitioning and aligns the student’s distribution with the teacher’s via a sparse projection matrix W \(initialized from tokenizer\-level string rules\) to address the uncommon\-token failure\.H\-KLretains the hybrid form while relaxing matching to align each student token with its top\-ranked teacher mapping underWW\. Both objectives shareWWand extend naturally to multiple teachers\. Empirically, on Llama\-3\.2\-1B, X\-Token outperforms the current state of the art GOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\]by \+3\.82 average points with a Qwen3\-4B teacher and by \+0\.5 with a Phi\-4\-Mini teacher\. Further, a two\-teacher setup \(Phi\-4\-mini\+\+Llama\-3B\) improves over single\-teacher distillation by \+1\.3 points\.

## 1Introduction

Knowledge distillation \(KD\)\[hinton2015distilling,romero2014fitnets,furlanello2018born,park2019relational\]transfers the ‘dark knowledge’ in a teacher’s output distribution to a student, typically via per\-position Kullback–Leibler \(KL\) divergence over next\-token probability distribution\. This formulation requires a shared tokenizer, effectively tying the student to same\-family teachers\. As a result, a practitioner committed to a given tokenizer \(*e\.g\.*, Llama\-3\.2\-1B\[grattafiori2024llama\]\) cannot leverage stronger or more specialized teachers with incompatible tokenizers \(*e\.g\.*, Phi\-4\-mini\[abouelenin2025phi\], Qwen3\-4B\[yang2025qwen3\]\)\. This constraint also prevents multi\-teacher distillation across tokenizer families, limiting the ability to combine teachers with complementary strengths \(*e\.g\.*, reasoning, code, multilingual\) into a unified training signal\. Cross\-tokenizer distillation removes this restriction, enabling both freedom from teacher\-tokenizer lock\-in and effective multi\-teacher learning from diverse sources\.

![Refer to caption](https://arxiv.org/html/2605.21699v1/figures/Fig1.png)Figure 1:Left:multi\-teacher distillation routes each teacher through its appropriate loss — KL for the same\-tokenizer Llama\-3\.2\-3B, P\-KL/H\-KL for cross\-tokenizer Qwen3\-4B and Phi\-4\-mini\.Right:X\-Token addresses two failure modes of GOLD’s string\-equality partition and composes across teachers\.Right, top:the critical token201has no GOLD match and receives erroneous signal;P\-KLconnects it to\{2,0,1\}\\\{2,0,1\\\}in𝒱T\\mathcal\{V\}\_\{T\}via the projectionWW\.Right, bottom:Hundredsis excluded from GOLD’s common\-KL term;H\-KLadmits\(Hundreds,Hund\)\(\\texttt\{Hundreds\},\\texttt\{Hund\}\)via the top\-11ofWW\.Existing cross\-tokenizer KD methods fall into two broad families:*representation\-based*approaches that align the teacher and student at the embedding or hidden\-state level \(*e\.g\.*, DSKD\[zhang2024dual\]\), and*logit\-distribution\-based*approaches that operate directly on output distributions \(*e\.g\.*, ULD\[boizard2024towards\], GOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\]\)\. The latter approaches are particularly attractive at continual\-pretraining scale, as they require no auxiliary trainable components and integrate as drop\-in replacements for the standard KD loss, without modifying the model architecture or introducing additional forward passes\.

GOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\]approach, applies a hybrid loss that partitions tokens into a 1\-to\-1 string\-matched*common*subset trained with KL divergence and an*uncommon*remainder matched via rank\-based L1 following ULD\[boizard2024towards\]\. However, this hybrid design exhibits two structural limitations\. First, an*uncommon\-token failure*: when tokenizers fragment differently \(*e\.g\.*, Qwen3 splits multi\-digit numerals while Llama\-3 packs them as single tokens\),*critical tokens*—tokens whose correct prediction directly determines task accuracy \(*e\.g\.*, multi\-digit numerals in GSM8k\)—are forced into the unmatched subset\. These tokens are then degraded by \(i\) identity\-agnostic noise from rank\-based matching and \(ii\) suppressive gradients from the common\-KL term acting through the full\-vocabulary softmax\. Second,*over\-conservative matching*: strict string\-equality excludes equivalent token pairs across tokenizers, both in surface form and over teacher multi\-token decompositions—leaving useful alignment signal unexploited even when the partition is otherwise well\-formed \(*e\.g\.*, a student tokenHundredscorresponds to teacher tokensHundfollowed byreds, but strict matching discards this correspondence due to lack of exact string equality\)\.

We proposeX\-Token\(Figure[1](https://arxiv.org/html/2605.21699#S1.F1)\), which addresses the limitations of current methods and makes the following contributions:

- •Deterministic cross\-tokenizer alignment\.We introduce a sparse projection matrixWW, constructed via canonicalized string matching and multi\-token decoding rules, enabling direct alignment across tokenizers;WWcan be optionally refined during KD for additional gains\.
- •Complementary loss formulations \(P\-KL and H\-KL\) and loss\-selection criteria\.P\-KLremoves partitioning and aligns full distributions, whileH\-KLrelaxes matching via top\-ranked mappings underWW\. A simple coverage audit over token categories \(*e\.g\.*, numerals\) guides selection: use P\-KL when critical tokens fall outside the common set, and H\-KL otherwise\. P\-KL improves over GOLD by\+3\.82avg\. with Qwen3\-4B \(including a𝟔×\\mathbf\{6\\times\}GSM8k,2\.56→15\.542\.56\\rightarrow\\mathbf\{15\.54\}\); H\-KL adds consistent gains of\+0\.5with Phi\-4\-mini\.
- •Multi\-teacher KD across tokenizer families\.X\-Token enables distillation from heterogeneous teachers\. We show that*complementarity*is key \(Phi\-4\-mini\+\+Llama\-3\.2\-3B yields\+1\.3over single\-teacher KD\) and that simple*static*weighting outperforms adaptive schemes\.
- •Robust sequence alignment for KD\.We provide deterministic, scalable DP\-based alignment of student and teacher input sequences for KD\.

## 2Method

X\-Token consists of three components: \(i\) span alignment to produce text\-consistent units, \(ii\) a projection matrixWWto bridge vocabularies, and \(iii\) two complementary loss formulations,P\-KLandH\-KL, with an optional multi\-teacher extension\. Together, these enable distillation across mismatched tokenizers\. All loss formulations operate on chunk\-level distributions obtained via span alignment and chain\-rule merge, which combines per\-token probabilities within each aligned span via the autoregressive product into a single chunk\-level distribution\.

### 2\.1Sequence Alignment

When teacher and student use different tokenizers𝒯S,𝒯T\\mathcal\{T\}\_\{S\},\\mathcal\{T\}\_\{T\}, token sequences differ in length and lack positional correspondence, making per\-position KD ill\-defined\.

We address this via*span alignment*, grouping tokens into chunks\{\(AkS,AkT\)\}k=1K\\\{\(A\_\{k\}^\{S\},A\_\{k\}^\{T\}\)\\\}\_\{k=1\}^\{K\}that decode to the same underlying text\. We then apply a chain\-rule merge over each chunk to obtain chunk\-level distributionsp^S\(k\)\\hat\{p\}\_\{S\}^\{\(k\)\}andp^T\(k\)\\hat\{p\}\_\{T\}^\{\(k\)\}, which serve as aligned units for distillation\. Such approach was inspired by\[minixhofer2025universal\]\.

### 2\.2X\-Token Projection MatrixWW

Even after alignment, teacher and student distributions are defined over different vocabularies\. We introduce a projection matrixW∈ℝ\|𝒱S\|×\|𝒱T\|W\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{S\}\|\\times\|\\mathcal\{V\}\_\{T\}\|\}that maps student\-token probabilities into teacher vocabulary space, where𝒱S\\mathcal\{V\}\_\{S\}and𝒱T\\mathcal\{V\}\_\{T\}represents the student and teacher vocabularies\.

![Refer to caption](https://arxiv.org/html/2605.21699v1/x1.png)Figure 2:Subset of the projection matrixWWfor a Llama\-3\.2 student and Qwen\-3 teacher\. Exact matches include\_the,\_cat, and\_run\. For tokens without exact matches, the multi\-token rule is applied:*e\.g\.*,201→\(2,0,1\)\\to\(2,0,1\), and the Greek prefix\_π​ε​ρ​ι​o\\pi\\varepsilon\\rho\\iota omaps to five teacher sub\-tokens, with the lower weight entries \(hatched\) truncated \(top\-KK=4\)\.As visualized in Figure[2](https://arxiv.org/html/2605.21699#S2.F2), the projection matrixW∈ℝ\|𝒱S\|×\|𝒱T\|W\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{S\}\|\\times\|\\mathcal\{V\}\_\{T\}\|\}maps each student token to a weighted combination of teacher tokens\. We constructWWdeterministically in two passes\.*\(1\) Exact\-match pass*: for every\(s,t\)∈𝒱S×𝒱T\(s,t\)\\in\\mathcal\{V\}\_\{S\}\\times\\mathcal\{V\}\_\{T\}whose decoded strings match after canonicalization \(unifies space prefixes such asĠ/\_and newline markers\), setW​\[s,t\]=1W\[s,t\]=1\. We usessandttinterchangeably with their integer indices under the canonical vocabulary ordering when indexing arrays or matrices\.*\(2\) Multi\-token\-rule pass*: for each remaining student tokenss, re\-tokenize its decoded text under the teacher tokenizer to yield a sequence\(τ0,…,τℓ\)∈𝒱T\(\\tau\_\{0\},\\ldots,\\tau\_\{\\ell\}\)\\in\\mathcal\{V\}\_\{T\}\(hereτi∈𝒱T\\tau\_\{i\}\\in\\mathcal\{V\}\_\{T\}is theii\-th index in this re\-tokenization\), and setW​\[s,τi\]=β​γiW\[s,\\tau\_\{i\}\]=\\beta\\,\\gamma^\{i\}with\(β,γ\)=\(0\.9,0\.1\)\(\\beta,\\gamma\)=\(0\.9,0\.1\)\. Each row is truncated to its top\-4 entries and normalized\. The matrixWWis constructed once before training and can be optionally fine\-tuned during KD; full pseudocode is provided in Appendix[8](https://arxiv.org/html/2605.21699#S8)\.

### 2\.3Knowledge Distillation

We adopt the standard KD objective\[hinton2015distilling\], but apply it over aligned chunks\. Given chunk\-level distributions\{p^S\(k\),p^T\(k\)\}k=1K\\\{\\hat\{p\}\_\{S\}^\{\(k\)\},\\hat\{p\}\_\{T\}^\{\(k\)\}\\\}\_\{k=1\}^\{K\}, we compute KL on the top\-KKteacher logits \(withK=8192K\{=\}8192\):

ℒKD=1K​∑k=1KKL​\(p^T\(k\)∥p^S\(k\)\)\.\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathrm\{KL\}\\\!\\bigl\(\\hat\{p\}\_\{T\}^\{\(k\)\}\\;\\\|\\;\\hat\{p\}\_\{S\}^\{\(k\)\}\\bigr\)\.

### 2\.4Hybrid Loss Formulation

We first formalize the partition\-based hybrid loss used in GOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\], which serves as the baseline for our loss variants\. This formulation partitions the vocabularies into a 1\-to\-1 string\-matched*common*subset𝒞\\mathcal\{C\}and uncommon remainders𝒰S,𝒰T\\mathcal\{U\}\_\{S\},\\mathcal\{U\}\_\{T\}\. It applies direct KL on𝒞\\mathcal\{C\}and rank\-sortedL1L\_\{1\}matching on𝒰=𝒰S∪𝒰T\\mathcal\{U\}=\\mathcal\{U\}\_\{S\}\\cup\\mathcal\{U\}\_\{T\}:

ℒcommon\(k\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{common\}\}^\{\(k\)\}=∑\(s,t\)∈𝒞p^T\(k\)​\[t\]​\(log⁡p^T\(k\)​\[t\]−log⁡p^S\(k\)​\[s\]\),\\displaystyle=\\sum\_\{\(s,t\)\\in\\mathcal\{C\}\}\\hat\{p\}\_\{T\}^\{\(k\)\}\[t\]\\,\\bigl\(\\log\\hat\{p\}\_\{T\}^\{\(k\)\}\[t\]\-\\log\\hat\{p\}\_\{S\}^\{\(k\)\}\[s\]\\bigr\),\(1\)ℒULD\(k\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ULD\}\}^\{\(k\)\}=∥sort↓\(p^S\(k\)\|𝒰S\)−sort↓\(p^T\(k\)\|𝒰T\)∥1,\\displaystyle=\\Bigl\\\|\\mathrm\{sort\}\_\{\\downarrow\}\\\!\\bigl\(\\hat\{p\}\_\{S\}^\{\(k\)\}\\big\|\_\{\\mathcal\{U\}\_\{S\}\}\\bigr\)\-\\mathrm\{sort\}\_\{\\downarrow\}\\\!\\bigl\(\\hat\{p\}\_\{T\}^\{\(k\)\}\\big\|\_\{\\mathcal\{U\}\_\{T\}\}\\bigr\)\\Bigr\\\|\_\{1\},\(2\)ℒGOLD\(k\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GOLD\}\}^\{\(k\)\}=λKL​ℒcommon\(k\)\+λULD​ℒULD\(k\)\.\\displaystyle=\\lambda\_\{\\mathrm\{KL\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{common\}\}^\{\(k\)\}\+\\lambda\_\{\\mathrm\{ULD\}\}\\,\\mathcal\{L\}\_\{\\mathrm\{ULD\}\}^\{\(k\)\}\.\(3\)
While this hybrid formulation enables cross\-tokenizer KD, it introduces undesirable gradient behaviors on tokens in the uncommon set, which we analyze next\.

### 2\.5P\-KL: Addressing Erroneous and Suppressive Gradients in Hybrid Loss

GOLD’s hybrid loss induces two undesirable gradient behaviors on uncommon student logits \(Figure[1](https://arxiv.org/html/2605.21699#S1.F1)\)\.

Erroneous gradients from rank\-based matching:the ULD termℒULD\\mathcal\{L\}\_\{\\mathrm\{ULD\}\}matches tokens in the uncommon set by rank, pairing each student token with a teacher token of similar rank rather than semantic correspondence\. This produces identity\-agnostic gradients that misalign critical tokens \(*e\.g\.*, numerals\) with unrelated teacher tokens \(*e\.g\.*, special characters\), degrading supervision quality\.

Suppressive gradients from the common\-KL term:the common\-KL termℒcommon\\mathcal\{L\}\_\{\\mathrm\{common\}\}\(Eq\.[1](https://arxiv.org/html/2605.21699#S2.E1)\) is computed using full\-vocabulary softmax\. Although uncommon tokens do not appear explicitly in the loss, the normalization induces gradients on all logits, reducing the relative probability of tokens in𝒰\\mathcal\{U\}\. Detailed proof can be found in the Appendix\.

Together, these effects yield weak or misdirected supervision for uncommon tokens, particularly when critical tokens fall into𝒰\\mathcal\{U\}\(*e\.g\.*, Llama’s1,1001\{,\}100multi\-digit numerals under a digit\-splitting Qwen tokenizer\), leading to degraded performance\. \(*e\.g\.*, GSM8k drops to2\.562\.56vs\.12\.8912\.89for same\-tokenizer KD from a weaker teacher\)\.

To address this limitation, we leverageP\-KL, which projects the student distributionp^S\(k\)\\hat\{p\}\_\{S\}^\{\(k\)\}into teacher vocabulary spacep~S\(k\)\\tilde\{p\}\_\{S\}^\{\(k\)\}, enabling direct alignment with the teacher distribution \(Figure[1](https://arxiv.org/html/2605.21699#S1.F1)\)\. Here,iiindexes the student vocabulary𝒱S\\mathcal\{V\}\_\{S\}andjjindexes the teacher vocabulary𝒱T\\mathcal\{V\}\_\{T\}:

p~S\(k\)​\[t\]=∑s∈𝒱SW​\[s,t\]⋅p^S\(k\)​\[s\],ℒP\(k\)=KL​\(p^T\(k\)∥p~S\(k\)\)\.\\tilde\{p\}\_\{S\}^\{\(k\)\}\[t\]=\\sum\_\{s\\in\\mathcal\{V\}\_\{S\}\}W\[s,t\]\\cdot\\hat\{p\}\_\{S\}^\{\(k\)\}\[s\],\\;\\;\\mathcal\{L\}\_\{P\}^\{\(k\)\}=\\mathrm\{KL\}\\\!\\bigl\(\\hat\{p\}\_\{T\}^\{\(k\)\}\\,\\\|\\,\\tilde\{p\}\_\{S\}^\{\(k\)\}\\bigr\)\.\(4\)
This formulation replaces both sources of error with teacher\-aware supervision over all tokens, including those in𝒰\\mathcal\{U\}\(*e\.g\.*,201onto\[2,0,1\]\[2,0,1\]\), by directly aligning the student distribution with the teacher distribution, restoring the guidance the partition discards\.

### 2\.6H\-KL: Relaxing the 1\-to\-1 Matching

When no critical token is routed into𝒰\\mathcal\{U\}— a condition we audit per category on the student vocabulary \(Table[8](https://arxiv.org/html/2605.21699#S8.T8)\) — the partition itself is a useful feature: the common\-KL on identity\-aligned pairs delivers per\-pair KL targets that are sharper than P\-KL’s projection, which blends student probability mass across multiple teacher tokens through the multi\-token\-rule rows ofWW\. The opportunity here is not to drop the partition but to make it less wasteful: GOLD’s string\-equality criterion is conservative compared to the richer set of sub\-token matches thatWWexposes through teacher\-side re\-tokenization of the student’s decoded text\. Table[2](https://arxiv.org/html/2605.21699#S3.T2)confirms this empirically: H\-KL outperforms P\-KL by\+1\.68\+1\.68avg\. on the Phi\-4\-mini teacher where this precondition holds\.

To address this,H\-KLretains the hybrid structure but relaxes the definition of𝒞\\mathcal\{C\}using the projection matrixWW\. For each student tokenss, we select its top\-ranked teacher tokent∗t^\{\*\}:

t∗=arg⁡maxt′∈𝒱T⁡W​\[s,t′\],W​\[s,t∗\]\>0,t^\{\*\}=\\arg\\max\_\{t^\{\\prime\}\\in\\mathcal\{V\}\_\{T\}\}\\,W\[s,t^\{\\prime\}\],\\;\\;W\[s,t^\{\*\}\]\>0,\(5\)and extend the common𝒞\\mathcal\{C\}set with\(s,t∗\)\{\(s,t^\{\*\}\)\}\. This construction expands𝒞\\mathcal\{C\}beyond strict string matches by incorporating high\-confidence alignments induced byWW\. Exact matches are preserved since they receive the highest weight inWW, while additional near\-equivalent pairs are included when no exact correspondence exists\.

H\-KL then applies the hybrid loss formulation \(Eq\.[3](https://arxiv.org/html/2605.21699#S2.E3)\) over this expanded set𝒞\\mathcal\{C\}\. This improves coverage of aligned token pairs while retaining the benefits of direct KL supervision, yielding a\+0\.5\+0\.5average accuracy gain \(Table[1](https://arxiv.org/html/2605.21699#S3.T1)\)\. A pair like\(Hundreds,Hund\)\(\\texttt\{Hundreds\},\\texttt\{Hund\}\)that strict equality excludes is now admitted into𝒞\\mathcal\{C\}and contributes the same direct\-KL signal as a native exact match\.

### 2\.7Multi\-Teacher Distillation

GivenMMteachers indexed bym∈\{1,…,M\}m\\in\\\{1,\\ldots,M\\\}, each with its own projection matrixWmW\_\{m\}and choice of P\-KL or H\-KL, X\-Token naturally extends to the multi\-teacher distillation by aggregating per\-teacher losses:

ℒKD,multi=∑m=1Mαm​1\|𝒦m\|​∑k∈𝒦mℒ\*,m\(k\)\\mathcal\{L\}\_\{\\mathrm\{KD,multi\}\}=\\sum\_\{m=1\}^\{M\}\\alpha\_\{m\}\\,\\frac\{1\}\{\|\\mathcal\{K\}\_\{m\}\|\}\\sum\_\{k\\in\\mathcal\{K\}\_\{m\}\}\\mathcal\{L\}\_\{\\text\{\*\},m\}^\{\(k\)\}\(6\)whereℒ\*,m\(k\)∈\{ℒP\(k\),ℒH\(k\),ℒKL\(k\)\}\\mathcal\{L\}\_\{\\text\{\*\},m\}^\{\(k\)\}\\in\\\{\\mathcal\{L\}\_\{P\}^\{\(k\)\},\\mathcal\{L\}\_\{H\}^\{\(k\)\},\\mathcal\{L\}\_\{\\mathrm\{KL\}\}^\{\(k\)\}\\\}denotes the selected loss for teachermm— P\-KL or H\-KL for cross\-tokenizer teachers, and standard token\-level KL for same\-tokenizer teachers\. We consider several choices forαm\\alpha\_\{m\}, based on cross\-entropy, entropy, and maximum predicted probability, with the goal of assigning higher weight to more confident teachers\. In practice, however, we find that simple static weighting performs best \(Table[6](https://arxiv.org/html/2605.21699#S3.T6)\)\.

Beyond weighting, our results highlight that*teacher complementarity*plays a critical role: combinations of teachers with diverse strengths \(*e\.g\.*, math vs\. general knowledge\) consistently outperform more homogeneous pairings \(Table[1](https://arxiv.org/html/2605.21699#S3.T1)\)\. This suggests that effective multi\-teacher distillation benefits not only from how teachers are weighted, but also from which teachers are selected\.

### 2\.8Dynamic KD/CE Scaling

We combine the distillation lossℒKD\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\(single teacher\) orℒKD,multi\\mathcal\{L\}\_\{\\mathrm\{KD,multi\}\}\(multi\-teacher\) with the next\-token cross\-entropyℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\. As these terms can differ significantly in magnitude and vary throughout training, a fixed weighting leads to unstable optimization\. We instead rescale the KD term at each step to match the scale ofℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}:

ℒ=sg​\(ℒCEℒKD\)⋅ℒKD\+ℒCE,\\mathcal\{L\}\\;=\\;\\mathrm\{sg\}\\\!\\left\(\\frac\{\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\}\{\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\}\\right\)\\cdot\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\\,\+\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\},\(7\)wheresg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)denotes stop\-gradient\. This maintains a consistent balance between KD and CE throughout training\. In the multi\-teacher setting, the rescaling is applied to the aggregated KD loss, ensuring that the effective KD contribution remains stable as the number of teachers varies\. Results are detailed in Table[4](https://arxiv.org/html/2605.21699#S3.T4)\.

### 2\.9Selecting P\-KL vs H\-KL via Coverage Analysis

We select between P\-KL and H\-KL using a coverage\-based criterion across the student and teacher vocabularies\. Tokens are grouped into character classes \(*e\.g\.*, digits by length, alphabetic, punctuation, multi\-byte / non\-ASCII\), and we measure their retention in the common set𝒞\\mathcal\{C\}\.

For math tasks, multi\-digit numerals are critical: under Qwen3\-4B, all1,1001\{,\}100Llama two and three\-digit numerals fall into𝒰\\mathcal\{U\}, whereas under Phi\-4\-mini\-Instruct they remain in𝒞\\mathcal\{C\}\. In contrast, ASCII punctuation and single\-digit numerals are fully covered in both cases \(Table[8](https://arxiv.org/html/2605.21699#S8.T8)\)\. Accordingly, we useP\-KLwhen critical tokens fall outside𝒞\\mathcal\{C\}\(Qwen\) andH\-KLotherwise \(Phi\-4\-mini\)\.

#### X\-Token training step \(single teacher\)\.

Algorithm[1](https://arxiv.org/html/2605.21699#alg1)summarizes the per\-step computation\. The mode parameterℳ∈\{P\-KL,H\-KL\}\\mathcal\{M\}\\in\\\{\\textsf\{P\-KL\},\\textsf\{H\-KL\}\\\}is fixed per teacher\.

Algorithm 1X\-Token training step \(single teacher\)\.1:Student

fSf\_\{S\}\(trainable\); teacher

fTf\_\{T\}\(frozen\); top\-

44projection matrix

WW\(rule\-based init, jointly learned for P\-KL\); loss mode

ℳ∈\{P\-KL,H\-KL\}\\mathcal\{M\}\\\!\\in\\\!\\\{\\textsf\{P\-KL\},\\textsf\{H\-KL\}\\\}; input text

xx; temperature

τ\\tau; loss weights

λKL,λULD\\lambda\_\{\\mathrm\{KL\}\},\\lambda\_\{\\mathrm\{ULD\}\}\.

2:\# Preprocessing \(cached across epochs\)

3:

𝐬←𝒯S​\(x\),𝐭←𝒯T​\(x\)\\mathbf\{s\}\\leftarrow\\mathcal\{T\}\_\{S\}\(x\),\\quad\\mathbf\{t\}\\leftarrow\\mathcal\{T\}\_\{T\}\(x\)
4:

\{\(AkS,AkT\)\}k=1K←DPAlign​\(𝐬,𝐭\)\\\{\(A\_\{k\}^\{S\},A\_\{k\}^\{T\}\)\\\}\_\{k=1\}^\{K\}\\leftarrow\\texttt\{DPAlign\}\(\\mathbf\{s\},\\mathbf\{t\}\)
5:\# Forward passes

6:Run

fS​\(𝐬\)f\_\{S\}\(\\mathbf\{s\}\)with gradient and

fT​\(𝐭\)f\_\{T\}\(\\mathbf\{t\}\)without gradient\.

7:\# Per\-chunk KD loss

8:for

k=1,…,Kk=1,\\ldots,Kdo

9:Merge chunk distributions

p^S\(k\),p^T\(k\)\\hat\{p\}\_\{S\}^\{\(k\)\},\\hat\{p\}\_\{T\}^\{\(k\)\}via the inherited chain\-rule merge\.

10:if

ℳ=P\-KL\\mathcal\{M\}=\\textsf\{P\-KL\}then\(partition\-free direct projection KL\)

11:

p~S\(k\)←W⊤​p^S\(k\)\\tilde\{p\}\_\{S\}^\{\(k\)\}\\leftarrow W^\{\\\!\\top\}\\hat\{p\}\_\{S\}^\{\(k\)\}\.

12:

ℒ\(k\)←KL​\(p^T\(k\)∥p~S\(k\)\)\\mathcal\{L\}^\{\(k\)\}\\leftarrow\\mathrm\{KL\}\\bigl\(\\hat\{p\}\_\{T\}^\{\(k\)\}\\,\\\|\\,\\tilde\{p\}\_\{S\}^\{\(k\)\}\\bigr\)\.

13:else\(H\-KL: hybrid common\-KL\+\+ULD on relaxed𝒞\\mathcal\{C\}\)

14:

ℒ\(k\)←λKL​ℒcommon\(k\)\+λULD​ℒULD\(k\)\\mathcal\{L\}^\{\(k\)\}\\leftarrow\\lambda\_\{\\mathrm\{KL\}\}\\mathcal\{L\}\_\{\\mathrm\{common\}\}^\{\(k\)\}\+\\lambda\_\{\\mathrm\{ULD\}\}\\mathcal\{L\}\_\{\\mathrm\{ULD\}\}^\{\(k\)\}\.

15:endif

16:endfor

17:\# Loss aggregation and dynamic KD/CE scaling

18:

ℒKD←τ2⋅1K​∑k=1Kℒ\(k\)\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\\leftarrow\\tau^\{2\}\\cdot\\tfrac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathcal\{L\}^\{\(k\)\}\.

19:

ℒCE←\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\\leftarrownext\-token cross\-entropy of

fSf\_\{S\}on

𝐬\\mathbf\{s\}\.

20:

γ←sg​\(ℒCE/ℒKD\)\\gamma\\leftarrow\\mathrm\{sg\}\\\!\\bigl\(\\mathcal\{L\}\_\{\\mathrm\{CE\}\}/\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\\bigr\)\(stop\-gradient\)

21:

ℒ←γ⋅ℒKD\+ℒCE\\mathcal\{L\}\\leftarrow\\gamma\\cdot\\mathcal\{L\}\_\{\\mathrm\{KD\}\}\+\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\.

22:Update

fSf\_\{S\}via

∇fSℒ\\nabla\_\{f\_\{S\}\}\\mathcal\{L\}\.

## 3Experiments

#### Teachers and per\-teacher loss selection\.

We use three teachers: Llama\-3\.2\-3B \(same tokenizer as the student\) as a same\-family reference, and two cross\-tokenizer teachers, Qwen3\-4B and Phi\-4\-mini\-Instruct\.

#### Student and training setup\.

We useLlama\-3\.2\-1Bas the student and train on the Nemotron\-ClimbMix dataset\[diao2025nemotron\]for30,00030\{,\}000steps with a batch size of768768and context length40964096\. We use AdamW with initial learning rate5×10−55\\\!\\times\\\!10^\{\-5\},5%5\\%warmup followed by cosine decay to0, weight decay0\.10\.1, and gradient clipping at1\.01\.0\. Distillation and cross\-entropy losses are combined via Eq\. \([7](https://arxiv.org/html/2605.21699#S2.E7)\) with temperatureτ=1\.0\\tau\{=\}1\.0\. The projection matrixWWis initialized from tokenizer\-level string rules, truncated to the top\-44teacher tokens per student token, and jointly refined with the student under P\-KL \(LR=10−2=10^\{\-2\}, no gradient clipping\), while kept fixed under H\-KL\. For both GOLD and H\-KL, we useλKL=λULD=1\\lambda\_\{\\mathrm\{KL\}\}=\\lambda\_\{\\mathrm\{ULD\}\}=1\.

#### Evaluation\.

We evaluate 3\-shot accuracy on five benchmarks spanning knowledge, mathematical reasoning, and commonsense: MMLU\[hendrycks2020measuring\], GSM8k\[cobbe2021training\], MATH\-Hendrycks\[hendrycks2021measuring\], Winogrande\[sakaguchi2021winogrande\], and HellaSwag\[zellers2019hellaswag\]\. All numbers are reported on the official test splits using identical evaluation settings across methods\.

#### Baselines\.

We compare X\-Token against: \(1\)no distillation\(Llama\-3\.2\-1B base and continued pre\-training without a teacher\); \(2\)same\-tokenizer KDfrom Llama\-3\.2\-3B using standard token\-level KL \(same\-family ceiling\); andcross\-tokenizer baselineson Qwen\-4B and Phi\-mini:ULD\[boizard2024towards\], which matches rank\-sorted distributions, andGOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\], which combines common\-KL on string\-equality pairs with a ULD\-style term on the remainder\. All baselines are reimplemented under identical training settings, so differences isolate the distillation mechanism\. In this approach, X\-Token varies along two axes: \(a\)*loss choice*—P\-KL removes the partition entirely, used when critical tokens fall into the uncommon set; and \(b\)*matching criterion*—H\-KL relaxes matching via top\-1 mappings underWW, used when the partition is structurally sound\.

#### Computational Resources\.

Each reported experiment is feasible on a single NVIDIA H100 GPU, but we use 128 H100 GPUs in practice to speed up training and enable faster iteration\.

### 3\.1Main Results

Table 1:Main results on the Llama\-3\.2\-1B student \(3\-shot\)\.Llama\-1B: Llama\-3\.2\-1B;Llama\-3B: Llama\-3\.2\-3B;Qwen\-4B: Qwen3\-4B;Phi\-mini: Phi\-4\-mini\-Instruct\. Teacher rows \(italic\) report standalone performance and are excluded from best\-in\-column comparisons\. All methods share identical settings except for the distillation loss and teacher configuration\.Bolddenotes the best student result per column\.SettingMethodMMLUGSM8kMATHWGHSAvg\.No distillationLlama\-1B \(base\)32\.055\.695\.4861\.4865\.0833\.96Continued pre\-training40\.5010\.256\.9061\.6063\.9036\.63Teachers\(reference\)Llama\-3B55\.9724\.948\.8270\.0174\.9246\.93Phi\-mini68\.3282\.7119\.3074\.9073\.3663\.72Qwen\-4B72\.4384\.6127\.7672\.3075\.0066\.42Same tokenizerLlama\-3B→\\to1B43\.8312\.898\.1662\.7064\.4238\.40Cross tokenizer\(single teacher\)Qwen\-4B, ULD40\.3414\.564\.0461\.9662\.9336\.77Qwen\-4B, GOLD42\.562\.564\.5062\.9562\.5935\.03Qwen\-4B,X\-Token \(P\-KL\)44\.6715\.547\.9663\.4662\.6338\.85Phi\-mini, ULD41\.4317\.976\.2462\.5963\.3238\.31Phi\-mini, GOLD43\.5016\.507\.8062\.6062\.9238\.66Phi\-mini,X\-Token \(H\-KL\)43\.9319\.118\.3261\.8762\.6739\.18Multi\-teacherPhi\-mini\+\+Llama\-3B \(X\-Token\)46\.3220\.399\.0263\.363\.3840\.48Phi\-mini\+\+Qwen\-4B \(X\-Token\)43\.9814\.638\.1062\.7463\.0038\.49Phi\-mini\+\+Qwen\-4B\+\+Llama\-3B \(X\-Token\)45\.8619\.188\.5663\.6163\.5540\.15

Table[1](https://arxiv.org/html/2605.21699#S3.T1)reports all configurations under a fixed training budget\. Continued pre\-training of Llama\-1B without a teacher yields only modest gains over the frozen baseline \(33\.96→36\.6333\.96\\\!\\to\\\!36\.63avg\.\), and remains well below all distillation variants, indicating that improvements stem from distillation rather than additional compute\. Same\-tokenizer KD from Llama\-3B reaches38\.4038\.40avg\., providing a same\-family reference for cross\-tokenizer methods\.

#### P\-KL on Qwen\-4B \(uncommon\-token regime\)\.

On the Qwen pair, multi\-digit numerals fall into the uncommon subset, where GOLD’s gradients suppress them\. GOLD achieves35\.0335\.03avg\. \(2\.562\.56on GSM8k\), underperforming even continued pre\-training without a teacher \(36\.6336\.63avg\.\), indicating that its partition is harmful in this regime\.

P\-KL removes the partition and routes student mass throughWWover teacher multi\-token decompositions, improving to38\.8538\.85avg\. \(\+3\.82\+3\.82over GOLD;6\.07×6\.07\{\\times\}on GSM8k,2\.56→15\.542\.56\\\!\\to\\\!15\.54\)\. This also surpasses same\-tokenizer KD from Llama\-3\.2\-3B \(12\.8912\.89on GSM8k\), showing that cross\-tokenizer KD with P\-KL can exceed same\-family KD on math\.

Since X\-Token and GOLD share alignment and training setup, this gap isolates the loss formulation\. Notably, pure ULD already improves over GOLD \(36\.7736\.77vs\.35\.0335\.03\), indicating that the partition is the primary source of failure\. P\-KL further outperforms ULD by\+2\.08\+2\.08avg\. by adding identity\-aware projection\.

#### H\-KL on Phi\-mini \(sound\-partition regime\)\.

On the Phi\-mini pair, multi\-digit numerals remain in the common subset, so GOLD’s partition is structurally sound and achieves38\.6638\.66avg\. H\-KL relaxes strict string matching to top\-ranked teacher mappings underWW, improving coverage while retaining the hybrid loss\. This yields39\.1839\.18avg\. \(\+0\.52\+0\.52over GOLD\), isolating the benefit of relaxed matching\.

Pure ULD performs slightly worse \(38\.3138\.31,−0\.35\-0\.35vs\. GOLD\), consistent with the partition being well\-formed: when critical tokens lie in the common set, dropping the partition and using P\-KL sacrifices identity\-aligned signal and leads to degradations as shown in Table[2](https://arxiv.org/html/2605.21699#S3.T2)\.

#### Multi\-teacher distillation gives complementary gains\.

Combining Phi\-mini \(H\-KL\) with Llama\-3B \(same\-tokenizer\) under static weighting reaches40\.4840\.48avg\., exceeding the same\-family reference by\+2\.1\+2\.1and the best single\-teacher cross\-tokenizer run \(39\.1839\.18\) by\+1\.3\+1\.3, demonstrating strong complementarity: Phi\-mini contributes math/reasoning, while Llama\-3B contributes commonsense knowledge\.

In contrast, combining two cross\-tokenizer reasoning teachers \(Phi\-mini\+\+Qwen\-4B\) achieves38\.4938\.49, below the best single\-teacher result, suggesting overlapping capabilities and interference\. Adding Qwen\-4B as a third teacher yields40\.1540\.15avg\., similar overall but with trade\-offs: math/reasoning degrades \(MMLU46\.32→45\.8646\.32\\\!\\to\\\!45\.86, GSM8k20\.39→19\.1820\.39\\\!\\to\\\!19\.18, MATH9\.02→8\.569\.02\\\!\\to\\\!8\.56\) while commonsense improves slightly, again indicating redundancy or interference\.

### 3\.2Ablations and Design Checks

We ablate four design choices: the loss mode \(P\-KL vs\. H\-KL\), the projection matrixWW\(frozen vs\. learned\), the teacher weighting strategy, and the KD/CE scaling scheme\.

Table 2:Average accuracy across the five benchmarks for each loss mode on each teacher\. The two modes flip rankings between teachers \(bold: per\-teacher winner\)\.TeacherP\-KLH\-KLQwen\-4B38\.8535\.30Phi\-mini37\.5039\.18•Finding 1:Tokenizer\-dependent loss selection is crucial for effective cross\-tokenizer KD\.•Finding 2:P\-KL is preferred when critical tokens fall outside the common set\.•Finding 3:H\-KL performs better when token alignment is preserved, leveraging sharper identity\-aligned supervision\.•Finding 4:Complementary teachers drive multi\-teacher gains\.•Finding 5:Simple static weighting suffices for combining teacher signals\.

#### P\-KL vs\. H\-KL on each teacher\.

This ablation shows that neither loss mode dominates: each exhibits a sharp drop when applied to the wrong teacher\. We evaluate both P\-KL and H\-KL on each teacher with all else fixed \(Table[2](https://arxiv.org/html/2605.21699#S3.T2)\)\. P\-KL outperforms H\-KL by\+3\.55\+3\.55avg\. on Qwen3\-4B, while H\-KL outperforms P\-KL by\+1\.68\+1\.68on Phi\-4\-mini\. This reversal aligns with the mechanism: P\-KL partition\-free projection is preferred when critical tokens fall in the uncommon set, whereas H\-KL is favored when the partition is structurally sound\. These results validate the per\-teacher loss selection used in Table[1](https://arxiv.org/html/2605.21699#S3.T1)\.

#### Frozen vs\. learnable projection matrix\.

Table[3](https://arxiv.org/html/2605.21699#S3.T3)compares a frozenWWagainst jointly learningWWon the Qwen3\-4B \(P\-KL\) pair\. LearningWWyields a modest but consistent improvement \(38\.8538\.85vs\.38\.3738\.37avg\., winning5/65/6columns\), indicating that the rule\-based construction provides a strong initialization that can be further refined by the distillation loss with minimal overhead\.

Table 3:Frozen vs\. jointly learned projection matrixWWon Qwen3\-4B \(P\-KL\) with a Llama\-3\.2\-1B student \(3\-shot\)\. H\-KL is omitted sinceWWaffects it only via a discrete top\-1 selection and receives no gradient\.Boldindicates the best result per column\.Projection matrixMMLUGSM8kMATHWGHSAvg\.Frozen \(default\)43\.3615\.777\.9462\.5962\.1738\.37Jointly learned44\.6715\.547\.9663\.4662\.6338\.85

#### Dynamic KD/CE scaling\.

Table 4:Dynamic KD/CE scaling vs\. fixed\-weight combinations on the Llama\-3\.2\-1B student with the Qwen3\-4B teacher \(P\-KL\),3,0003\{,\}000training steps, 3\-shot\.Bold: best per column\.KD/CE combinationMMLUGSM8kMATHWGHSAvg\.Fixed \(λKL=1\.0,λCE=0\.1\\lambda\_\{\\mathrm\{KL\}\}\{=\}1\.0,\\lambda\_\{\\mathrm\{CE\}\}\{=\}0\.1\)40\.088\.495\.8862\.1962\.9735\.92Fixed \(λKL=0\.5,λCE=0\.5\\lambda\_\{\\mathrm\{KL\}\}\{=\}0\.5,\\lambda\_\{\\mathrm\{CE\}\}\{=\}0\.5\)40\.079\.486\.0062\.5963\.2236\.27Fixed \(λKL=0\.1,λCE=1\.0\\lambda\_\{\\mathrm\{KL\}\}\{=\}0\.1,\\lambda\_\{\\mathrm\{CE\}\}\{=\}1\.0\)39\.758\.575\.9863\.1463\.8136\.25Dynamic \(default\)40\.159\.706\.0463\.1462\.9436\.39

Dynamic scaling rebalances KD and CE at each step based on their relative magnitudes \(Eq\. \([7](https://arxiv.org/html/2605.21699#S2.E7)\)\)\. Table[4](https://arxiv.org/html/2605.21699#S3.T4)compares it to three fixed\-weight settings spanning KD\-heavy \(λKL=1\.0,λCE=0\.1\\lambda\_\{\\mathrm\{KL\}\}\{=\}1\.0,\\lambda\_\{\\mathrm\{CE\}\}\{=\}0\.1\), balanced \(0\.5/0\.50\.5/0\.5\), and CE\-heavy \(0\.1/1\.00\.1/1\.0\) regimes on the Qwen3\-4B \(P\-KL\) teacher\. All configurations are run for3,0003\{,\}000steps to keep the four\-way sweep tractable\.

Table 5:Multi\-teacher weighting on Phi\-4\-mini\+\+Llama\-3\.2\-3B with a Llama\-3\.2\-1B student,30,00030\{,\}000steps, 3\-shot\.Bold: best per column among complete rows\. Static weights given as\(αLlama,αPhi\)\(\\alpha\_\{\\mathrm\{Llama\}\},\\alpha\_\{\\mathrm\{Phi\}\}\)\.WeightingMMLUGSM8kMATHWGHSAvg\.Static\(0\.8,0\.2\)\(0\.8,0\.2\)44\.4814\.108\.6262\.5164\.0938\.76Static\(0\.5,0\.5\)\(0\.5,0\.5\)45\.9719\.568\.8263\.1463\.9840\.29Static\(0\.2,0\.8\)\(0\.2,0\.8\)46\.3220\.399\.0263\.3063\.3840\.48Adaptive \(CE\)45\.8418\.809\.0463\.5463\.8540\.21Adaptive \(entropy\)45\.6318\.358\.5462\.9063\.6539\.81Adaptive \(max\-prob\)45\.6119\.118\.8263\.3863\.8840\.16

Table 6:Multi\-teacher weighting on Phi\-4\-mini\+\+Qwen3\-4B with a Llama\-3\.2\-3B student,30,00030\{,\}000steps\. Static ratios given as\(αPhi,αQwen\)\(\\alpha\_\{\\mathrm\{Phi\}\},\\alpha\_\{\\mathrm\{Qwen\}\}\)\.WeightingAvg\.Static\(0\.5,0\.5\)\(0\.5,0\.5\)51\.48Adaptive \(max\-prob\)51\.30Adaptive \(CE\)51\.33Adaptive \(entropy\)51\.59Static\(0\.8,0\.2\)\(0\.8,0\.2\)52\.19

#### Multi\-Teacher weighting\.

We compare static and confidence\-adaptive softmax parameterizations of the per\-teacher weightαm\\alpha\_\{m\}\(Eq\. \([6](https://arxiv.org/html/2605.21699#S2.E6)\)\) across two setups\. Table[6](https://arxiv.org/html/2605.21699#S3.T6)\(1B student; Phi\-4\-mini\+\+Llama\-3\.2\-3B\) evaluates three static ratios and two adaptive variants, while Table[6](https://arxiv.org/html/2605.21699#S3.T6)\(3B student; Phi\-4\-mini\+\+Qwen3\-4B\) reports analogous results\.

Across both, Phi\-heavy static weighting performs best:\(0\.2,0\.8\)\(0\.2,0\.8\)reaches40\.4840\.48avg\. on the 1B run and\(0\.8,0\.2\)\(0\.8,0\.2\)reaches52\.1952\.19on the 3B run, both exceeding adaptive schemes \(CE:40\.21/51\.3340\.21/51\.33, entropy:39\.81/51\.5939\.81/51\.59, max\-prob:40\.16/51\.3040\.16/51\.30\)\. This supports our observation that adaptive weighting adds tuning complexity without consistent gains\. We therefore adopt static weighting in the main results\.

## 4Related Work

We organize prior cross\-tokenizer KD methods using the two\-family taxonomy:*logit\-distribution\-based*methods that operate on output distributions, and*representation\-based*methods that operate on embeddings or hidden states\. X\-Token belongs to the logit\-distribution family, alongside our primary baselines, GOLD and ULD\.

### 4\.1Logit\-distribution\-based methods

This family integrates as a drop\-in loss without modifying the student architecture or requiring additional forward passes\.ULD\[boizard2024towards\]sidesteps vocabulary alignment by rank\-sorting both distributions and minimizing anL1L\_\{1\}distance, discarding token identity\.GOLD\[patiño2025\_unlocking\_on\_policy\_distillation\_for\_any\_model\_family\]adds span alignment, chain\-rule chunk aggregation, and a hybrid loss that partitions tokens into a 1\-to\-1 string\-equality common set \(direct KL\) and an uncommon remainder \(ULD on the tail\); it is the current state of the art and our primary point of comparison\.ALM\[minixhofer2025universal\]aligns student and teacher at the byte level, aggregates chunk\-level log\-probabilities, and applies a Binary Cross Entropy/KL\-style loss\.MinED\[wan2024knowledge\]maps each student token to the closest teacher token under string edit distance, yielding a rule\-based 1\-to\-1 alignment baseline\.

Within this family, X\-Token introduces two complementary modes that addresses key limitations of current approaches:P\-KLremoves the partition and matches full distributions via a sparse projectionWW, whileH\-KLretains the hybrid form but relaxes matching using the top\-1 mapping underWW\.

### 4\.2Representation\-based methods

This family aligns teacher and student at the embedding or hidden\-state level, typically requiring auxiliary trainable components or architectural modifications\.DSKD\[zhang2024dual\]projects teacher hidden states into the student space via cross\-attention and distills on these representations\.ZETT\[minixhofer2024zero\]generates embeddings for a new vocabulary using a hypernetwork conditioned on token strings, enabling tokenizer transfer at the embedding level; in practice, the ALM pipeline\[minixhofer2025universal\]combines this with a logit\-level loss\.BLD\[singh2026cross\]converts teacher token distributions to byte\-level distributions and augments the student with auxiliary byte\-projection heads \(discarded at inference\), modifying the architecture to handle low\-overlap vocabularies\.

In contrast, X\-Token avoids architectural changes and auxiliary trainable components, operating entirely within the logit\-distribution regime\.

## Conclusions

Cross\-tokenizer knowledge distillation requires addressing both sequence and vocabulary mismatches arising from heterogeneous tokenization\. In this paper, we presentedX\-Token, a logit\-distribution\-based approach that enables effective distillation across mismatched tokenizers via a sparse projection matrixWW, initialized from tokenizer rules \(training\-free and optionally refined jointly with the student\), and two complementary loss formulations\.P\-KLaligns full distributions throughWW, recovering signal for uncommon but critical tokens, whileH\-KLretains the partition structure and improves matching via top\-ranked alignments underWW\. Together, these modes provide a unified approach that adapts to tokenizer mismatch regimes and enables*multi\-teacher distillation*across heterogeneous models\. Empirically, X\-Token consistently improves over state of the art, achieving gains of\+3\.8\+3\.8avg\. on Qwen3\-4B and\+0\.5\+0\.5on Phi\-4\-mini\-Instruct, and enabling complementary multi\-teacher gains \(up to\+1\.3\+1\.3over single teacher KD\)\. Overall, X\-Token demonstrates that careful alignment at both the sequence and vocabulary levels, combined with adaptive loss design, is key to unlocking the full potential of cross\-tokenizer knowledge distillation\.

#### Limitations and future work\.

We evaluate a limited set of cross\-tokenizer teacher pairs with a Llama\-3\.2\-1B student under continued pre\-training\. Extending to instruction\-tuned and preference\-aligned models, larger students, and low\-overlap tokenizer pairs \(*e\.g\.*, SentencePiece\[kudo2018sentencepiece\], BPE\[sennrich2016neural\], byte\-level\) remains for future work\. A promising direction for multi\-teacher distillation is to replace static teacher weights with domain\-conditioned routing \(*e\.g\.*, math, code, commonsense\), especially in instruction\-tuned settings where specialization signals are stronger\.

## 5Acknowledgments

We would like to thank our colleagues and leaders at NVIDIA for their valuable support and feedback\. We are especially grateful to Shizhe Diao for assistance with datasets, and to Marcin Chochowski, Sepehr Sameni, and Daniel Korzekwa for their insightful discussions and constructive feedback\.

## References

## 6Suppressive Gradients From The Common\-KL Term

GOLD’s common\-KL termℒcommon\\mathcal\{L\}\_\{\\mathrm\{common\}\}\(Eq\.[1](https://arxiv.org/html/2605.21699#S2.E1)\) is a sum over matched columns of full\-vocab softmaxes; the dependency onlog⁡Zfull\\log Z\_\{\\mathrm\{full\}\}insidelog⁡pS​\[i\]\\log p\_\{S\}\[i\]propagates gradient back to every uncommon student logit, even though those logits do not appear in the loss\.

###### Proposition 1\(Common\-KL induces a suppressive gradient on uncommon logits\)\.

Letz∈ℝ\|𝒱S\|z\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{S\}\|\}be the student logits in a chunk,pS=softmax​\(z\)p\_\{S\}=\\mathrm\{softmax\}\(z\), and letpTp\_\{T\}be the \(fixed\) teacher distribution\. Let𝒞T\\mathcal\{C\}\_\{T\}be the teacher\-side projection of the common subset and𝒰=𝒱S∖𝒞S\\mathcal\{U\}=\\mathcal\{V\}\_\{S\}\\setminus\\mathcal\{C\}\_\{S\}the uncommon set\. Then for every uncommon student logitj∈𝒰j\\in\\mathcal\{U\},

∂ℒcommon∂zj=pS​\[j\]⋅M𝒞​\(T\)≥0,whereM𝒞​\(T\):=∑t∈𝒞TpT​\[t\]∈\[0,1\]\.\\begin\{split\}\\frac\{\\partial\\mathcal\{L\}\_\{\\mathrm\{common\}\}\}\{\\partial z\_\{j\}\}&=p\_\{S\}\[j\]\\cdot M\_\{\\mathcal\{C\}\}\(T\)\\geq 0,\\\\ \\text\{where\}\\quad M\_\{\\mathcal\{C\}\}\(T\)&:=\\sum\_\{t\\in\\mathcal\{C\}\_\{T\}\}p\_\{T\}\[t\]\\in\[0,1\]\.\\end\{split\}\(8\)Because the gradient is non\-negative, gradient descent with stepη\>0\\eta\>0decreaseszjz\_\{j\}at every step:Δ​zj=−η​\(∂ℒcommon/∂zj\)=−η​pS​\[j\]​M𝒞​\(T\)≤0\\Delta z\_\{j\}=\-\\eta\\,\(\\partial\\mathcal\{L\}\_\{\\mathrm\{common\}\}/\\partial z\_\{j\}\)=\-\\eta\\,p\_\{S\}\[j\]\\,M\_\{\\mathcal\{C\}\}\(T\)\\leq 0\. Since the softmax is monotonically increasing in each logit, drivingzjz\_\{j\}downward shrinkspS​\[j\]p\_\{S\}\[j\]relative to all other student probabilities — the probability mass of every uncommon token is suppressed, even though no uncommon token appears inℒcommon\\mathcal\{L\}\_\{\\mathrm\{common\}\}and independent of the ground\-truth token at the position\.

#### Setup\.

We fix a chunk and letz∈ℝ\|𝒱S\|z\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{S\}\|\}be the student logits\. The full\-vocab softmax is

pS​\[s\]=exp⁡\(zs\)Zfull,Zfull=∑s′∈𝒱Sexp⁡\(zs′\),p\_\{S\}\[s\]\\;=\\;\\frac\{\\exp\(z\_\{s\}\)\}\{Z\_\{\\mathrm\{full\}\}\},\\;\\;Z\_\{\\mathrm\{full\}\}\\;=\\;\\sum\_\{s^\{\\prime\}\\in\\mathcal\{V\}\_\{S\}\}\\exp\(z\_\{s^\{\\prime\}\}\),\(9\)andpTp\_\{T\}is a fixed distribution over𝒱T\\mathcal\{V\}\_\{T\}that does not depend onzz\. The bijective common subset𝒞⊆𝒱S×𝒱T\\mathcal\{C\}\\subseteq\\mathcal\{V\}\_\{S\}\\times\\mathcal\{V\}\_\{T\}has projections𝒞S\\mathcal\{C\}\_\{S\}and𝒞T\\mathcal\{C\}\_\{T\}, and we write𝒰=𝒱S∖𝒞S\\mathcal\{U\}=\\mathcal\{V\}\_\{S\}\\setminus\\mathcal\{C\}\_\{S\}\. The common\-KL term \(Eq\.[1](https://arxiv.org/html/2605.21699#S2.E1)\) is

ℒcommon​\(z\)=∑\(s,t\)∈𝒞pT​\[t\]​\(log⁡pT​\[t\]−log⁡pS​\[s\]\)\.\\mathcal\{L\}\_\{\\mathrm\{common\}\}\(z\)\\;=\\;\\sum\_\{\(s,t\)\\in\\mathcal\{C\}\}p\_\{T\}\[t\]\\,\\bigl\(\\log p\_\{T\}\[t\]\-\\log p\_\{S\}\[s\]\\bigr\)\.\(10\)

#### Preliminary identities\.

Treating distinct logits as independent variables, for anys,j∈𝒱Ss,j\\in\\mathcal\{V\}\_\{S\},

∂zs∂zj=𝟏​\[s=j\],∂log⁡Zfull∂zj=exp⁡\(zj\)Zfull=pS​\[j\]\.\\frac\{\\partial z\_\{s\}\}\{\\partial z\_\{j\}\}=\\mathbf\{1\}\[s=j\],\\;\\;\\frac\{\\partial\\log Z\_\{\\mathrm\{full\}\}\}\{\\partial z\_\{j\}\}=\\frac\{\\exp\(z\_\{j\}\)\}\{Z\_\{\\mathrm\{full\}\}\}=p\_\{S\}\[j\]\.\(11\)
The first identity is immediate\. The second follows fromlog⁡Zfull=log​∑s′exp⁡\(zs′\)\\log Z\_\{\\mathrm\{full\}\}=\\log\\sum\_\{s^\{\\prime\}\}\\exp\(z\_\{s^\{\\prime\}\}\)by the chain rule\.

Combining withlog⁡pS​\[s\]=zs−log⁡Zfull\\log p\_\{S\}\[s\]=z\_\{s\}\-\\log Z\_\{\\mathrm\{full\}\},

∂log⁡pS​\[s\]∂zj=1​\[s=j\]−pS​\[j\]\.\\frac\{\\partial\\log p\_\{S\}\[s\]\}\{\\partial z\_\{j\}\}\\;=\\;\\mathbf\{1\}\[s=j\]\\;\-\\;p\_\{S\}\[j\]\.\(12\)

#### Proof of Proposition[1](https://arxiv.org/html/2605.21699#Thmproposition1)\.

Fixj∈𝒰j\\in\\mathcal\{U\}\. Since𝒞S\\mathcal\{C\}\_\{S\}and𝒰\\mathcal\{U\}are disjoint, everys∈𝒞Ss\\in\\mathcal\{C\}\_\{S\}satisfiess≠js\\neq j, so𝟏​\[s=j\]=0\\mathbf\{1\}\[s=j\]=0in Eq\. \([12](https://arxiv.org/html/2605.21699#S6.E12)\) and thus∂log⁡pS​\[s\]/∂zj=−pS​\[j\]\\partial\\log p\_\{S\}\[s\]/\\partial z\_\{j\}=\-p\_\{S\}\[j\]for everys∈𝒞Ss\\in\\mathcal\{C\}\_\{S\}\. The teacher termpT​\[t\]​log⁡pT​\[t\]p\_\{T\}\[t\]\\log p\_\{T\}\[t\]in Eq\. \([10](https://arxiv.org/html/2605.21699#S6.E10)\) has no dependence onzz\. Differentiating Eq\. \([10](https://arxiv.org/html/2605.21699#S6.E10)\) with respect tozjz\_\{j\}therefore yields

∂ℒcommon∂zj\\displaystyle\\frac\{\\partial\\mathcal\{L\}\_\{\\mathrm\{common\}\}\}\{\\partial z\_\{j\}\}=−∑\(s,t\)∈𝒞pT​\[t\]⋅∂log⁡pS​\[s\]∂zj\\displaystyle=\-\\sum\_\{\(s,t\)\\in\\mathcal\{C\}\}p\_\{T\}\[t\]\\cdot\\frac\{\\partial\\log p\_\{S\}\[s\]\}\{\\partial z\_\{j\}\}=−∑\(s,t\)∈𝒞pT​\[t\]⋅\(−pS​\[j\]\)\\displaystyle=\-\\sum\_\{\(s,t\)\\in\\mathcal\{C\}\}p\_\{T\}\[t\]\\cdot\\bigl\(\-p\_\{S\}\[j\]\\bigr\)=pS​\[j\]​∑t∈𝒞TpT​\[t\]=pS​\[j\]​M𝒞​\(T\)\\displaystyle=p\_\{S\}\[j\]\\sum\_\{t\\in\\mathcal\{C\}\_\{T\}\}p\_\{T\}\[t\]=p\_\{S\}\[j\]M\_\{\\mathcal\{C\}\}\(T\)\(13\)where the second\-to\-last equality uses the bijection between𝒞S\\mathcal\{C\}\_\{S\}and𝒞T\\mathcal\{C\}\_\{T\}\(eacht∈𝒞Tt\\in\\mathcal\{C\}\_\{T\}appears in exactly one pair\(s,t\)∈𝒞\(s,t\)\\in\\mathcal\{C\}\)\. SincepS​\[j\]≥0p\_\{S\}\[j\]\\geq 0andM𝒞​\(T\)∈\[0,1\]M\_\{\\mathcal\{C\}\}\(T\)\\in\[0,1\]\(the teacher is a probability distribution, so∑t∈𝒱TpT​\[t\]=1\\sum\_\{t\\in\\mathcal\{V\}\_\{T\}\}p\_\{T\}\[t\]=1and𝒞T⊆𝒱T\\mathcal\{C\}\_\{T\}\\subseteq\\mathcal\{V\}\_\{T\}\), the gradient is non\-negative and vanishes only when one of the two factors is zero\. Under gradient descent with stepη\>0\\eta\>0,Δ​zj=−η​pS​\[j\]​M𝒞​\(T\)≤0\\Delta z\_\{j\}=\-\\eta\\,p\_\{S\}\[j\]\\,M\_\{\\mathcal\{C\}\}\(T\)\\leq 0\. No quantity in the derivation depends on the ground\-truth token, establishing both claims of Proposition[1](https://arxiv.org/html/2605.21699#Thmproposition1): the gradient is non\-negative on every uncommon logit, and its dependence onpTp\_\{T\}alone makes it independent of the ground\-truth token at the position\.□\\square

## 7Algorithm Details

#### DP span alignment scoring and recurrence\.

For each training sequence we precompute a set of*aligned chunks*\{\(AkS,AkT\)\}k=1K\\\{\(A\_\{k\}^\{S\},A\_\{k\}^\{T\}\)\\\}\_\{k=1\}^\{K\}via a dynamic\-programming span alignment, where each pair of spans decodes to the same text substring; alignment is cached per sequence and adds no per\-step training overhead, and the same alignment is used for both X\-Token and the GOLD baseline so our comparison isolates loss\-level differences\. LetD​\(i,j\)D\(i,j\)denote the maximum score achievable over student prefix𝐬1:i\\mathbf\{s\}\_\{1:i\}and teacher prefix𝐭1:j\\mathbf\{t\}\_\{1:j\}; the recurrence is:

D​\(i,j\)=max⁡\{D​\(i−1,j−1\)\+match​\(si,tj\)\(diagonal, 1\-to\-1\)max2≤k≤L⁡D​\(i−1,j−k\)\+αcomb​k⋅𝟙​\[si≡𝐭j−k\+1:j\]\(1\-to\-​k​combination\)max2≤k≤L⁡D​\(i−k,j−1\)\+αcomb​k⋅𝟙​\[𝐬i−k\+1:i≡tj\]\(​k​\-to\-1 combination\)D​\(i−1,j\)\+αgap​\(gap in teacher\)D​\(i,j−1\)\+αgap​\(gap in student\)\\scriptsize D\(i,j\)=\\max\\\!\\begin\{cases\}D\(i\-1,j\-1\)\+\\mathrm\{match\}\(s\_\{i\},t\_\{j\}\)\\\\ \\hfill\\text\{\(diagonal, 1\-to\-1\)\}\\\\\[5\.0pt\] \\displaystyle\\max\_\{2\\leq k\\leq L\}D\(i\-1,j\-k\)\+\\alpha\_\{\\mathrm\{comb\}\}k\\cdot\\mathbb\{1\}\[s\_\{i\}\\equiv\\mathbf\{t\}\_\{j\-k\+1:j\}\]\\\\ \\hfill\\text\{\(1\-to\-\}k\\text\{ combination\)\}\\\\\[5\.0pt\] \\displaystyle\\max\_\{2\\leq k\\leq L\}D\(i\-k,j\-1\)\+\\alpha\_\{\\mathrm\{comb\}\}k\\cdot\\mathbb\{1\}\[\\mathbf\{s\}\_\{i\-k\+1:i\}\\equiv t\_\{j\}\]\\\\ \\hfill\\text\{\(\}k\\text\{\-to\-1 combination\)\}\\\\\[5\.0pt\] D\(i\-1,j\)\+\\alpha\_\{\\mathrm\{gap\}\}\\hfill\\text\{\(gap in teacher\)\}\\\\\[2\.0pt\] D\(i,j\-1\)\+\\alpha\_\{\\mathrm\{gap\}\}\\hfill\\text\{\(gap in student\)\}\\end\{cases\}\(14\)wherematch​\(si,tj\)=\+αexact\\mathrm\{match\}\(s\_\{i\},t\_\{j\}\)=\+\\alpha\_\{\\mathrm\{exact\}\}if the two \(canonicalized\) tokens agree and−αexact\-\\alpha\_\{\\mathrm\{exact\}\}otherwise, and “≡\\equiv” denotes canonicalized string equality between a single token and the concatenation of a span\. The boundary conditions areD​\(i,0\)=i⋅αgapD\(i,0\)=i\\cdot\\alpha\_\{\\mathrm\{gap\}\}andD​\(0,j\)=j⋅αgapD\(0,j\)=j\\cdot\\alpha\_\{\\mathrm\{gap\}\}\. We use:

αexact=3,αcomb=1\.5,αgap=−1\.5,\\alpha\_\{\\mathrm\{exact\}\}=3,\\quad\\alpha\_\{\\mathrm\{comb\}\}=1\.5,\\quad\\alpha\_\{\\mathrm\{gap\}\}=\-1\.5,\(15\)in all experiments\. A backtrace fromD​\(n,m\)D\(n,m\)recovers the set of aligned chunks; transitions selected as gaps produce token positions that are marked unaligned and excluded from the loss\.

#### Why soft scoring\.

A hard\-constraint DP \(align\-or\-fail\) has two failure modes on realistic data: \(i\) a local tokenization edge case \(a byte\-fallback token, an unusual whitespace glyph\) makes the entire sequence misalign or propagates error into neighbouring chunks; \(ii\) two locally\-plausible alignments tie, and an arbitrary tie\-breaker produces inconsistent alignments across training runs\. The scoring formulation resolves both: gaps cost\|αgap\|\|\\alpha\_\{\\mathrm\{gap\}\}\|, so the DP prefers to insert a single gap rather than distort a long stretch, and mismatched diagonals are dominated by gap sequences whenever two or more consecutive positions would otherwise mismatch\. The score parameters were chosen so that \(a\)αexact\>\|αgap\|\\alpha\_\{\\mathrm\{exact\}\}\>\|\\alpha\_\{\\mathrm\{gap\}\}\|to reward alignment over walking around it, and \(b\) akk\-token combination \(\+αcomb​k\+\\alpha\_\{\\mathrm\{comb\}\}k\) competes favourably withkkindividual 1\-to\-1 matches \(\+αexact​k\+\\alpha\_\{\\mathrm\{exact\}\}k\) when the exact span\-level match is available\. We did not tune these values per dataset\.

#### Failure mode of TRL surface\-substring alignment\.

An alternative to surface\-level DP, used in TRL’s111[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)Goldtrainer, pairs tokens by substring equality on incrementally\-decoded text: per\-side decoded buffers are extended one piece at a time and an alignment group is flushed whenever the two buffers compare equal as raw strings\. The algorithm is brittle in a specific way: any byte\-level disagreement between the two decoded streams that is not canceled by a later piece prevents future flushes, and the end\-of\-sequence force\-flush dumps everything from the divergence point onward into a single mis\-grouped bucket\. Table[7](https://arxiv.org/html/2605.21699#S7.T7)shows a routine setting where this occurs in cross\-tokenizer KD; DP recovers the alignment via a single gap move\.

Table 7:Failure mode of TRL surface\-substring alignment under default\-configuration BOS asymmetry\. The Llama\-3 tokenizer auto\-prepends<bos\>\(=<\|begin\_of\_text\|\>\) underadd\_bos\_token=True\(its config default\) while Qwen\-3 and Phi\-4\-mini\-Instruct default toFalse\. Same input string"Hello world\."on both sides; decoded streams differ on byte 0\. Blue cells are student tokens, orange cells are teacher tokens\.TRL alignment\(top block\) emits a singlesuper\-groupbundling all student and teacher tokens together\.DP alignment\(bottom block\) emits one alignment pair per row: the spurious<bos\>is marked as a one\-sidedgap, and the remaining tokens are diagonalmatches\.PairStudent tokensTeacher tokensInput\.<bos\>Helloworld\.Helloworld\.TRL alignment✗super\-group\#1\{\\bigl\\\{<bos\>Helloworld\.\}\\bigr\\\}\{\\bigl\\\{Helloworld\.\}\\bigr\\\}DP alignment✗gap\#1<bos\>\(no teacher token\)✓match\#2HelloHello✓match\#3worldworld✓match\#4\.\.

Why TRL fails\.TRL accumulates per\-side decoded buffers and only flushes when buffers compare equal as raw strings\. After the first piece is appended,sbuf=s\_\{\\mathrm\{buf\}\}\{=\}"<bos\>"\(="<\|begin\_of\_text\|\>", 16 chars\) vs\.tbuf=t\_\{\\mathrm\{buf\}\}\{=\}"Hello"\(5 chars\)\. Length\-driven extension keeps the two buffers character\-misaligned through every prefix, so the buffer\-equality flush never fires; the end\-of\-sequence force\-flush emits both sides as a singlesuper\-group\(Pair \#1 in the top block of Table[7](https://arxiv.org/html/2605.21699#S7.T7)\)\.

Why DP works\.DP’s recurrence has explicit gap moves at fixed cost\. It marks the spurious<bos\>as a one\-sidedgapof unit cost \(Pair \#1 in the bottom block\) and aligns the three content tokens diagonally as 1\-to\-1matches \(Pairs \#2–\#4\)\. The disagreement is localized to a singlegappair regardless of how long the sentence is\.

#### Projection matrix as a probability\-preserving operator\.

Because each row ofWWis non\-negative and sums to 1, left\-multiplication byW⊤W^\{\\\!\\top\}acts as a convex combination of rows and is probability\-preserving: if𝐩S\\mathbf\{p\}\_\{S\}is a probability vector, then𝐖⊤​𝐩S\\mathbf\{W\}^\{\\\!\\top\}\\mathbf\{p\}\_\{S\}is also a probability vector over𝒱T\\mathcal\{V\}\_\{T\}\. In particular,

∑t∈𝒱T\(W⊤​𝐩S\)​\[t\]\\displaystyle\\sum\_\{t\\in\\mathcal\{V\}\_\{T\}\}\\bigl\(W^\{\\\!\\top\}\\mathbf\{p\}\_\{S\}\\bigr\)\[t\]=∑t∑sW​\[s,t\]​pS​\[s\]\\displaystyle=\\sum\_\{t\}\\sum\_\{s\}W\[s,t\]\\,p\_\{S\}\[s\]=∑spS​\[s\]​∑tW​\[s,t\]⏟=1\\displaystyle=\\sum\_\{s\}p\_\{S\}\[s\]\\underbrace\{\\sum\_\{t\}W\[s,t\]\}\_\{=1\}=∑spS​\[s\]=1\.\\displaystyle=\\sum\_\{s\}p\_\{S\}\[s\]=1\.\(16\)This ensures that P\-KL produces a valid student distribution over the teacher vocabulary without additional normalization tricks\.

Algorithm 2Rule\-based projection matrix construction\.1:Student tokenizer

𝒯S\\mathcal\{T\}\_\{S\}with vocab

𝒱S\\mathcal\{V\}\_\{S\}; teacher tokenizer

𝒯T\\mathcal\{T\}\_\{T\}with vocab

𝒱T\\mathcal\{V\}\_\{T\}; max span length

L=4L\{=\}4; decay

\(β,γ\)=\(0\.9,0\.1\)\(\\beta,\\gamma\)\{=\}\(0\.9,0\.1\); final top\-

K=4K\{=\}4\.

2:Initialize

W←𝟎∈ℝ\|𝒱S\|×\|𝒱T\|W\\leftarrow\\mathbf\{0\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{S\}\|\\times\|\\mathcal\{V\}\_\{T\}\|\}\.

3:\# Pass 1: canonicalized exact match

4:foreach

\(s,t\)∈𝒱S×𝒱T\(s,t\)\\in\\mathcal\{V\}\_\{S\}\\times\\mathcal\{V\}\_\{T\}do

5:if

canon\(𝒯S\.decode\(s\)\)=canon\(𝒯T\.decode\(t\)\)\\mathrm\{canon\}\(\\mathcal\{T\}\_\{S\}\.\\texttt\{decode\}\(s\)\)=\\mathrm\{canon\}\(\\mathcal\{T\}\_\{T\}\.\\texttt\{decode\}\(t\)\)then

6:

W​\[s,t\]←1\.0W\[s,t\]\\leftarrow 1\.0
7:endif

8:endfor

9:\# Pass 2: multi\-token decoding rules

10:foreach

s∈𝒱Ss\\in\\mathcal\{V\}\_\{S\}where

W​\[s,⋅\]W\[s,\\cdot\]has no exact matchdo

11:

text←𝒯S\.decode​\(s\)\\text\{text\}\\leftarrow\\mathcal\{T\}\_\{S\}\.\\texttt\{decode\}\(s\)
12:

\(τ0,…,τℓ−1\)←𝒯T\.encode​\(text\)\(\\tau\_\{0\},\\ldots,\\tau\_\{\\ell\-1\}\)\\leftarrow\\mathcal\{T\}\_\{T\}\.\\texttt\{encode\}\(\\text\{text\}\)
13:if

ℓ≤L\\ell\\leq Lthen

14:for

i←0,…,ℓ−1i\\leftarrow 0,\\ldots,\\ell\-1do

15:

W​\[s,τi\]←β⋅γiW\[s,\\tau\_\{i\}\]\\leftarrow\\beta\\cdot\\gamma^\{i\}
16:endfor

17:endif

18:endfor

19:\# Finalize: sort, truncate, row\-normalize

20:foreach

s∈𝒱Ss\\in\\mathcal\{V\}\_\{S\}do

21:Retain only the

KKlargest entries of

W​\[s,⋅\]W\[s,\\cdot\]; zero the rest\.

22:

W​\[s,⋅\]←W​\[s,⋅\]/∑jW​\[s,j\]W\[s,\\cdot\]\\leftarrow W\[s,\\cdot\]/\\sum\_\{j\}W\[s,j\]
23:endfor

24:Sparse rule\-based projection matrix

WW\.

#### Confidence\-adaptive weight schedules\.

The confidence\-adaptive variants in computeαm\\alpha\_\{m\}from a per\-teacher confidence score derived from teachermm’s predictive distribution\. For a batch withBBsequences of lengthNN, letpTm\(b,n\)∈ℝ\|𝒱Tm\|p\_\{T\_\{m\}\}^\{\(b,n\)\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\_\{T\_\{m\}\}\|\}denote teachermm’s next\-token distribution at positionnnof batch elementbb, and lety\(b,n\)∈𝒱Tmy^\{\(b,n\)\}\\in\\mathcal\{V\}\_\{T\_\{m\}\}denote the ground\-truth next token\. The three per\-token confidence scores are:

CEm\(b,n\)\\displaystyle\\mathrm\{CE\}\_\{m\}^\{\(b,n\)\}=−log⁡pTm\(b,n\)​\[y\(b,n\)\],\\displaystyle=\-\\log p\_\{T\_\{m\}\}^\{\(b,n\)\}\[y^\{\(b,n\)\}\],\(17\)Hm\(b,n\)\\displaystyle\\mathrm\{H\}\_\{m\}^\{\(b,n\)\}=−∑v∈𝒱TmpTm\(b,n\)​\[v\]​log⁡pTm\(b,n\)​\[v\],\\displaystyle=\-\\\!\\sum\_\{v\\in\\mathcal\{V\}\_\{T\_\{m\}\}\}p\_\{T\_\{m\}\}^\{\(b,n\)\}\[v\]\\,\\log p\_\{T\_\{m\}\}^\{\(b,n\)\}\[v\],\(18\)maxpm\(b,n\)\\displaystyle\\mathrm\{maxp\}\_\{m\}^\{\(b,n\)\}=maxv∈𝒱Tm⁡pTm\(b,n\)​\[v\]\.\\displaystyle=\\max\_\{v\\in\\mathcal\{V\}\_\{T\_\{m\}\}\}\\,p\_\{T\_\{m\}\}^\{\(b,n\)\}\[v\]\.\(19\)LowerCE\\mathrm\{CE\}means the teacher better predicts the ground truth; lower entropy and higher max\-probability both indicate higher teacher confidence\. We aggregate to a per\-teacher scalar by averaging over the batch and sequence dimensions:

w¯m=1B​N​∑b=1B∑n=1Nwm\(b,n\),wm\(b,n\)∈\{−CEm\(b,n\),−Hm\(b,n\),maxpm\(b,n\)\}\.\\begin\{split\}\\bar\{w\}\_\{m\}&=\\frac\{1\}\{BN\}\\sum\_\{b=1\}^\{B\}\\sum\_\{n=1\}^\{N\}w\_\{m\}^\{\(b,n\)\},\\\\ w\_\{m\}^\{\(b,n\)\}&\\in\\bigl\\\{\-\\mathrm\{CE\}\_\{m\}^\{\(b,n\)\},\-\\mathrm\{H\}\_\{m\}^\{\(b,n\)\},\\mathrm\{maxp\}\_\{m\}^\{\(b,n\)\}\\bigr\\\}\.\\end\{split\}\(20\)whereCE\\mathrm\{CE\}andH\\mathrm\{H\}are negated so that higherwwcorresponds to higher teacher confidence in all three variants\. The per\-teacher mixing weights are then

αm=exp⁡\(w¯m\)∑m′=1Mexp⁡\(w¯m′\),\\alpha\_\{m\}\\;=\\;\\frac\{\\exp\(\\bar\{w\}\_\{m\}\)\}\{\\sum\_\{m^\{\\prime\}=1\}^\{M\}\\exp\(\\bar\{w\}\_\{m^\{\\prime\}\}\)\},\(21\)producing one\(α1,…,αM\)\(\\alpha\_\{1\},\\ldots,\\alpha\_\{M\}\)tuple per training step\. We also explored a per\-token variant computingαm\(b,n\)\\alpha\_\{m\}^\{\(b,n\)\}per position, but observed no improvement over the per\-batch formulation; the per\-batch form is the default reported in our experiments\.

## 8Projection Matrix Construction Details

#### Pseudocode for the two\-pass construction\.

Algorithm[2](https://arxiv.org/html/2605.21699#alg2)details the rule\-based construction of the sparse top\-44projection matrixWW\. Pass 1 enumerates string\-identical token pairs after canonicalization \(logically a double loop over𝒱S×𝒱T\\mathcal\{V\}\_\{S\}\\times\\mathcal\{V\}\_\{T\}as written; in practice implemented inO​\(\|𝒱S\|\+\|𝒱T\|\)O\(\|\\mathcal\{V\}\_\{S\}\|\+\|\\mathcal\{V\}\_\{T\}\|\)via a hashmap keyed on canonicalized decoded strings\)\. Pass 2 decodes each student token, re\-tokenizes under the teacher tokenizer, and adds exponentially\-weighted entries for each resulting teacher sub\-token\. After both passes, each row is row\-normalized and then truncated to its top\-KKentries; the truncation drops the smallest weights, so post\-truncation rows can sum to slightly less than11\. This is intentional: H\-KL only usesarg⁡maxt⁡W​\[s,t\]\\arg\\max\_\{t\}W\[s,t\], and P\-KL projects throughWWfollowed by re\-normalization over𝒱T\\mathcal\{V\}\_\{T\}, so neither mode requires exact row\-stochasticity of the truncatedWW\.

#### Multi\-token weight decay\.

When a student tokenssmaps to a multi\-token teacher sequence\(τ0,τ1,…,τℓ−1\)\(\\tau\_\{0\},\\tau\_\{1\},\\ldots,\\tau\_\{\\ell\-1\}\)via teacher\-side re\-tokenization in Pass 2, we assign weights via exponential decay and row\-normalize:

w~i=β⋅γi,W​\[s,τi\]=w~i∑j=0ℓ−1w~j,i=0,1,…,ℓ−1,\\tilde\{w\}\_\{i\}\\;=\\;\\beta\\cdot\\gamma^\{i\},\\;W\[s,\\tau\_\{i\}\]\\;=\\;\\frac\{\\tilde\{w\}\_\{i\}\}\{\\sum\_\{j=0\}^\{\\ell\-1\}\\tilde\{w\}\_\{j\}\},\\;i=0,1,\\ldots,\\ell\-1,\(22\)withβ=0\.9\\beta=0\.9andγ=0\.1\\gamma=0\.1in all our experiments\. Explicitly, a length\-2 span receives weights\(0\.909,0\.091\)\(0\.909,0\.091\), a length\-3 span receives\(0\.9009,0\.0901,0\.0090\)\(0\.9009,0\.0901,0\.0090\), and a length\-4 span receives\(0\.9000,0\.0900,0\.0090,0\.0009\)\(0\.9000,0\.0900,0\.0090,0\.0009\)after normalization\. Concentrating mass on the leading sub\-token reflects the observation that it typically carries the most informative probability mass for cross\-tokenizer distillation \(*e\.g\.*, “\_inter” in \[“\_inter”, “national”\] or “\_20” in \[“\_20”, “24”\]\), while trailing sub\-tokens’ probability mass is less relevant given the prefix\. We did not tune\(β,γ\)\(\\beta,\\gamma\); the default values above were used throughout\.

#### Canonicalization rules\.

The canonicalization functioncanon​\(⋅\)\\mathrm\{canon\}\(\\cdot\)maps the decoded string of a token to a normalized form so that functionally identical tokens compare equal across tokenizer families\. We apply the following rules, in order:

- •Space prefix unification:Ġ\(GPT\-2/Llama BPE\),\_\(SentencePiece\), and␣\(Unicode space\) all map to a single literal space character at the start of the token\.
- •Newline unification:Ċ, escaped\\n, and the literal newline all map to\\n\.
- •Byte\-fallback tokens: SentencePiece byte tokens of the form<0xHH\>\(for hex byteHH\) are replaced by the literal character with that byte value\.
- •Leading whitespace\+punctuation pairs: combinations likeĠ,,Ġ\.,Ġ:are normalized to the punctuation alone if the combined token has an ambiguous whitespace interpretation\.
- •Special tokens: BOS, EOS, PAD, and model\-specific chat\-template tokens \(<\|im\_start\|\>, etc\.\) are handled separately via an explicit special\-token mapping that pairs corresponding roles across tokenizer families \(when unambiguous\)\.

These rules are applied consistently at both projection\-matrix construction time and inside the DP alignment’s string\-equality check\. Canonicalization is idempotent and rule\-based, with no learned parameters involved\.

Table 8:Per\-category coverage check on our two cross\-tokenizer teacher pairs: fraction of Llama tokens in each category surviving the 1\-to\-1 bijective common set𝒞\\mathcal\{C\}\. The Qwen partition drops every multi\-digit Llama numeral into𝒰\\mathcal\{U\}; the Phi\-4\-mini partition keeps them all in𝒞\\mathcal\{C\}\.Llama categoryQwen commonPhi\-4 commonCategory size1\-digit numerals13/13 \(100%\)13/13 \(100%\)132\-digit numerals0/100 \(0%\)100/100 \(100%\)1003\-digit numerals0/1000 \(0%\)1000/1000 \(100%\)1000ASCII punctuation88/88 \(100%\)88/88 \(100%\)88