When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

# When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Source: [https://arxiv.org/html/2603.26556](https://arxiv.org/html/2603.26556)
Juan Gabriel Kostelecjuan\.gabriel\.kostelec@huawei\.com Huawei Zurich Research Center, SwitzerlandQinghai Guoguoqinghai@huawei\.com ACS Lab, Huawei Technologies

###### Abstract

Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs\. However, achieving high\-quality generation in distilled models requires careful joint design of both the student architecture and the distillation process\. Many prior distillation works evaluate downstream multiple\-choice benchmarks by ranking candidate answers with log\-likelihood rather than requiring autoregressive generation, which can obscure important differences in model quality\. For example, on overlapping benchmarks, we show that a 7B parameter distilled model that nearly matches its teacher to within 0\.2 pp under log\-likelihood scoring actually falls behind by 20\.8 pp when the model must generate answers autoregressively\.

We investigate this phenomenon with GenDistill, a multi\-stage pipeline we designed for distilling a pretrained Transformer into an efficient Hybrid Kimi Delta Attention \(Hybrid\-KDA\) student\. Using it as a controlled testbed on Qwen3\-0\.6B, we systematically ablate six design axes \(training objective, loss masking, training duration, dataset selection, parameter freezing, and architecture choice\) and evaluate every choice under both log\-likelihood and generation\-based protocols\. We find that log\-likelihood\-based evaluation consistently underestimates the gap between teacher and student, and can in some cases reverse the ranking of design choices, so conclusions drawn from perplexity\-only evaluation may be misleading\. Among the factors we study, dataset selection, completion\-only masking, and freezing attention layers during post\-training have the largest impact on generation quality\.

Our best distillation recipe, using a Hybrid\-KDA model as the student, retains 86–90% of teacher accuracy on knowledge benchmarks while reducing KV cache memory by up to 75% and improving time\-to\-first\-token by 2–4×\\timesat 128K\-token contexts\.

## 1Introduction

Transformer\-based language models dominate modern NLP due to their strong accuracy and scaling properties\. However, self\-attention incurs𝒪​\(n2\)\\mathcal\{O\}\(n^\{2\}\)time and memory complexity in the sequence lengthnn, and practical deployments additionally pay a linearly growing key\-value \(KV\) cache cost during autoregressive inference\.

In parallel, efficient sequence models with linear\-time recurrent\-style computation, including state space models \(SSMs\), have emerged as an attractive alternative\. These architectures can offer linear\-time processing and𝒪​\(1\)\\mathcal\{O\}\(1\)*per\-step*memory growth during generation \(i\.e\., no KV cache\), enabling faster decoding and a smaller memory footprint in long\-context settings\(Gu and Dao,[2023](https://arxiv.org/html/2603.26556#bib.bib26); Yanget al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib27);[2025c](https://arxiv.org/html/2603.26556#bib.bib28); Dao and Gu,[2024](https://arxiv.org/html/2603.26556#bib.bib25)\)\.

Because training large transformers from scratch is computationally expensive \(often on the order of trillions of tokens\), a growing line of work studies*cross\-architecture distillation*: transferring capabilities from a pretrained transformer teacher into a more efficient recurrent student\(Wanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib14); Bicket al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib15);[2024](https://arxiv.org/html/2603.26556#bib.bib3); Goldsteinet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib17)\)\.

Despite encouraging results, prior methods for distilling Transformers into SSMs both optimize for and evaluate on perplexity, without targeting generation quality at any stage of the pipeline\. While the broader NLP community has long recognized that perplexity is an imperfect proxy for generation quality, this gap takes on particular severity in cross\-architecture distillation\. The crux of the issue lies in how models are evaluated\.*Perplexity\-based*evaluation computes the log\-likelihood of each candidate answer conditioned on the prompt and selects the highest\-scoring option, testing whether the model*assigns high probability*to correct answers\.*Generation\-based*evaluation requires the model to produce answers autoregressively, after which outputs are typically parsed from an expected format or verified with task\-specific criteria \(e\.g\., exact\-match answers or executable code\), testing whether the model can*generate*correct answers\. In this work, we use LM Evaluation Harness\(Gaoet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib32)\)for perplexity\-based evaluation and EvalScope\(ModelScope Team,[2024](https://arxiv.org/html/2603.26556#bib.bib31)\)for generation\-based evaluation\. The distinction is consequential because a model can rank correct answers highly without being able to generate them\. To illustrate the severity of this discrepancy, we evaluate QRWKV6\-7B\-Instruct\(Linet al\.,[2025b](https://arxiv.org/html/2603.26556#bib.bib44)\), distilled from Qwen2\.5\-7B\-Instruct, under both evaluation protocols\. Figure[1](https://arxiv.org/html/2603.26556#S1.F1)contrasts the perplexity\-based results reported by the original paper with our generation\-based evaluation of the same model checkpoint \(full numbers in Table[5](https://arxiv.org/html/2603.26556#A1.T5), Appendix[A](https://arxiv.org/html/2603.26556#A1)\)\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x1.png)Figure 1:Evaluation protocol gap for QRWKV6\-7B\-Instruct \(distilled from Qwen2\.5\-7B\-Instruct\), with both panels evaluating the same checkpoint\. Perplexity\-based numbers \(left\) are taken from the original paper \(ARC\-E, ARC\-C, HS, WG\); generation\-based numbers \(right\) are our own and additionally include GSM8K, HEval, and IFEval\. Full values appear in Table[5](https://arxiv.org/html/2603.26556#A1.T5)\(Appendix[A](https://arxiv.org/html/2603.26556#A1)\)\.The gap between the two protocols is substantial\. As shown in Figure[1](https://arxiv.org/html/2603.26556#S1.F1), QRWKV6\-7B\-Instruct nearly matches its teacher under PPL\-based evaluation \(average gap of just 0\.2 pp\), yet suffers catastrophic drops in generation\-based performance \(average gap of 20\.8 pp\)\. The pattern is representative of the broader literature, where cross\-architecture distillation pipelines typically optimize on pretraining data and report their main downstream results via log\-likelihood\-based evaluation \(e\.g\., LM Evaluation Harness\)\(Wanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib14); Bicket al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib15);[2024](https://arxiv.org/html/2603.26556#bib.bib3); Goldsteinet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib17)\)\. This gap motivates a systematic investigation of what it takes to produce distilled models with genuine generation quality\. While works such as M1\(Wanget al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib59)\), and recently MiniCPM\-SALA\(Teamet al\.,[2026](https://arxiv.org/html/2603.26556#bib.bib60)\)have aimed to improve the reasoning ability of distilled linearized models, they rely on additional SFT and RL stages to improve the performance of the student model\. These stages themselves can be quite significant \(e\.g\. 1T tokens in the MiniCPM\-SALA\)\.

In contrast, we investigate the fundamental distillation choices themselves and how to maximize the generation quality of the distilled model without additional continual training\. In this work, we develop GenDistill, a multi\-stage distillation pipeline, making each of its design choices by empirically evaluating how that choice affects the student’s downstream instruction\-following and reasoning performance\. We then obtain the strongest student by pairing the pipeline with a Hybrid Kimi Delta Attention \(Hybrid\-KDA\) architecture\.

Our main contributions are:

- •We show that perplexity\-based evaluation is a systematically misleading proxy for generation quality in cross\-architecture distillation\. The divergence appears consistently across every design axis we vary, and holds at both 0\.6B and 1\.7B parameters\.
- •To establish this systematically, we develop GenDistill, a multi\-stage distillation pipeline whose design space we vary one axis at a time over a fixed teacher \(training objective, loss masking, training duration, dataset selection, parameter freezing, and architecture choice\)\. We will release our model code and distilled checkpoints upon publication\.
- •Based on these empirical results, we propose a concrete distillation recipe that maximizes the generation quality of the student model, combining knowledge distillation with completion\-only masking, teacher\-aligned instruction data, and frozen attention layers, which departs from the recipe a perplexity\-guided search would have selected\.

The remainder of the paper is organized as follows\. Sections[3\.1](https://arxiv.org/html/2603.26556#S3.SS1)and[3\.2](https://arxiv.org/html/2603.26556#S3.SS2)describe the Hybrid\-KDA architecture and the GenDistill distillation stages\. Section[4](https://arxiv.org/html/2603.26556#S4)defines the two evaluation protocols whose disagreement is our subject\. Section[5](https://arxiv.org/html/2603.26556#S5)then walks through each design choice and reports it under both protocols, where the recurring outcome is that perplexity and generation disagree, often reversing which choice looks better\. The recipe we recommend is what this design space yields once the choices are made under generation rather than perplexity\.

## 2Related Work

The computational demands of Transformer\-based language models have motivated substantial research on converting pre\-trained models to architectures with subquadratic complexity\. Chronologically, this line of work first focused on fine\-tuning\-based approaches that directly modify attention mechanisms and continue training with standard language modeling objectives, and later expanded to distillation\-based approaches that transfer knowledge from Transformer teachers to structurally different student models\.

Early works focused on replacing the softmax attention layer with expressive linear kernels, followed by fine\-tuning the resulting model for 1–40B tokens\(Chenet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib9); Kostelec and Guo,[2025](https://arxiv.org/html/2603.26556#bib.bib13); Zhanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib10); Mercatet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib12)\)\. Notably, LoLCats\(Zhanget al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib11)\)achieved conversion with as few as 40M tokens via a two\-stage approach: 20M tokens for attention alignment followed by 20M tokens to train LoRA adapters to recover model performance\. However, these methods still exhibited a performance gap, particularly on benchmarks such as MMLU\.

More recent works have leveraged multi\-stage distillation for model conversion and generally report substantially stronger student performance\. Table[1](https://arxiv.org/html/2603.26556#S2.T1)summarizes the pipelines most related to ours\. MOHAWK\(Bicket al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib3)\)introduced the bottom\-up three\-stage template of attention\-matrix orientation, hidden\-state alignment, and end\-to\-end knowledge distillation\. Similar pipelines were proposed byWanget al\.\([2024](https://arxiv.org/html/2603.26556#bib.bib14)\)\(focusing on hybrid models and adding SFT and DPO post\-training stages\),Bicket al\.\([2025a](https://arxiv.org/html/2603.26556#bib.bib15)\)\(scaling MOHAWK to larger models\), andYanget al\.\([2025b](https://arxiv.org/html/2603.26556#bib.bib16)\)\(distilling into a hybrid model with multi\-head latent attention and Mamba layers\)\. A concurrent work\(Goldsteinet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib17)\)proposed a highly efficient distillation process, using only 700M tokens while reporting state\-of\-the\-art results on downstream tasks, however, they focused on purely recurrent models and, like the other pipelines, did not contrast likelihood\-based with generation\-based evaluation\.

Table 1:Multi\-stage cross\-architecture distillation pipelines\. GenDistill follows the same bottom\-up align\-then\-distill structure as this line of work; what is specific to it is the selection of every recipe choice under generation\-based evaluation, a hybrid Kimi Delta Attention student, and a completion\-only\-masked instruction\-distillation stage on teacher\-aligned data\. Token counts are as reported by each work under its own teacher and data\.One fundamental limitation of SSMs and linear\-attention models is their fixed hidden\-state size, which can significantly degrade performance on in\-context retrieval tasks\. Consequently, many works have explored hybrid models to mitigate this gap\. Two directions are relevant here: intra\-layer hybrids, which replace only a subset of attention heads within a layer with SSM heads, and inter\-layer hybrids, which replace entire attention layers with SSM layers\. Although intra\-layer hybrids allow a finer\-grained choice of which attention heads to keep and which to convert to SSMs\(Zuoet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib7); Donget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib8)\), they can increase system complexity and inference overhead, complicating distributed parallelism optimizations\(Teamet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib6)\)\. For inter\-layer hybrids, the choice of which attention layers are kept intact is crucial, as some layers appear to be critical for retrieval performance\(Bicket al\.,[2025b](https://arxiv.org/html/2603.26556#bib.bib18)\)\.

Prior approaches have selected attention layers uniformly\(Lieberet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib63); Deet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib64); Bicket al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib3)\)\. SMART\(Yanget al\.,[2025b](https://arxiv.org/html/2603.26556#bib.bib16)\)studied the sensitivity of each layer \(measured by the reduction in teacher–student KL divergence\) when swapping a linear layer with an attention layer, and then heuristically selected layers to maximize total sensitivity\. PostNAS\(Guet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib20)\)used a complex search procedure, training a once\-for\-all supernet and using beam search to find the optimalKKlayers for specific downstream tasks\. A concurrent approach proposed greedily adding attention layers back to a distilled fully linear model by measuring the KL divergence of the output logits between the resulting hybrid model and the original teacher model after distilling the teacher into the hybrid model\(Liet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib19)\)\.

## 3The GenDistill pipeline

Producing a high\-quality distilled model requires jointly designing the student architecture and the distillation process\. This section describes both components: the Hybrid\-KDA student architecture \(Section[3\.1](https://arxiv.org/html/2603.26556#S3.SS1)\), including a beam\-search method for selecting which layers retain softmax attention \(Section[3\.1\.1](https://arxiv.org/html/2603.26556#S3.SS1.SSS1)\), and the multi\-stage distillation process that progressively aligns the student with the teacher \(Section[3\.2](https://arxiv.org/html/2603.26556#S3.SS2)\)\.

### 3\.1Student Model Architecture

To preserve the foundational knowledge acquired during pretraining, our student model inherits the majority of its architecture directly from the pretrained teacher model\. Specifically, the vocabulary, embedding layers, layer normalizations, and multi\-layer perceptrons \(MLPs\) are initialized from the teacher, while select attention layers are replaced with more efficient sequence mixers to improve inference efficiency\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x2.png)Figure 2:Architecture overview\.Left:The two block types in the hybrid transformer: an Attention block \(7 of 28 layers, frozen from the teacher\) and a KDA block \(21 of 28 layers\)\. MLPs and layer norms are initialized from the teacher and trained only during end\-to\-end distillation \(Stage 3\)\.Right:Internal structure of the KDA layer\. Fill colors indicate the earliest distillation stage in which each component begins training; once introduced, components continue training in all subsequent stages\. Green dashed borders denote initialization from teacher weights; dark\-red dotted borders denote identity initialization; the absence of a custom border denotes random initialization\.The choice of sequence mixers used in the student model is crucial for the success of the distillation pipeline, both in terms of final student model performance, as well as how efficiently we can distill the transformer into the new architecture\. Recent work has proposed several linear sequence mixers that maintain efficiency while improving model expressivity \(e\.g\. Gated DeltaNet\(Yanget al\.,[2025c](https://arxiv.org/html/2603.26556#bib.bib28)\), Kimi\(Teamet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib6)\), RWKV7\(Penget al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib45)\)\)\. In this work, we focus on two linear architectures: Kimi Delta Attention \(KDA\) as our primary student mixer, and Mamba2\(Dao and Gu,[2024](https://arxiv.org/html/2603.26556#bib.bib25)\)as a baseline for comparison\. We select Mamba2 because it is one of the most widely adopted SSMs and has been used as the student architecture in several prior distillation works \(Phi\-Mamba\(Wanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib14)\), Llamba\(Bicket al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib15)\)\)\. We select KDA because of its strong reported performance\. KDA refines Gated DeltaNet with a per\-channel rather than scalar forget gate, and we give the full recurrence in Appendix[B](https://arxiv.org/html/2603.26556#A2)\.

Additionally, there are several smaller architectural choices that have also proven important for a good and efficient distillation process\. We enumerate them here, while an ablation is available in Appendix[F](https://arxiv.org/html/2603.26556#A6)\. Similar toBicket al\.\([2024](https://arxiv.org/html/2603.26556#bib.bib3)\), we observe that removing the output norm in the SSM improves the distillation process\. We also use identity initialization of the short convolutions and the output gating in the Kimi layer, however, contrary to the findings inTeamet al\.\([2025](https://arxiv.org/html/2603.26556#bib.bib6)\), we find that using SiLU gating is better than using Sigmoid gates in the distillation setting\. Since we are distilling from the Qwen3 model family\(Yanget al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib4)\), which uses QK norms in the attention layer, we also keep normalization of QK values in the Kimi layer111We use L2 normalization without learnable scales, following the Kimi Delta Attention layer implementation inYang and Zhang \([2024](https://arxiv.org/html/2603.26556#bib.bib5)\)\.\.

#### 3\.1\.1Attention layer selection

In hybrid architectures, the selection of which layers retain softmax attention versus those replaced with linear alternatives significantly impacts model performance\. While certain attention heads appear universally important for in\-context retrieval\(Bicket al\.,[2025b](https://arxiv.org/html/2603.26556#bib.bib18)\), our initial experiments revealed that optimal layer selection is more task\-dependent\.

Greedy layer selection may yield suboptimal configurations, as it fails to account for interactions between layers\. To address this limitation, we employ beam search to explore layer combinations more thoroughly, similar to the approach in PostNAS\(Guet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib20)\)\. However, unlike PostNAS, we avoid training an extensive supernet by leveraging only the pretrained teacher model and a fully linear student model \(comprising Kimi layers\) obtained through our distillation pipeline\.

Our search procedure,*Beam Search \(Add\)*, initializes from the fully linear student and iteratively restores attention layers from the teacher, retaining at each step the configurations that achieve the lowest perplexity\. To account for task dependence, we evaluate perplexity on three complementary datasets: \(i\) a synthetic associative recall \(AR\) task that probes in\-context retrieval, \(ii\) C\-EVAL for Chinese language understanding, and \(iii\) FineWeb\-EDU for general language modeling, and select layers based on the average across all three\.

Training a student per candidate would be the ideal ranking signal but is prohibitively expensive, so we instead rank by perplexity as a cheap comparative proxy\. This within\-pipeline use scores an undertrained scaffold to decide only which layer positions matter, and is distinct from the final\-evaluation use of perplexity that this paper argues against\. Appendix[E](https://arxiv.org/html/2603.26556#A5)expands on this distinction, positions our proxy against the compute\-heavy PostNAS and GA\-S2\(Liet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib19)\)alternatives, reports the resulting layer selections, and shows that the cheap signal does not cost selection quality on Qwen3\-0\.6B\.

### 3\.2Distillation Stages

GenDistill uses a multi\-stage distillation process designed to progressively align the hybrid student model with the Transformer teacher\. Direct end\-to\-end distillation can be inefficient under architectural mismatch, so we follow the bottom\-up multi\-stage blueprint of MOHAWK\(Bicket al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib3)\), with Stage 1 adapted to KDA by aligning hidden states rather than attention logits \(KDA cannot efficiently materialize the fullN×NN\\times Nattention matrix\)\. We then perform end\-to\-end KD in Stage 3, separating pretraining\-data distillation \(Stage 3a\) from instruction\-tuning distillation \(Stage 3b\)\. We report generation\-based evaluation alongside perplexity to capture quality differences that are not visible under likelihood\-only protocols\. Overall, using KDA as the student mixer allows us to operate at a smaller total token budget \(∼\{\\sim\}1B\) than prior multi\-stage distillation pipelines \(MOHAWK:∼\{\\sim\}3B; Llamba:∼\{\\sim\}8B\(Bicket al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib15)\)\)\.

#### 3\.2\.1Stage 1: Initialization and Attention Matrix Alignment

The student model inherits the embedding, normalization, and MLP layers directly from the teacher\. For the sequence mixers, we replace specific attention layers with Kimi Delta Attention \(KDA\) layers\. We initialize the Value \(VV\) and Output \(OO\) projections of the KDA layers using the corresponding weights from the teacher, exploiting the architectural similarity, see Figure[2](https://arxiv.org/html/2603.26556#S3.F2)\. However, we find that initializing the Query \(QQ\) and Key \(KK\) projections from the teacher does not necessarily improve the training, and are thus initialized randomly\.

While MOHAWK\(Bicket al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib3)\)aligns the full attention logits to approximate the teacher’s attention patterns, KDA cannot efficiently materialize the fullN×NN\\times Nattention matrix\. Instead, we minimize the L2 distance between the hidden states of the attention layers of the teacher and student\. Only theQQandKKprojections are trained\. We use a filtered subset of FineWeb\-EDU and FineWeb\-Edu\-Chinese\-2\.1 \(score≥4\\geq 4\)222Datasets:[Fineweb\-EDU \(score≥4\\geq 4\)](https://huggingface.co/datasets/aynetdia/fineweb-edu-score-4-dedup)and[Fineweb\-Edu\-Chinese\-2\.1 \(filtered subset\)](https://huggingface.co/datasets/Mxode/Fineweb-Edu-Chinese-V2.1-merged-score4_5)\., training for 30M tokens with a sequence length of 2048\. We employ a warmup\-stable\-decay learning rate schedule \(max LR5​e−45e\-4, 10% warmup\)\. Samples are packed into fixed sequence lengths with cross\-document masking\(Zuoet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib7)\)\.

#### 3\.2\.2Stage 2: Block\-wise Hidden State Alignment

The second stage aligns the feature representations of the student and teacher blocks independently\. We minimize the L2 distance between the outputs of the student’s mixer block and the teacher’s attention block\. In this stage, all weights within the sequence mixer are trainable, while LayerNorms remain frozen as they are initialized from the teacher\. We train for another 30M tokens using the same dataset and schedule as Stage 1, but with a reduced learning rate \(1​e−41e\-4\)\.

Crucially, we find that this two\-step alignment \(Stage 1 followed by Stage 2\) is superior to training the mixer all at once or initializing all weights from the teacher\. Skipping Stage 1 results in significant perturbations to the initializedVVandOOweights, destabilizing the training\. By the end of this stage, after only 60M tokens and without any end\-to\-end training, the student model achieves a validation perplexity of∼\{\\sim\}25, approaching the teacher’s∼\{\\sim\}15–20\. Other initialization stages either reach much higher perplexity \(MOHAWK\(Bicket al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib3)\):∼\{\\sim\}1000 after 240M tokens\) or run far longer \(Llamba\(Bicket al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib15)\): 3B tokens\)\.333These perplexities use different teachers and evaluation datasets and are not directly comparable to ours, though all are measured on large web\-scale pretraining corpora\.

#### 3\.2\.3Stage 3: End\-to\-End Knowledge Distillation

Block\-wise alignment treats layers independently, leading to error accumulation when layers are composed during inference\. To address this and recover coherent text generation capabilities, we perform end\-to\-end knowledge distillation \(KD\)\. We minimize the forward KL divergence between the teacher and student output logits\. We freeze the attention layers and input embeddings, training all other parameters\.

This stage is divided into two phases\. First, we distill on the pretraining dataset to couple the layers and smooth internal representations \(Stage 3a; we select the token budget in Section[5\.2](https://arxiv.org/html/2603.26556#S5.SS2)\)\. Second, to adapt the model for instruction following, we switch to the AM\-Qwen3\-Distilled dataset\(Tianet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib21)\)444[https://huggingface\.co/datasets/a\-m\-team/AM\-Qwen3\-Distilled](https://huggingface.co/datasets/a-m-team/AM-Qwen3-Distilled)\(Stage 3b\)\. We increase the sequence length to 4096 and use a learning rate of2\.5​e−52\.5e\-5\. The Stage 3b design choices, in particular the training objective and the loss\-masking scope, are exactly where perplexity and generation\-based evaluation diverge, so we select each one under generation and report it under both protocols in Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\. We separate these two phases so that the student first recovers general language modeling over a broad distribution before acquiring instruction\-following and reasoning from the teacher\.

## 4Evaluation Setup

Having described the Hybrid\-KDA architecture and GenDistill pipeline \(Section[3](https://arxiv.org/html/2603.26556#S3)\), we now turn to evaluation\. A central design choice in our evaluation is the use of*two complementary protocols*that, as we show in Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1), can tell contradictory stories about model quality:

- •Generation\-based evaluation: The model generates answers autoregressively\. Responses are scored against reference answers\. This protocol tests whether the model can*produce*correct answers, i\.e\. the practical use case for deployed models\.
- •Log\-likelihood ranking on multiple\-choice tasks: The model scores each candidate answer by its log\-likelihood conditioned on the prompt, and the highest\-scoring candidate is selected\. This protocol tests whether the model assigns high probability to correct answers, which can diverge substantially from generation quality\.

In our implementation, we use EvalScope\(ModelScope Team,[2024](https://arxiv.org/html/2603.26556#bib.bib31)\)for generation\-based evaluation and LM Evaluation Harness\(Gaoet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib32)\)primarily for the log\-likelihood\-based multiple\-choice evaluations\.

##### Benchmarks\.

We evaluate on short\-context benchmarks across five categories:language modeling\(LAMBADA\(Papernoet al\.,[2016](https://arxiv.org/html/2603.26556#bib.bib39)\)\);common sense\(HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2603.26556#bib.bib36)\), WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2603.26556#bib.bib37)\), PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2603.26556#bib.bib38)\)\);knowledge\(MMLU\-Redux555We use MMLU\-Redux\(Gemaet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib33)\)for generation\-based evaluation \(EvalScope\), as it corrects a number of erroneous questions identified in the original MMLU\. For perplexity\-based evaluation \(LM Evaluation Harness\), we use the standard MMLU, since MMLU\-Redux is only available in a generative format in that framework\. The two benchmarks are tightly correlated in terms of performance\.\(Gemaet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib33)\), C\-Eval\(Huanget al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib34)\), CMMLU\(Liet al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib41)\)\);reasoning\(ARC\(Clarket al\.,[2018](https://arxiv.org/html/2603.26556#bib.bib35)\), BBH\(Suzgunet al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib42)\), GSM8k\(Cobbeet al\.,[2021](https://arxiv.org/html/2603.26556#bib.bib46)\), HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2603.26556#bib.bib47)\)\); andinstruction following\(IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib40)\)\)\. We evaluate tasks in both frameworks where available to analyze evaluation discrepancies666For BBH, HumanEval, GSM8K the performance on the lm\-evaluation\-harness was trivially low, around 0, which is likely an issue with the task setup, so we do not report these tasks in the lm\-evaluation harness framework\.\. We additionally evaluate long\-context quality using LongBench\(Baiet al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib1)\)and the NIAH\-single subtasks from RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib2)\)

## 5Results

We organize the results around the central finding\. We first show that log\-likelihood\-based and generation\-based evaluation disagree on the design choices that strongly affect the student, the training objective, the loss\-masking scope, and attention freezing, and that this disagreement persists at 1\.7B parameters \(Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\)\. Treating generation as the reliable criterion, we then fix the remaining recipe choices, the post\-training dataset, the pretraining\-distillation budget, and the training duration \(Section[5\.2](https://arxiv.org/html/2603.26556#S5.SS2)\)\. Finally, we compare the architectures this recipe produces, confirm that the selected architecture also scales to 1\.7B, and quantify their efficiency \(Section[5\.3](https://arxiv.org/html/2603.26556#S5.SS3)\)\.

### 5\.1Where perplexity and generation disagree

Three design choices strongly shape the distilled student\. They are the training objective, the loss\-masking scope, and whether the retained attention layers are frozen during instruction tuning\. We evaluate each under both protocols, log\-likelihood\-based \(LM Evaluation Harness\) and generation\-based \(EvalScope\)\. On all three, the two protocols disagree, and in places reverse the ranking of the same pair of models, with perplexity consistently understating differences that generation makes plain\. We select each choice under generation\.

Figure[3](https://arxiv.org/html/2603.26556#S5.F3)profiles the four \(objective, masking\) configurations and the teacher under both protocols\. Under perplexity the four students sit almost on top of one another and on top of the teacher\. Under generation they spread apart and fall well below the teacher, and they separate further on the reasoning, code, and instruction\-following tasks that perplexity cannot score at all \(full per\-benchmark numbers in Table[17](https://arxiv.org/html/2603.26556#A11.T17), Appendix[K](https://arxiv.org/html/2603.26556#A11)\)\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x3.png)

Figure 3:Stage 3b ablation at the 250M\-token budget: the four student configurations \(SFT and KD, each with full\-sequence P\+C or completion\-only C masking\) and the Qwen3\-0\.6B teacher, under both evaluation protocols \(accuracy in %\)\.Left:spider profiles\. The perplexity panel covers only the six benchmarks that log\-likelihood scoring can produce, where the four students sit almost on top of one another and on the teacher, while the generation panel adds the reasoning, code, and instruction\-following tasks that have no perplexity counterpart and spreads the configurations apart\.Right:average accuracy over the six benchmarks common to both protocols \(WG, HS, ARC\-E, ARC\-C, MMLU, CMMLU\), with the gap to the teacher shown inred\. Perplexity places every student within 3\.8pp of the teacher, whereas generation opens gaps of 4\.6 to 14\.7pp and re\-ranks them: SFT looks competitive under perplexity but drops furthest under generation\. Full per\-benchmark numbers, including ranking reversals on HellaSwag, ARC\-C, and MMLU, are in Table[17](https://arxiv.org/html/2603.26556#A11.T17), Appendix[K](https://arxiv.org/html/2603.26556#A11)\.##### KD vs\. SFT\.

The training objective strongly affects generation quality: KD is consistently performing better than SFT across generation benchmarks \(Table[17](https://arxiv.org/html/2603.26556#A11.T17)\), with its advantages most visible on reasoning, code generation, and instruction following\. By contrast, SFT often appears competitive under perplexity\-based evaluation, highlighting that next\-token likelihood on post\-training data is not a reliable proxy for answer generation quality\. On HellaSwag the disagreement is an outright inversion, with SFT attaining lower perplexity than KD but markedly lower generation accuracy\. Appendix[N](https://arxiv.org/html/2603.26556#A14)characterises the qualitative shape of this disagreement on the completion\-only 500M\-token pair, where the SFT generations exhibit multiple pathologies that are essentially absent from KD: template wrapping, multiple\-choice position collapse, wordier\-but\-less\-accurate math chains, and recency mimicry of in\-context demonstrations\.

##### Loss masking\.

Which tokens contribute to the training loss matters as much as the objective itself\. When the loss is computed over the full sequence, SFT can match or exceed KD on some benchmarks, especially those that emphasize language modeling \(e\.g\. Lambada\)\. By contrast, computing the loss only on completion tokens consistently improves generation quality for both objectives \(Table[17](https://arxiv.org/html/2603.26556#A11.T17)\), with KD benefiting more from this choice\. The trade\-off is that language modeling ability improves more slowly, since the loss is applied to fewer tokens\.

##### Freezing attention\.

Table 2:Effect of freezing attention layers during Stage 3b\. Generation\-based evaluation \(left\) reveals large improvements from freezing \(Δ¯\\overline\{\\Delta\}=5\.2pp\), while perplexity\-based evaluation \(right\) shows negligible differences \(Δ¯\\overline\{\\Delta\}=0\.5pp\)\. Best result per column inbold\.A natural question is how to preserve the quality of the retained attention layers during instruction tuning\. Table[2](https://arxiv.org/html/2603.26556#S5.T2)compares frozen versus unfrozen attention layers during Stage 3b under both evaluation protocols\. Freezing attention weights significantly improves generation\-based performance across every benchmark\. Unfrozen attention layers allow the instruction\-tuning loss to overwrite pretrained representations, causing catastrophic forgetting of the reasoning capabilities acquired during pretraining\. Critically, perplexity\-based evaluation almost entirely masks this effect\. The same frozen/unfrozen comparison that produces 5–8pp generation improvements yields less than 1pp differences on perplexity metrics \(Δ\\Delta\-row, Table[2](https://arxiv.org/html/2603.26556#S5.T2), right\)\. We further investigate whether MLP parameters exhibit the same sensitivity in Appendix[I](https://arxiv.org/html/2603.26556#A9)\.

##### Persistence at 1\.7B\.

We verify the three ablations at the 1\.7B scale, distilling from a Qwen3\-1\.7B teacher into the same Hybrid\-KDA architecture\. The protocol disagreement reproduces: KD again leads SFT on generation while SFT attains the lower perplexity, and freezing again improves generation while perplexity is nearly unchanged\. Completion\-only masking is the one axis that does not transfer cleanly\. It keeps its advantage on instruction following, code, and reasoning, but loses the commonsense multiple\-choice benchmarks to full\-sequence masking, where the completion\-only student produces near\-chance answers consistent with an extraction failure rather than a loss of ability\. Full results are in Appendix[M](https://arxiv.org/html/2603.26556#A13)\.

### 5\.2Distillation recipe

Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)fixes three choices, the training objective \(KD\), the loss\-masking scope \(completion\-only\), and freezing the retained attention layers during Stage 3b\. Two choices remain, the post\-training dataset and the token budget of each distillation stage, and we select them under generation as well\.

For the dataset choice, we compare six instruction corpora, AM\-Qwen3\-Distilled\(Tianet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib21)\), SFTv3\(Wanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib14)\), Smoltalk2\(Bakouchet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib56)\), OpenHermes 2\.5\(Teknium,[2023](https://arxiv.org/html/2603.26556#bib.bib55)\), Nemotron Post Training Dataset v1\(Nathawaniet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib58)\), and Chinese\-Instruct\(Zhang,[2025](https://arxiv.org/html/2603.26556#bib.bib57)\), each under the same configuration \(Hybrid\-KDA student, KD, 500M\-token budget, varying only the Stage 3b data source\)\. The dataset turns out to be the largest single lever in the recipe, with average generation accuracy ranging from 26\.2% to 46\.9% and individual tasks moving by as much as 30\.5 pp on ARC\-E \(Table[3](https://arxiv.org/html/2603.26556#S5.T3)\)\. AM\-Qwen3\-Distilled, which contains Qwen3\-generated responses verified for correctness, performs best by a wide margin, consistent with prior evidence that distilling on data close to the teacher’s distribution transfers more effectively\(Goldsteinet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib17); Tianet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib21)\)\. We adopt it for all subsequent runs\.

Table 3:Effect of the Stage 3b dataset on downstream performance \(generation\-based evaluation\)\. All runs use the same token budget \(500M\); Avg is the mean over the six benchmarks\.We also study how many tokens to spend on each distillation stage\. Adding Stage 3a tokens keeps reducing held\-out KL, but these loss improvements largely do not lead to better downstream generation after Stage 3b \(Appendix[J](https://arxiv.org/html/2603.26556#A10)\)\. Training Stage 3b longer leaves average generation roughly flat, as gains on Chinese, instruction\-following, and code benchmarks offset declines on English ones \(Appendix[K](https://arxiv.org/html/2603.26556#A11)\)\. We use 500M tokens for each stage, capturing the modest Stage 3a gains on the tasks that do respond while balancing the Stage 3b trade\-offs\.

### 5\.3Architecture comparison

We now compare the architectures produced by the full GenDistill pipeline, evaluating how the choice of sequence mixer and the retention of attention layers \(Hybrid vs\. Pure\) affect downstream generation quality\. We distil five sequence mixers under the identical pipeline, spanning a range of the linear\-attention design space: KDA, Mamba2, Gated DeltaNet\(Yanget al\.,[2025c](https://arxiv.org/html/2603.26556#bib.bib28)\), GLA\(Yanget al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib27)\), and Lightning Attention\(Qinet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib29)\)\. Mamba2 serves as the established baseline, being one of the most widely adopted SSMs and used in prior distillation work \(Phi\-Mamba, Llamba\)\. To isolate architectural effects, we change only the sequence mixer and whether attention layers are retained, keeping all other hyperparameters fixed\. The design choices behind the pipeline \(pretraining budget, dataset, objective, and parameter freezing\) are established in the preceding Sections[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)and[5\.2](https://arxiv.org/html/2603.26556#S5.SS2), and full training and configuration details are provided in Appendix[D](https://arxiv.org/html/2603.26556#A4)\.

Table 4:Comparison of distilled architectures \(generation\-based evaluation, EvalScope\)\. “Hybrid” models retain \(the selected\) 7 attention layers, “Pure” models use 0\. The lower block repeats the selected Hybrid\-KDA architecture and recipe at 1\.7B against a Qwen3\-1\.7B teacher\. Results are averaged over evaluation seeds \(3 at 0\.6B, 2 at 1\.7B\)\. Best 0\.6B student result per column inbold; the 1\.7B rows are compared to their own teacher and are not bolded\.†Completion\-only masking yields near\-chance WinoGrande and HellaSwag*generation*accuracy at 1\.7B \(extraction artifact, Appendix[M](https://arxiv.org/html/2603.26556#A13)\); the corresponding perplexity\-based scores are intact\.Table[4](https://arxiv.org/html/2603.26556#S5.T4)shows that retaining a subset of attention layers is critical: Hybrid models consistently outperform their Pure counterparts, with the largest separations on multi\-step reasoning benchmarks \(BBH, GSM8K\)\. This supports preserving attention for compositional computation while using linear mixers for the majority of layers\. Among the linear mixers, KDA is the strongest, consistently outperforming Mamba2, with the advantage most pronounced on the hardest reasoning and code\-generation tasks\. Every mixer here is distilled through the identical GenDistill pipeline, so the comparison isolates the sequence mixer\. Because Mamba2 is the student used by MOHAWK and Llamba, the Mamba rows act as a controlled prior\-student baseline at the same teacher, token budget, and data\. The three additional mixers trail KDA and Hybrid Mamba on average and trade off against each other across task categories: GLA is stronger on knowledge and common\-sense benchmarks, whereas Gated DeltaNet is stronger on math and code generation\. Lightning Attention is the weakest of the sequence mixers, failing to reach a usable generation regime under the shared recipe777Lightning Attention’s weak performance may stem from the shared GenDistill recipe not being tuned to this specific mixer, although the same untuned recipe is applied to all the other mixers as well\.\. Perplexity\-based results for all architectures are reported in Appendix[O](https://arxiv.org/html/2603.26556#A15)\(Table[22](https://arxiv.org/html/2603.26556#A15.T22)\)\.

Overall, the best Hybrid\-KDA student retains 86–99% of teacher accuracy on knowledge and common\-sense tasks, confirming that the majority of the teacher’s factual and linguistic knowledge transfers successfully\. The same architecture and recipe scale to a Qwen3\-1\.7B teacher \(Table[4](https://arxiv.org/html/2603.26556#S5.T4), lower block\), where Hybrid\-KDA again tracks its teacher across knowledge, reasoning, code, and instruction following, with only the commonsense multiple\-choice benchmarks depressed by the completion\-only extraction artifact noted in the table\. The largest remaining gaps appear on multi\-step reasoning \(BBH: 69%, GSM8K: 60%\) and code generation \(HumanEval: 56%\), tasks that require extended autoregressive coherence\. Closing these gaps likely requires either larger token budgets, on\-policy distillation, or dedicated RL training\.

#### 5\.3\.1Efficiency Analysis

Having selected the architecture on quality, we now turn to the efficiency that motivates replacing attention in the first place, since linear\-time mixers avoid the quadratic compute and growing KV cache that attention incurs at long context\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x4.png)Figure 4:Inference efficiency comparison \(single GPU, batch size 1\)\.\(a\)Memory usage: SSMs use constant𝒪​\(1\)\\mathcal\{O\}\(1\)state, reducing hybrid memory footprint by up to 75%\.\(b\)Time to First Token: Linear SSM scaling yields 2–4×\\timesspeedups at long contexts \(128K\) despite short\-context overhead\.\(c\)Decode speed: SSMs maintain constant throughput; KDA reaches parity with Transformers at 32K tokens\.Figure[4](https://arxiv.org/html/2603.26556#S5.F4)characterizes inference efficiency\. Hybrid architectures reduce KV cache memory by up to 75%888Replacing attention with KDA or Mamba2 slightly*increases*total parameter count \(613M vs\. 596M non\-embedding\) due to extra gating and convolution parameters; the efficiency gains stem from𝒪​\(n\)\\mathcal\{O\}\(n\)vs\.𝒪​\(n2\)\\mathcal\{O\}\(n^\{2\}\)complexity scaling, not parameter reduction\.compared to Transformers, enabling 4×\\timeslarger batch sizes or sequence lengths\. While SSM prefill \(𝒪​\(n\)\\mathcal\{O\}\(n\)\) incurs kernel overhead at short sequences, it scales linearly, becoming 2–4×\\timesfaster than attention \(𝒪​\(n2\)\\mathcal\{O\}\(n^\{2\}\)\) at 128K tokens\. Decoding throughput is stable; KDA is∼\\sim30% slower than Transformers at short contexts but reaches parity by 32K tokens\. Throughput under GPU saturation \(sweeping batch sizes to OOM\) is reported in Appendix[P](https://arxiv.org/html/2603.26556#A16); at 128K tokens, hybrid models achieve up to 5×\\timesthe teacher’s peak throughput by fitting larger batches\.

## 6Discussion

Our results show that generation quality in cross\-architecture distillation depends on both the student architecture and the distillation recipe, and that evaluating models by generation is necessary to judge either one reliably\. In our study, perplexity and generation accuracy tell different stories on the design choices that most affect the student, namely the training objective \(KD vs\. SFT\), the loss\-masking scope, attention freezing \(Table[2](https://arxiv.org/html/2603.26556#S5.T2)\), and the choice of sequence mixer \(Section[5\.3](https://arxiv.org/html/2603.26556#S5.SS3)\)\. SFT achieves*lower*perplexity than KD but*worse*generation accuracy by 8–11 pp, and freezing attention layers yields a 5\.2 pp gain in generation accuracy with less than 1 pp difference in perplexity\. The same disagreement persists when we scale the student to a 1\.7B teacher \(Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\), and it also appears when evaluating a public distilled checkpoint, where QRWKV6\-7B\-Instruct nearly matches its Qwen2\.5\-7B teacher under perplexity but trails by 20\.8 pp under generation\-based evaluation \(Table[5](https://arxiv.org/html/2603.26556#A1.T5)\)\. Although the limitations of perplexity as a quality proxy are well known, our results suggest that the problem is particularly severe in cross\-architecture distillation, where perplexity\-only evaluation is still common\. We therefore argue that generation\-based evaluation should be the primary protocol for distilled models in this setting\.

With this evaluation lens established, our ablations identify three critical design choices for cross\-architecture distillation at the sub\-1B scale\. First,use KD with completion\-only masking: Table[17](https://arxiv.org/html/2603.26556#A11.T17)shows that it dominates on generation benchmarks across knowledge, reasoning, and instruction\-following tasks, while SFT only appears competitive under perplexity\-based evaluation\. Second,use teacher\-aligned instruction data: Table[3](https://arxiv.org/html/2603.26556#S5.T3)shows a 20 pp swing in average accuracy from dataset choice alone, with teacher\-aligned data \(AM\-Qwen3\-Distilled\) outperforming broader instruction corpora by a wide margin\. Third,freeze attention layers during Stage 3b: Table[2](https://arxiv.org/html/2603.26556#S5.T2)shows a 5\.2 pp average generation improvement from freezing, with minimal effect under perplexity\-based evaluation\. These choices share a common logic, each preserving the teacher’s pretrained structure and steering the student’s limited capacity toward generation rather than letting instruction tuning overwrite what pretraining established\.

Two further choices are less clear\-cut\. Extending Stage 3b training is a trade\-off rather than a uniform gain, improving Chinese knowledge and instruction following while degrading some English benchmarks \(Table[17](https://arxiv.org/html/2603.26556#A11.T17)\), so the best duration depends on the target evaluation profile\. The pipeline structure itself also matters, as merging Stages 3a and 3b into a single interleaved phase underperforms the sequential pipeline at matched compute \(Appendix[L](https://arxiv.org/html/2603.26556#A12)\), confirming the value of staged training with dedicated objectives\.

The architecture comparison points to a single best student, Hybrid\-KDA, but its value is conditional on the deployment regime\. The efficiency that motivates replacing attention only materializes at long context, where the hybrid wins decisively on memory, time\-to\-first\-token, and decode throughput \(Figure[4](https://arxiv.org/html/2603.26556#S5.F4)\), whereas at short context it is slower than the Transformer teacher it replaces\. The hybrid is therefore best understood as a targeted substitute for long\-context workloads rather than a universal drop\-in\.

### 6\.1Limitations

Comparing whole distillation pipelines is difficult, since published pipelines differ along many axes at once \(e\.g\. the teacher, the student mixer, the number of stages, the loss functions, token budgets, datasets\)\. These axes are also interdependent\. A head\-to\-head between two pipelines then shows which bundle of choices wins on a given setup, and cannot isolate the ingredient responsible\. Consequently, our aim is not to make concrete claims that GenDistill produces higher\-quality models than other distillation pipelines\. Our contribution is the generation\-based evaluation analysis and the resulting recipe, a controlled attribution of which distillation ingredients drive generation quality\.

A further limitation is long\-context*quality*, as distinct from long\-context*efficiency*\. The hybrid wins decisively on efficiency \(Figure[4](https://arxiv.org/html/2603.26556#S5.F4)\), but distillation on 4K\-token sequences does not transfer to longer contexts: Hybrid KDA retains only 55% of teacher LongBench accuracy and degrades on NIAH retrieval beyond 16K tokens \(Appendix[Q](https://arxiv.org/html/2603.26556#A17)\)\. This reflects the missing long\-context training stage rather than an architectural limit, since the teacher itself acquired its 32K capability through a dedicated phase\(Yanget al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib4)\), and concurrent work recovers long\-context quality once such a stage is added\(Chenet al\.,[2026](https://arxiv.org/html/2603.26556#bib.bib61); Teamet al\.,[2026](https://arxiv.org/html/2603.26556#bib.bib60)\)\.

Finally, our Stage 3b training used only 250M–750M tokens from static datasets\. On\-policy distillation using student\-generated responses\(Agarwalet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib53)\)and larger compute budgets could close the remaining quality gap\. Dataset composition effects such as curriculum ordering and mixing ratios were also not explored and may yield further improvements\.

## 7Conclusion

This work shows that producing distilled hybrid models with genuine generation quality requires jointly designing the student architecture and the distillation pipeline, guided by generation\-based evaluation\. Perplexity\-based metrics, still the norm in cross\-architecture distillation, are a systematically misleading proxy\. The divergence appeared consistently across every design axis in our 0\.6B\-parameter ablations, persisted in a 1\.7B scale\-up \(Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\), and is corroborated at 7B scale \(Table[5](https://arxiv.org/html/2603.26556#A1.T5)\)\. While the general limitation of perplexity is well\-known, our results show this gap is especially severe in cross\-architecture distillation, where perplexity\-only reporting can lead to incorrect conclusions about which methods actually produce models that generate well\. We argue that generation\-based evaluation should be the primary protocol in this setting\.

On the practical side, GenDistill’s systematic ablations yield a concrete distillation recipe: KD with completion\-only masking, teacher\-aligned instruction data, and frozen attention layers\. This combination produces a Hybrid\-KDA student that retains 86–90% of teacher accuracy on knowledge benchmarks while reducing KV cache memory by∼\{\\sim\}75%\. Confirming the recipe at multi\-billion\-parameter scale, and extending it with dedicated long\-context training, remain the most pressing directions for future work\.

## Broader Impact Statement

By lowering the compute and memory cost of running capable language models, cross\-architecture distillation can broaden access to these models and enable deployment at larger scale and on more constrained hardware\. At the same time, cheaper and faster generation lowers the cost of producing text at scale, so the standard misuse considerations for large language models, such as large\-scale generation of low\-quality or misleading content, apply here as well, with efficiency changing the scale of these risks rather than their qualitative nature\. Beyond this, we do not foresee risks specific to this work\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.External Links:2306\.13649,[Link](https://arxiv.org/abs/2306.13649)Cited by:[§6\.1](https://arxiv.org/html/2603.26556#S6.SS1.p3.1)\.
- A\. Ali, L\. Wolf, and I\. Titov \(2024\)Mitigating copy bias in in\-context learning through neuron pruning\.External Links:2410\.01288,[Link](https://arxiv.org/abs/2410.01288)Cited by:[Appendix N](https://arxiv.org/html/2603.26556#A14.SS0.SSS0.Px4.p2.2)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2023\)LongBench: a bilingual, multitask benchmark for long context understanding\.arXiv preprint arXiv:2308\.14508\.Cited by:[Appendix Q](https://arxiv.org/html/2603.26556#A17.p1.1),[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- E\. Bakouch, L\. Ben Allal, A\. Lozhkov, N\. Tazi, L\. Tunstall, C\. M\. Patiño, E\. Beeching, A\. Roucher, A\. J\. Reedi, Q\. Gallouédec, K\. Rasul, N\. Habib, C\. Fourrier, H\. Kydlicek, G\. Penedo, H\. Larcher, M\. Morlon, V\. Srivastav, J\. Lochner, X\. Nguyen, C\. Raffel, L\. von Werra, and T\. Wolf \(2025\)SmolLM3: smol, multilingual, long\-context reasoner\.Note:[https://huggingface\.co/blog/smollm3](https://huggingface.co/blog/smollm3)Cited by:[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- A\. Bick, T\. Katsch, N\. Sohoni, A\. Desai, and A\. Gu \(2025a\)Llamba: scaling distilled recurrent models for efficient language processing\.External Links:2502\.14458,[Link](https://arxiv.org/abs/2502.14458)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p3.1),[§1](https://arxiv.org/html/2603.26556#S1.p5.1),[§2](https://arxiv.org/html/2603.26556#S2.p3.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1),[§3\.2\.2](https://arxiv.org/html/2603.26556#S3.SS2.SSS2.p2.5),[§3\.2](https://arxiv.org/html/2603.26556#S3.SS2.p1.4)\.
- A\. Bick, K\. Li, E\. Xing, J\. Z\. Kolter, and A\. Gu \(2024\)Transformers to ssms: distilling quadratic knowledge to subquadratic models\.Advances in Neural Information Processing Systems37,pp\. 31788–31812\.Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p3.1),[§1](https://arxiv.org/html/2603.26556#S1.p5.1),[§2](https://arxiv.org/html/2603.26556#S2.p3.1),[§2](https://arxiv.org/html/2603.26556#S2.p5.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p3.1),[§3\.2\.1](https://arxiv.org/html/2603.26556#S3.SS2.SSS1.p2.5),[§3\.2\.2](https://arxiv.org/html/2603.26556#S3.SS2.SSS2.p2.5),[§3\.2](https://arxiv.org/html/2603.26556#S3.SS2.p1.4),[Table 4](https://arxiv.org/html/2603.26556#S5.T4.4.2.15.2.1.1)\.
- A\. Bick, E\. Xing, and A\. Gu \(2025b\)Understanding the skill gap in recurrent language models: the role of the gather\-and\-aggregate mechanism\.External Links:2504\.18574,[Link](https://arxiv.org/abs/2504.18574)Cited by:[2nd item](https://arxiv.org/html/2603.26556#A5.I2.i2.p1.1),[§2](https://arxiv.org/html/2603.26556#S2.p4.1),[§3\.1\.1](https://arxiv.org/html/2603.26556#S3.SS1.SSS1.p1.1)\.
- Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- H\. Chen, Z\. Liu, X\. Wang, Y\. Tian, and Y\. Wang \(2024\)DiJiang: efficient large language models through compact kernelization\.External Links:2403\.19928,[Link](https://arxiv.org/abs/2403.19928)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p2.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Chen, Z\. L\. Thai, Z\. Zhou, Z\. Zhang, X\. Shen, S\. Wang, C\. Xiao, X\. Han, and Z\. Liu \(2026\)Hybrid linear attention done right: efficient distillation and effective architectures for extremely long contexts\.External Links:2601\.22156,[Link](https://arxiv.org/abs/2601.22156)Cited by:[Appendix Q](https://arxiv.org/html/2603.26556#A17.p4.1),[§6\.1](https://arxiv.org/html/2603.26556#S6.SS1.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- T\. Dao and A\. Gu \(2024\)Transformers are ssms: generalized models and efficient algorithms through structured state space duality\.External Links:2405\.21060,[Link](https://arxiv.org/abs/2405.21060)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p2.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1)\.
- S\. De, S\. L\. Smith, A\. Fernando, A\. Botev, G\. Cristian\-Muraru, A\. Gu, R\. Haroun, L\. Berrada, Y\. Chen, S\. Srinivasan, G\. Desjardins, A\. Doucet, D\. Budden, Y\. W\. Teh, R\. Pascanu, N\. D\. Freitas, and C\. Gulcehre \(2024\)Griffin: mixing gated linear recurrences with local attention for efficient language models\.External Links:2402\.19427,[Link](https://arxiv.org/abs/2402.19427)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p5.1)\.
- X\. Dong, Y\. Fu, S\. Diao, W\. Byeon, Z\. Chen, A\. S\. Mahabaleshwarkar, S\. Liu, M\. V\. Keirsbilck, M\. Chen, Y\. Suhara, Y\. Lin, J\. Kautz, and P\. Molchanov \(2024\)Hymba: a hybrid\-head architecture for small language models\.External Links:2411\.13676,[Link](https://arxiv.org/abs/2411.13676)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p4.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golber, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Link](https://zenodo.org/records/10256836)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p4.1),[§4](https://arxiv.org/html/2603.26556#S4.p3.1)\.
- A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani, C\. Gardent, M\. Wu, A\. F\. Aji, H\. T\. Madabushi, and P\. Minervini \(2024\)Are we done with mmlu?\.External Links:2406\.04127,[Link](https://arxiv.org/abs/2406.04127)Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1),[footnote 5](https://arxiv.org/html/2603.26556#footnote5)\.
- D\. Goldstein, E\. Alcaide, J\. Lu, and E\. Cheah \(2025\)RADLADS: rapid attention distillation to linear attention decoders at scale\.External Links:2505\.03005,[Link](https://arxiv.org/abs/2505.03005)Cited by:[Appendix F](https://arxiv.org/html/2603.26556#A6.p2.1),[§1](https://arxiv.org/html/2603.26556#S1.p3.1),[§1](https://arxiv.org/html/2603.26556#S1.p5.1),[§2](https://arxiv.org/html/2603.26556#S2.p3.1),[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- A\. Gu and T\. Dao \(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p2.1)\.
- Y\. Gu, Q\. Hu, S\. Yang, H\. Xi, J\. Chen, S\. Han, and H\. Cai \(2025\)Jet\-nemotron: efficient language model with post neural architecture search\.External Links:2508\.15884,[Link](https://arxiv.org/abs/2508.15884)Cited by:[Appendix E](https://arxiv.org/html/2603.26556#A5.p2.6),[§2](https://arxiv.org/html/2603.26556#S2.p5.1),[§3\.1\.1](https://arxiv.org/html/2603.26556#S3.SS1.SSS1.p2.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[Appendix Q](https://arxiv.org/html/2603.26556#A17.p1.1),[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, J\. Lei, Y\. Fu, M\. Sun, and J\. He \(2023\)C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.Advances in Neural Information Processing Systems36\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret \(2020\)Transformers are RNNs: fast autoregressive transformers with linear attention\.InInternational Conference on Machine Learning,pp\. 5156–5165\.Cited by:[Appendix B](https://arxiv.org/html/2603.26556#A2.SS0.SSS0.Px1.p1.1)\.
- J\. G\. Kostelec and Q\. Guo \(2025\)FlashEVA: accelerating llm inference via efficient attention\.External Links:2511\.00576,[Link](https://arxiv.org/abs/2511.00576)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p2.1)\.
- H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. Baldwin \(2023\)CMMLU: Measuring massive multitask language understanding in Chinese\.arXiv preprint arXiv:2306\.09212\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, S\. Yang, S\. Tan, M\. Mishra, R\. Panda, J\. Zhou, and Y\. Kim \(2025\)Distilling to hybrid attention models via kl\-guided layer selection\.External Links:2512\.20569,[Link](https://arxiv.org/abs/2512.20569)Cited by:[3rd item](https://arxiv.org/html/2603.26556#A5.I1.i3.p1.1),[Appendix E](https://arxiv.org/html/2603.26556#A5.p2.6),[§2](https://arxiv.org/html/2603.26556#S2.p5.1),[§3\.1\.1](https://arxiv.org/html/2603.26556#S3.SS1.SSS1.p4.1)\.
- O\. Lieber, B\. Lenz, H\. Bata, G\. Cohen, J\. Osin, I\. Dalmedigos, E\. Safahi, S\. Meirom, Y\. Belinkov, S\. Shalev\-Shwartz, O\. Abend, R\. Alon, T\. Asida, A\. Bergman, R\. Glozman, M\. Gokhman, A\. Manevich, N\. Ratner, N\. Rozen, E\. Shwartz, M\. Zusman, and Y\. Shoham \(2024\)Jamba: a hybrid transformer\-mamba language model\.External Links:2403\.19887,[Link](https://arxiv.org/abs/2403.19887)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p5.1)\.
- B\. Lin, Y\. Zhou, J\. Yu, M\. Yan, H\. Yang, W\. Yao, Y\. Wang, Y\. Zhai, H\. Lin, B\. Han, X\. Liu, and T\. Liu \(2025a\)Debunk the myth of sft generalization\.External Links:2510\.00237,[Link](https://arxiv.org/abs/2510.00237)Cited by:[Appendix N](https://arxiv.org/html/2603.26556#A14.SS0.SSS0.Px4.p2.2)\.
- Y\. Lin, Z\. Li, P\. Yue, and X\. Liu \(2025b\)ARWKV: pretrain is not what we need, an rnn\-attention\-based language model born from transformer\.External Links:2501\.15570Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p4.1)\.
- J\. Mercat, I\. Vasiljevic, S\. Keh, K\. Arora, A\. Dave, A\. Gaidon, and T\. Kollar \(2024\)Linearizing large language models\.External Links:2405\.06640,[Link](https://arxiv.org/abs/2405.06640)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p2.1)\.
- ModelScope Team \(2024\)EvalScope: a framework for model evaluation\.External Links:[Link](https://github.com/modelscope/evalscope)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p4.1),[§4](https://arxiv.org/html/2603.26556#S4.p3.1)\.
- D\. Nathawani, I\. Gitman, S\. Majumdar, E\. Bakhturina, A\. Sunil Mahabaleshwarkar, J\. Zhang, and J\. Polak Scowcroft \(2025\)Nemotron\-Post\-Training\-Dataset\-v1\.NVIDIA\.External Links:[Link](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v1)Cited by:[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, G\. Boleda, R\. Fernández, and M\. Baroni \(2016\)The lambada dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 1525–1534\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- B\. Peng, E\. Alcaide, H\. Hou, G\. Song, J\. S\. Wind, R\. Zhang, W\. Merrill, D\. Goldstein, T\. Wu, S\. Utpala, J\. Lu, K\. Tan, N\. Wilce, D\. Wuttke, and C\. Zhou\-Zheng \(2025\)RWKV\-7 "goose" with expressive dynamic state evolution\.External Links:2503\.14456,[Link](https://arxiv.org/abs/2503.14456)Cited by:[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1)\.
- Z\. Qin, W\. Sun, D\. Li, X\. Shen, W\. Sun, and Y\. Zhong \(2024\)Lightning attention\-2: a free lunch for handling unlimited sequence lengths in large language models\.External Links:2401\.04658,[Link](https://arxiv.org/abs/2401.04658)Cited by:[§5\.3](https://arxiv.org/html/2603.26556#S5.SS3.p1.1)\.
- K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi \(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- I\. Schlag, K\. Irie, and J\. Schmidhuber \(2021\)Linear transformers are secretly fast weight programmers\.InInternational Conference on Machine Learning,pp\. 9355–9366\.Cited by:[Appendix B](https://arxiv.org/html/2603.26556#A2.SS0.SSS0.Px2.p1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2023\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.External Links:2310\.11324,[Link](https://arxiv.org/abs/2310.11324)Cited by:[Appendix N](https://arxiv.org/html/2603.26556#A14.SS0.SSS0.Px4.p2.2)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. Wei \(2023\)Challenging BIG\-Bench Tasks and Whether Chain\-of\-Thought Can Solve Them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Team, Y\. Zhang, Z\. Lin, X\. Yao, J\. Hu, F\. Meng, C\. Liu, X\. Men, S\. Yang, Z\. Li, W\. Li, E\. Lu, W\. Liu, Y\. Chen, W\. Xu, L\. Yu, Y\. Wang, Y\. Fan, L\. Zhong, E\. Yuan, D\. Zhang, Y\. Zhang, T\. Y\. Liu, H\. Wang, S\. Fang, W\. He, S\. Liu, Y\. Li, J\. Su, J\. Qiu, B\. Pang, J\. Yan, Z\. Jiang, W\. Huang, B\. Yin, J\. You, C\. Wei, Z\. Wang, C\. Hong, Y\. Chen, G\. Chen, Y\. Wang, H\. Zheng, F\. Wang, Y\. Liu, M\. Dong, Z\. Zhang, S\. Pan, W\. Wu, Y\. Wu, L\. Guan, J\. Tao, G\. Fu, X\. Xu, Y\. Wang, G\. Lai, Y\. Wu, X\. Zhou, Z\. Yang, and Y\. Du \(2025\)Kimi linear: an expressive, efficient attention architecture\.External Links:2510\.26692,[Link](https://arxiv.org/abs/2510.26692)Cited by:[Appendix B](https://arxiv.org/html/2603.26556#A2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2603.26556#S2.p4.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p3.1),[footnote 11](https://arxiv.org/html/2603.26556#footnote11)\.
- M\. Team, W\. An, Y\. Chen, Y\. Fang, J\. Li, X\. Li, Y\. Li, Y\. Li, Y\. Li, B\. Lin, C\. Liu, H\. Liu, S\. Liu, H\. Lyu, Y\. Pan, S\. Ren, X\. Shen, Z\. Su, H\. Sun, Y\. Sun, Z\. L\. Thai, X\. Tian, R\. Wang, X\. Wang, Y\. Wang, B\. Wu, X\. Xu, D\. Xu, S\. Xue, J\. Yang, B\. Zhang, J\. Zhang, L\. Zhang, S\. Zhang, X\. Zhang, X\. Zhang, Z\. Zhang, H\. Zhao, J\. Zhao, J\. Zhou, Z\. Zhou, S\. Wang, C\. Xiao, X\. Han, Z\. Liu, and M\. Sun \(2026\)MiniCPM\-sala: hybridizing sparse and linear attention for efficient long\-context modeling\.External Links:2602\.11761,[Link](https://arxiv.org/abs/2602.11761)Cited by:[Appendix Q](https://arxiv.org/html/2603.26556#A17.p4.1),[§1](https://arxiv.org/html/2603.26556#S1.p5.1),[§6\.1](https://arxiv.org/html/2603.26556#S6.SS1.p2.1)\.
- Teknium \(2023\)OpenHermes 2\.5: an open dataset of synthetic data for generalist llm assistants\.HuggingFace\.External Links:[Link](https://huggingface.co/datasets/teknium/OpenHermes-2.5)Cited by:[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- X\. Tian, Y\. Ji, H\. Wang, S\. Chen, S\. Zhao, Y\. Peng, H\. Zhao, and X\. Li \(2025\)Not all correct answers are equal: why your distillation source matters\.External Links:2505\.14464,[Link](https://arxiv.org/abs/2505.14464)Cited by:[§3\.2\.3](https://arxiv.org/html/2603.26556#S3.SS2.SSS3.p2.1),[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- J\. Wang, W\. Li, D\. Paliotta, D\. Ritter, A\. M\. Rush, and T\. Dao \(2025\)M1: towards scalable test\-time compute with mamba reasoning models\.External Links:2504\.10449,[Link](https://arxiv.org/abs/2504.10449)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p5.1)\.
- J\. Wang, D\. Paliotta, A\. May, A\. M\. Rush, and T\. Dao \(2024\)The mamba in the llama: distilling and accelerating hybrid models\.External Links:2408\.15237,[Link](https://arxiv.org/abs/2408.15237)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p3.1),[§1](https://arxiv.org/html/2603.26556#S1.p5.1),[§2](https://arxiv.org/html/2603.26556#S2.p3.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1),[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025a\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p3.1),[§6\.1](https://arxiv.org/html/2603.26556#S6.SS1.p2.1)\.
- M\. Yang, M\. Rezagholizadeh, G\. Li, V\. Appia, and E\. Barsoum \(2025b\)Zebra\-llama: towards extremely efficient hybrid models\.External Links:2505\.17272,[Link](https://arxiv.org/abs/2505.17272)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p3.1),[§2](https://arxiv.org/html/2603.26556#S2.p5.1)\.
- S\. Yang, J\. Kautz, and A\. Hatamizadeh \(2025c\)Gated delta networks: improving mamba2 with delta rule\.External Links:2412\.06464,[Link](https://arxiv.org/abs/2412.06464)Cited by:[Appendix B](https://arxiv.org/html/2603.26556#A2.SS0.SSS0.Px2.p1.3),[§1](https://arxiv.org/html/2603.26556#S1.p2.1),[§3\.1](https://arxiv.org/html/2603.26556#S3.SS1.p2.1),[§5\.3](https://arxiv.org/html/2603.26556#S5.SS3.p1.1)\.
- S\. Yang, B\. Wang, Y\. Shen, R\. Panda, and Y\. Kim \(2023\)Gated linear attention transformers with hardware\-efficient training\.External Links:2312\.06635,[Link](https://arxiv.org/abs/2312.06635)Cited by:[§1](https://arxiv.org/html/2603.26556#S1.p2.1),[§5\.3](https://arxiv.org/html/2603.26556#S5.SS3.p1.1)\.
- S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. Kim \(2024\)Parallelizing linear transformers with the delta rule over sequence length\.External Links:2406\.06484,[Link](https://arxiv.org/abs/2406.06484)Cited by:[Appendix B](https://arxiv.org/html/2603.26556#A2.SS0.SSS0.Px2.p1.1)\.
- S\. Yang and Y\. Zhang \(2024\)FLA: a triton\-based library for hardware\-efficient implementations of linear attention mechanism\.External Links:[Link](https://github.com/fla-org/flash-linear-attention)Cited by:[footnote 1](https://arxiv.org/html/2603.26556#footnote1)\.
- Z\. Yao, Y\. Liu, Y\. Chen, J\. Chen, J\. Fang, L\. Hou, J\. Li, and T\. Chua \(2025\)Are reasoning models more prone to hallucination?\.External Links:2505\.23646,[Link](https://arxiv.org/abs/2505.23646)Cited by:[Appendix N](https://arxiv.org/html/2603.26556#A14.SS0.SSS0.Px4.p2.2)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Zhang \(2025\)Max’s awesome datasets\.Note:[https://github\.com/Mxoder/Maxs\-Awesome\-Datasets](https://github.com/Mxoder/Maxs-Awesome-Datasets)Cited by:[§5\.2](https://arxiv.org/html/2603.26556#S5.SS2.p2.1)\.
- M\. Zhang, S\. Arora, R\. Chalamala, A\. Wu, B\. Spector, A\. Singhal, K\. Ramesh, and C\. Ré \(2025\)LoLCATs: on low\-rank linearizing of large language models\.External Links:2410\.10254,[Link](https://arxiv.org/abs/2410.10254)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p2.1)\.
- M\. Zhang, K\. Bhatia, H\. Kumbong, and C\. Ré \(2024\)The hedgehog & the porcupine: expressive linear attentions with softmax mimicry\.External Links:2402\.04347,[Link](https://arxiv.org/abs/2402.04347)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p2.1)\.
- C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. Huang \(2024\)Large language models are not robust multiple choice selectors\.External Links:2309\.03882,[Link](https://arxiv.org/abs/2309.03882)Cited by:[Appendix N](https://arxiv.org/html/2603.26556#A14.SS0.SSS0.Px4.p2.2)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§4](https://arxiv.org/html/2603.26556#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Zuo, M\. Velikanov, I\. Chahed, Y\. Belkada, D\. E\. Rhayem, G\. Kunsch, H\. Hacid, H\. Yous, B\. Farhat, I\. Khadraoui, M\. Farooq, G\. Campesan, R\. Cojocaru, Y\. Djilali, S\. Hu, I\. Chaabane, P\. Khanna, M\. E\. A\. Seddik, N\. D\. Huynh, P\. L\. Khac, L\. AlQadi, B\. Mokeddem, M\. Chami, A\. Abubaker, M\. Lubinets, K\. Piskorski, and S\. Frikha \(2025\)Falcon\-h1: a family of hybrid\-head language models redefining efficiency and performance\.External Links:2507\.22448,[Link](https://arxiv.org/abs/2507.22448)Cited by:[§2](https://arxiv.org/html/2603.26556#S2.p4.1),[§3\.2\.1](https://arxiv.org/html/2603.26556#S3.SS2.SSS1.p2.5)\.

This appendix collects the derivations, full result tables, and ablations that support the main text, organized into the thematic groups below\.

Backing for the opening example

Setup and reproducibility

Recipe and design\-choice ablations

Extended results and analysis

Efficiency and long\-context

## Appendix AMotivating Example, Evaluation Protocol Gap

This section gives the full numbers behind the motivating example shown in the introduction \(Figure[1](https://arxiv.org/html/2603.26556#S1.F1)\)\.

Table 5:Evaluation protocol gap for QRWKV6\-7B\-Instruct, distilled from Qwen2\.5\-7B\-Instruct\. PPL\-based results are taken from the original paper; generation\-based results are our evaluation using EvalScope\. Both columns evaluate the*same model checkpoint*; only the evaluation protocol differs\. On overlapping benchmarks, the average teacher–student gap jumps from0\.2 pp\(PPL\) to20\.8 pp\(generation\)\.
## Appendix BKimi Delta Attention Formulation

##### Linear attention as associative memory\.

Linear attention\(Katharopouloset al\.,[2020](https://arxiv.org/html/2603.26556#bib.bib48)\)replaces the softmax attention kernel with an outer\-product update to a matrix\-valued recurrent state𝐒t∈ℝdk×dv\\mathbf\{S\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{v\}\}:

𝐒t=𝐒t−1\+𝒌t​𝒗t⊤,𝒐t=𝐒t⊤​𝒒t\.\\mathbf\{S\}\_\{t\}=\\mathbf\{S\}\_\{t\-1\}\+\\bm\{k\}\_\{t\}\\bm\{v\}\_\{t\}^\{\\top\},\\qquad\\bm\{o\}\_\{t\}=\\mathbf\{S\}\_\{t\}^\{\\top\}\\bm\{q\}\_\{t\}\.\(1\)𝐒t\\mathbf\{S\}\_\{t\}serves as an associative memory that stores key–value associations\. However, the purely additive update accumulates information without any forgetting, causing interference as the context grows\.

##### The delta rule and Gated DeltaNet\.

DeltaNet\(Schlaget al\.,[2021](https://arxiv.org/html/2603.26556#bib.bib49); Yanget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib50)\)addresses this by interpreting the state update as online gradient descent on a per\-token reconstruction lossℒt​\(𝐒\)=12​‖𝐒⊤​𝒌t−𝒗t‖2\\mathcal\{L\}\_\{t\}\(\\mathbf\{S\}\)=\\frac\{1\}\{2\}\\\|\\mathbf\{S\}^\{\\top\}\\bm\{k\}\_\{t\}\-\\bm\{v\}\_\{t\}\\\|^\{2\}, yielding a rank\-1 corrective update:

𝐒t=\(𝐈−βt​𝒌t​𝒌t⊤\)​𝐒t−1\+βt​𝒌t​𝒗t⊤,\\mathbf\{S\}\_\{t\}=\(\\mathbf\{I\}\-\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{k\}\_\{t\}^\{\\top\}\)\\,\\mathbf\{S\}\_\{t\-1\}\+\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{v\}\_\{t\}^\{\\top\},\(2\)whereβt\\beta\_\{t\}is a learned step size\. This delta rule enables the memory to correct stale associations rather than merely accumulating them\. Gated DeltaNet \(GDN\)\(Yanget al\.,[2025c](https://arxiv.org/html/2603.26556#bib.bib28)\)further introduces a*scalar*forget gateαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\], acting as weight decay on the memory:

𝐒t=αt​\(𝐈−βt​𝒌t​𝒌t⊤\)​𝐒t−1\+βt​𝒌t​𝒗t⊤\.\\mathbf\{S\}\_\{t\}=\\alpha\_\{t\}\\,\(\\mathbf\{I\}\-\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{k\}\_\{t\}^\{\\top\}\)\\,\\mathbf\{S\}\_\{t\-1\}\+\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{v\}\_\{t\}^\{\\top\}\.\(3\)This provides a principled mechanism to control memory lifespan\. However, GDN’s scalar gate applies a uniform decay rate across all feature dimensions, limiting the model’s ability to selectively retain or forget information at a fine\-grained level\.

##### Channel\-wise gating in KDA\.

Kimi Delta Attention \(KDA\)\(Teamet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib6)\)refines GDN by replacing the scalar gate with a*channel\-wise*\(i\.e\., per\-dimension\) diagonal gateDiag​\(𝜶t\)∈ℝdk×dk\\mathrm\{Diag\}\(\\bm\{\\alpha\}\_\{t\}\)\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{k\}\}, where each feature dimension maintains an independent forgetting rate:

𝐒t=\(𝐈−βt​𝒌t​𝒌t⊤\)​Diag​\(𝜶t\)​𝐒t−1\+βt​𝒌t​𝒗t⊤,𝒐t=𝐒t⊤​𝒒t\.\\mathbf\{S\}\_\{t\}=\(\\mathbf\{I\}\-\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{k\}\_\{t\}^\{\\top\}\)\\,\\mathrm\{Diag\}\(\\bm\{\\alpha\}\_\{t\}\)\\,\\mathbf\{S\}\_\{t\-1\}\+\\beta\_\{t\}\\bm\{k\}\_\{t\}\\bm\{v\}\_\{t\}^\{\\top\},\\qquad\\bm\{o\}\_\{t\}=\\mathbf\{S\}\_\{t\}^\{\\top\}\\bm\{q\}\_\{t\}\.\(4\)

## Appendix CModel Architecture Details

##### Base model \(teacher\)\.

We distill from Qwen3\-0\.6B, a dense Transformer with grouped\-query attention\. Table[6](https://arxiv.org/html/2603.26556#A3.T6)summarizes its configuration\.

Table 6:Qwen3\-0\.6B base model \(teacher\) configuration\.The student models are based on the 596M\-parameter teacher \(non\-embedding\) but replace a subset of attention layers with alternative sequence mixers\. Each mixer layer introduces additional gating, convolution, and state\-management parameters beyond the original Q/K/V/O projections, increasing the total parameter count: KDA adds∼\\sim0\.8M per replaced layer \(short convolutions on Q/K/V, gating projections for the delta rule\), while Mamba2 adds∼\\sim2\.1M per replaced layer \(a larger joint input projection that maps to the SSM state components, time\-step, and gating dimensions\)\. Table[7](https://arxiv.org/html/2603.26556#A3.T7)details the mixer configurations\. Each mixer layer replaces the self\-attention block while retaining the original MLP, layer norms, and residual connections\. Pure KDA and Mamba variants use identical mixer configurations but replace all 28 layers\.

Table 7:Hybrid student mixer configurations \(Qwen3\-0\.6B scale\)\. KDA = Kimi Delta Attention\.

## Appendix DHyperparameter Details

Table[8](https://arxiv.org/html/2603.26556#A4.T8)lists the full hyperparameter configuration for each distillation stage\. Stage 1 aligns the outputs of the attention layer between the two models, while Stage 2 aligns the outputs of the whole attention block via L2 loss on pretraining data\. Stage 1 only trains the Q and K projections, keeping the conv1d \(identity initialized\) and other parameters fixed\. Stage 2, meanwhile, trains all mixer parameters\. Stages 3a and 3b perform end\-to\-end knowledge distillation with forward KL divergence\. Both stages freeze the attention layers to prevent catastrophic forgetting of the representations inherited from the teacher model\. Stage 3b switches to instruction data with completion\-only masking \(see Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1.SSS0.Px3)\)\. All stages use AdamW with\(β1,β2\)=\(0\.9,0\.95\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.95\), a warmup\-stable\-decay learning rate schedule \(10% warmup, 10% decay, minimum LR ratio 0\.02\), weight decay 0\.1, gradient clipping at 1\.0, bf16 mixed precision, and DeepSpeed ZeRO Stage 2\.

##### Evaluation configurations\.

We use two complementary evaluation frameworks, chosen to highlight the PPL\-vs\-generation gap that is a central finding of this work\. All evaluations use a batch size of 64 and the SDPA \(Scaled Dot\-Product Attention\) backend\.EvalScope \(generation\-based\):Models are evaluated using the recommended sampling setup: temperature==0\.7, top\-pp==0\.8, top\-kk==20, and seed==42\. Maximum new tokens is set to 1024\. Few\-shot settings: C\-Eval \(5\-shot\), MMLU / MMLU\-Redux \(5\-shot\), ARC\-Easy / ARC\-Challenge \(0\-shot\), HellaSwag \(0\-shot\), WinoGrande \(0\-shot\), GSM8K \(4\-shot chain\-of\-thought\), CMMLU \(0\-shot\), BBH \(0\-shot\), HumanEval \(0\-shot\), IFEval \(0\-shot\), TruthfulQA \(0\-shot\)\.LM\-Eval Harness \(perplexity\-based\):Models are evaluated on likelihood\-based scoring\. Tasks: ARC\-Easy, ARC\-Challenge, HellaSwag, WinoGrande, PIQA, LAMBADA \(OpenAI\), MMLU\. We use 3 seeds for the evals, seeds==1, 2, 3\.

Table 8:Hyperparameters for each distillation stage\.

## Appendix EAttention Layer Selection Ablation

The main text \(Section[3\.1\.1](https://arxiv.org/html/2603.26556#S3.SS1.SSS1)\) separates this within\-pipeline use of perplexity from the final\-evaluation use the paper argues against, and we expand on that distinction here\. The failure mode the paper documents concerns a fully trained student scored over completion tokens, where the teacher–student likelihood gap is what perplexity is asked to capture\. Here perplexity instead scores an undertrained scaffold whose role is exploration rather than deployment, so neither the artefact measured nor the quantity it stands for is the same\.

The design space for layer selection has four axes: the probe dataset, the search procedure, the scoring metric, and the amount of student\-side training performed before each measurement\. The principled upper bound on this spectrum is the attention\-placement step of PostNAS\(Guet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib20)\), which trains a once\-for\-all super\-network containing both attention and linear pathways and then runs beam search over its sub\-networks using direct downstream\-task signals such as loss on the correct MMLU answer span and task accuracy on math and retrieval\. PostNAS is generation\-aware in both probe and metric, and consistent with the broader thesis of this paper\. It is also expensive: per its Appendix A\.2, the attention\-placement step consumes5050B tokens and∼800\{\\sim\}800H100 GPU\-hours\. Two more compute\-efficient points on this spectrum trade per\-candidate training cost for proxy quality\. GA\-S2\(Liet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib19)\)performs a single\-replacement probe at every layer and scores each variant by teacher and student KL after a short distillation pass per variant \(∼700\{\\sim\}700M tokens per probe, totalling between∼7\{\\sim\}7and∼26\{\\sim\}26B tokens at our scale\)\. Our procedure removes per\-candidate training entirely\. We distill a fully\-linear scaffold once at100100M tokens, then run beam search over attention\-layer additions and score each candidate by held\-out perplexity averaged across three probes: associative recall, a key/value retrieval task in the same spirit as PostNAS’s KV signal; C\-EVAL, a knowledge benchmark; and FineWeb\-EDU, a generic language\-modelling probe\. Two of the three probes are task\-shaped, so the input side of our signal is closer in form to PostNAS\-style task\-aware evaluation than to a generic LM\-PPL baseline\.

We compare beam search layer selection against three baselines for the attention budget of 5 attention layers\. We choose this, as with increasing number of attention layers the performance will converge to the teacher model’s performance and the gap between methods will therefore be smaller\. We compare the following selection methods:

- •Uniform:Layers are spaced at equal intervals\.
- •Greedy:Selects top\-kklayers by individual perplexity impact\.
- •Greedy Aggregate \(Learned\):Starts from a linear model and adds attention layers based on held\-out KL divergence after training the resulting hybrid models\(Liet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib19)\)\.
- •Beam Search Add:Starts from a full KDA model and adds attention layers to maximize performance\.
- •Beam Search Replace:starts with a full transformer and progressively replaces attention layers with KDA layers, minimizing perplexity degradation\.

##### Evaluation Tasks\.

We consider evaluating the layer importance on three tasks, and use the average performance on all 3 tasks as the importance measure:

- •AR \(Associative Recall\):A synthetic key\-value retrieval task that directly tests in\-context retrieval capabilities\. We use 40 key\-value pairs per example with 500 total examples\.
- •CEVAL:Chinese language understanding benchmark \(similar task to MMLU, whose performance has been shown to be highly influenced by the choice of attention layers\(Bicket al\.,[2025b](https://arxiv.org/html/2603.26556#bib.bib18)\)\)\.
- •Fineweb\-EDU:General language modeling perplexity on educational web text\.

Table[9](https://arxiv.org/html/2603.26556#A5.T9)lists selected indices\. Table[10](https://arxiv.org/html/2603.26556#A5.T10)reports downstream performance\. Greedy Aggregate achieves the lowest evaluation loss \(Figure[5](https://arxiv.org/html/2603.26556#A5.F5)\) but underperforms on downstream tasks\. Beam Add achieves the best balance of reasoning and knowledge \(Table[10](https://arxiv.org/html/2603.26556#A5.T10)\), outperforming both Beam Replace and Greedy Learned\. We therefore use the attention layers selected by the Beam Search Add method in all our remaining experiments\.

Interestingly, the method with the lowest in\-pipeline evaluation loss \(Greedy Learned / GA\-S2\) is not the one with the strongest downstream performance \(Table[10](https://arxiv.org/html/2603.26556#A5.T10)and Figure[5](https://arxiv.org/html/2603.26556#A5.F5)\), which is itself an instance of the perplexity and generation disagreement this paper documents, here surfacing inside the layer\-selection step rather than at final evaluation\. We hypothesise that two choices in our procedure compensate for using a weaker per\-candidate metric than KL\. Beam search explores beyond locally greedy choices, and the three\-probe ensemble averages across capacity\-shaped, retrieval\-shaped, and knowledge\-shaped signals\. Together they avoid the∼100×\{\\sim\}100\\timessearch\-stage compute overhead of GA\-S2’s per\-layer retraining without degrading selection quality on aggregate downstream\. With PostNAS\-level compute a direct downstream\-task signal would still be preferable\. In the budget regime our pipeline targets, beam search over probe\-PPL is the most compute\-efficient point on the spectrum that does not degrade selection quality\.

Table 9:Attention layer selection for different methods and sparsity levels \(Qwen3\-0\.6B, 28 layers\)\. Layer indices are 0\-indexed\.Table 10:Downstream performance after Stage 3b for the selected attention\-layer configurations\.![Refer to caption](https://arxiv.org/html/2603.26556v2/x5.png)Figure 5:Evaluation loss curves for attention layer selection methods during Stage 3b\. Greedy Learned achieves the lowest loss but Beam Add yields better downstream performance\.

## Appendix FArchitecture Design Choices

We conduct an ablation study to analyze the impact of key architectural choices on the distilled hybrid model\. Specifically, we investigate: \(i\) the output gate activation function \(SiLU vs\. Sigmoid\), \(ii\) the presence of a post\-SSM normalization layer, and \(iii\) the initialization strategy for the mixer parameters \(VO initialization from the teacher’s value and output projections, QKVO initialization from all attention projections, or random initialization\)\. For each configuration, we run the three\-stage distillation pipeline on the FineWeb\-Edu dataset \(for stage 3 we do not train on the instruct data, but only the 100M tokens on the FineWeb\-Edu\) and report the evaluation perplexity at the end of training\.

Table 11:Ablation study on model architecture details\. We report evaluation perplexity on FineWeb\-Edu after completing Stages 1–3 of distillation\.We find Stage 1 to be critical for stable and strong distillation: skipping Stage 1 significantly degrades performance, even when initializing from the teacher, while fully random initialization remains competitive but worse overall, consistent with observations in prior attention\-to\-linear distillation work\(Goldsteinet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib17)\)\. Using a Sigmoid gate underperforms SiLU\. We hypothesize this is partly because Sigmoid does not admit an identity\-style initialization for the gate during distillation, which may hinder optimization\. Adding a post\-SSM normalization layer is slightly worse in perplexity and, in our downstream evaluation, shows a larger drop and we therefore omit it \(and we also observed it to be substantially worse in our Mamba\-based hybrids\)\. Finally, QKVO initialization performs similarly \(slightly better for the KDA models\) than VO\-only initialization, however, in early Mamba\-based experiments the gap was larger in the opposite direction, so we default to VO initialization for a fairer comparison across architectures\.

## Appendix GTop\-K Sparsification and CE Loss Weight

We ablate top\-KKsparsification of the KD target distribution and the cross\-entropy loss weightαCE\\alpha\_\{\\text\{CE\}\}\.

Table 12:Effect of top\-KKsparsification of the KD loss during Stage 3b with frozen attention\. All runs use 7 attention layers,freeze\_attn=True, KD with completion\-only masking, and 250M tokens\. Generation\-based evaluation \(EvalScope\)\. Best result per column inbold\.Table[12](https://arxiv.org/html/2603.26556#A7.T12)shows that restricting the KD loss to the teacher’s top\-KKtokens does not improve performance when attention layers are frozen\. The full\-vocabulary baseline achieves the highest average score across all eleven benchmarks, with the largest advantages on knowledge\-intensive tasks \(C\-Eval, MMLU\-Redux, GSM8K\)\. Commonsense benchmarks show no systematic trend acrossKKvalues\. We therefore use full\-vocabulary KD for all main experiments\.

Table 13:Effect of cross\-entropy loss weightαCE\\alpha\_\{\\text\{CE\}\}in the KD objective \(ℒ=ℒKL\+αCE​ℒCE\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{KL\}\}\+\\alpha\_\{\\text\{CE\}\}\\mathcal\{L\}\_\{\\text\{CE\}\}\)\. All runs use 5 attention layers,freeze\_attn=False, 500M tokens\.Adding any cross\-entropy weight substantially degrades knowledge benchmarks \(Table[13](https://arxiv.org/html/2603.26556#A7.T13)\): evenαCE=0\.01\\alpha\_\{\\text\{CE\}\}=0\.01drops C\-Eval by 8\.4 points and MMLU\-R by 1\.4 points, while higher values cause catastrophic drops\. We accordingly setαCE=0\.0\\alpha\_\{\\text\{CE\}\}=0\.0for all main experiments\.

## Appendix HLearning Rate Ablation

Several generation benchmarks \(HellaSwag, ARC\-E, MMLU\-Redux\) degrade as Stage 3b training continues at the default LR==2\.5e\-5 \(Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\)\. This appendix and the next isolate the cause\. Here we test whether a smaller learning rate \(LR==1e\-5\) mitigates the effect by slowing the update dynamics\. Appendix[I](https://arxiv.org/html/2603.26556#A9)instead tests whether the MLP parameters are responsible\. All other settings are held fixed \(KD with completion\-only masking\) and we vary the training token budget\.

Table 14:Learning rate ablation for Stage 3b training with KD loss \(KD\) and completion\-only masking \(C\)\. Best student result per column and evaluation method inbold\.Table[14](https://arxiv.org/html/2603.26556#A8.T14)shows that a modest learning\-rate reduction does not remove the downward trend on several generation metrics with additional Stage 3b tokens: ARC\-E and HellaSwag decrease as training continues under both LR settings\. In contrast, it slows improvement on knowledge\-heavy benchmarks at a fixed token budget: for example, C\-Eval at 250M tokens is substantially lower at LR==1e\-5 than at LR==2\.5e\-5\. Consequently, we stay with the chosen LR\. Note that PPL\-based evaluation changes little with learning rate, while differences in generation\-based metrics are substantial\.

## Appendix IFrozen MLP Ablation

Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)shows that several generation benchmarks degrade with extended Stage 3b training and the learning\-rate ablation \(Appendix[H](https://arxiv.org/html/2603.26556#A8)\) rules out learning rate as the primary explanation\. Here we test whether the degradation is driven by corruption of factual knowledge stored in MLP parameters by freezing all MLP layers during Stage 3b while keeping all other settings fixed \(KD with completion\-only masking\)\. Unfrozen KD baselines from Table[17](https://arxiv.org/html/2603.26556#A11.T17)are reproduced for comparison\.

Table 15:Frozen MLP ablation across training durations\. Generation\-based \(EvalScope, left\) and perplexity\-based \(LM\-eval, right\) results for Stage 3b with frozen vs\. unfrozen MLP parameters under KD with completion\-only masking\. Unfrozen rows are reproduced from Table[17](https://arxiv.org/html/2603.26556#A11.T17)\. HS generation is unavailable for frozen runs101010Due to a proxy issue that prevented re\-downloading the HellaSwag dataset, not an indication of any problem with the results or the benchmark\.\. Best student result per column inbold\.Generation\-based \(EvalScope\)Perplexity\-based \(LM\-eval\)MLPTokensWGHSARC\-EARC\-CC\-EvalMMLU\-RCMMLUGSM8KHEvalBBHIFEvalHSPIQALAM\.↓\\downarrowMMLUTeacher \(Qwen3\-0\.6B\)51\.239\.070\.855\.042\.946\.145\.257\.535\.327\.457\.147\.365\.924\.740\.2Unfrozen250M48\.935\.465\.648\.837\.941\.636\.235\.818\.320\.544\.744\.965\.240\.637\.7Unfrozen500M48\.133\.964\.150\.039\.940\.738\.239\.821\.922\.447\.545\.065\.136\.539\.2Unfrozen750M49\.432\.362\.347\.041\.039\.039\.237\.925\.022\.449\.045\.065\.037\.139\.1Frozen100M50\.7–62\.247\.429\.937\.832\.825\.112\.215\.638\.644\.767\.036\.939\.7Frozen250M51\.0–63\.047\.131\.438\.631\.121\.818\.316\.440\.844\.866\.939\.440\.1Frozen500M49\.8–61\.645\.932\.137\.527\.418\.717\.116\.439\.944\.766\.838\.840\.2Frozen750M50\.8–60\.845\.134\.538\.324\.617\.218\.315\.038\.544\.966\.439\.441\.4

Freezing the MLPs does not eliminate the degradation on the affected generation benchmarks as Stage 3b training is extended \(Table[10](https://arxiv.org/html/2603.26556#footnote10)\)\. Moreover, the unfrozen baselines generally retain reasoning performance better, suggesting that MLP adaptation is helpful rather than the root cause of the decline\. Knowledge\-heavy benchmarks exhibit mixed sensitivity to freezing: some remain competitive under frozen MLPs, indicating that mixer and routing updates can absorb part of the distillation signal, while others degrade, consistent with the need for MLP plasticity for broader knowledge transfer \(especially cross\-lingual\)\. Finally, perplexity\-based metrics change little across the duration sweep, again highlighting the gap between PPL\-based and generation\-based evaluation\. These results argue against MLP knowledge corruption as the primary driver of Stage 3b degradation and support keeping MLPs trainable during Stage 3b\.

## Appendix JStage 3a Budget Ablation

This section details the Stage 3a pretraining\-distillation budget summarized in Section[5\.2](https://arxiv.org/html/2603.26556#S5.SS2)\. The Stage 3b training\-duration pattern is shown alongside the extended Stage 3b results in Appendix[K](https://arxiv.org/html/2603.26556#A11), and the dataset ablation \(Table[3](https://arxiv.org/html/2603.26556#S5.T3)\) is in the main text\.

We vary the Stage 3a token budget while holding the instruction\-tuning stage \(Stage 3b\) fixed, so the student always learns the chat format and basic instruction\-following behavior\.111111These experiments use an earlier configuration with 5 attention layers\. We later adopted 7 attention layers to match the 1:3 attention\-to\-SSM ratio used in other hybrid models\(Teamet al\.,[2025](https://arxiv.org/html/2603.26556#bib.bib6)\)\. Since this ablation compares relative rankings of configurations rather than absolute scores, the conclusions transfer to the 7\-layer setup\.Increasing the budget monotonically reduces held\-out KL divergence \(Figure[6](https://arxiv.org/html/2603.26556#A10.F6)\), indicating better distributional alignment with the teacher\. This alignment does not translate proportionally into stronger downstream generation after Stage 3b \(Table[16](https://arxiv.org/html/2603.26556#A10.T16)\)\. PIQA and MMLU are essentially unchanged across Stage 3a budgets, whereas ARC \(Easy/Challenge\) and WinoGrande improve more noticeably\. Several perplexity\-based benchmarks appear already close to the teacher after a short Stage 3a \(e\.g\. PIQA\), but LAMBADA reveals a substantial remaining perplexity gap that narrows only gradually with more Stage 3a training\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x6.png)Figure 6:Held\-out KL loss during Stage 3b for different Stage 3a token budgets\.Table 16:Stage 3a dataset size ablation: downstream results after Stage 3a \(top, perplexity\-based\) and after Stage 3b \(bottom, generation\-based\)\. We report accuracy for all tasks except LAMBADA, where we report perplexity \(↓\\downarrow\)\.CheckpointWG↑\\uparrowHS↑\\uparrowLAMBADA↓\\downarrowARC\-E↑\\uparrowARC\-C↑\\uparrowPIQA↑\\uparrowMMLU↑\\uparrowStage 3aStage 3bTeacher \(Qwen3\-0\.6B\)55\.547\.324\.755\.634\.265\.940\.2After Stage 3a \(LM Evaluation Harness, perplexity\-based\)100M–54\.344\.338\.853\.832\.065\.938\.3250M–54\.944\.732\.850\.231\.566\.430\.2500M–54\.745\.031\.754\.932\.365\.936\.0After Stage 3b \(EvalScope, generation\-based\)100M250M31\.118\.943\.646\.635\.365\.230\.6250M250M30\.317\.842\.851\.138\.165\.130500M250M36\.118\.641\.352\.438\.164\.730\.2

## Appendix KExtended Stage 3b Results

Table[17](https://arxiv.org/html/2603.26556#A11.T17)presents the complete results for all Stage 3b configurations across 11 generation\-based and 7 perplexity\-based benchmarks, the full source for the generation\-only summary visualised in Figure[7](https://arxiv.org/html/2603.26556#A11.F7)and the protocol comparison in Figure[3](https://arxiv.org/html/2603.26556#S5.F3)\. KD with completion\-only masking achieves the best student result on every generation benchmark\. For generation columns, greedy decoding results are shown in parentheses\. We report them to confirm that the recipe ranking is not an artefact of sampling variance\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x7.png)Figure 7:Stage 3b ablation, generation\-based evaluation\. Each panel shows accuracy on one of ten generation benchmarks as a function of the Stage 3b token budget\. Colour encodes the training objective \(KD red, SFT blue\) and line style encodes the loss\-masking strategy \(P\+C dashed, C solid\)\. The dotted horizontal line is the Qwen3\-0\.6B teacher reference\. Full numbers, perplexity\-based scores, BBH, and greedy\-decoding variants are in Table[17](https://arxiv.org/html/2603.26556#A11.T17)\.Table 17:Extended Stage 3b ablation\. Generation\-based results \(EvalScope, left\) with greedy decoding inparentheses; perplexity\-based results \(LM\-eval, right\)\. Best student result per column inbold\.
## Appendix LCombined Stage Mixing

The four\-stage sequential pipeline \(Section[3\.2](https://arxiv.org/html/2603.26556#S3.SS2)\) trains on pretraining and instruction data in separate stages, which risks Stage 3b overwriting the general language representations acquired during Stage 3a, a pattern visible in the benchmark\-specific degradation reported in Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)\. We investigate whether merging Stages 3a and 3b into a single training phase with interleaved pretraining and instruction data can circumvent this overwrite pattern while simplifying the pipeline\.

Table 18:Combined stage mixing results\. Sequential baselines \(top\) use the standard pipeline: 500M tokens of Stage 3a pretraining KD followed by Stage 3b instruction KD with completion\-only masking and frozen attention\. Combined stage runs \(bottom\) interleave pretraining and instruction data in a single training stage after Stages 1–2\. Ratios denote pretrain : instruct data proportions;⋆indicates balanced FineWeb \(50:50 EN:CN rather than the natural 29:71 ratio\)\. Generation\-based evaluation \(EvalScope\)\. Best student result per column inbold\.Table[18](https://arxiv.org/html/2603.26556#A12.T18)compares the sequential pipeline against ten combined\-stage configurations varying in total token budget, pretrain\-to\-instruct ratio, and pretraining data composition\. Several patterns emerge\. First, at matched compute budgets, the sequential pipeline outperforms combined mixing, with the largest gaps on reasoning and code generation benchmarks; the dedicated pretraining phase in Stage 3a appears to provide a stronger representational foundation than interleaved pretraining can replicate at the same scale\. Second, among combined configurations, instruction\-heavy ratios consistently outperform pretrain\-heavy ones: allocating roughly 85% of tokens to instruction data and retaining only a small pretraining fraction yields the best balance across knowledge, reasoning, and instruction\-following tasks\. Third, balancing the English\-to\-Chinese ratio in the pretraining subset \(50:50 rather than the natural distribution\) consistently improves Chinese knowledge benchmarks without harming English performance, confirming that data composition within the pretraining fraction matters even when that fraction is small\. Fourth, unlike the sequential pipeline where extending Stage 3b degrades several benchmarks \(Table[17](https://arxiv.org/html/2603.26556#A11.T17)\), combined mixing scales gracefully with additional compute, showing steady improvement from medium to large token budgets without the characteristic degradation pattern\. Fifth, at sufficient total compute, the best combined configuration approaches or matches the sequential pipeline’s best results across most benchmarks and surpasses it on language modeling and several reasoning tasks, suggesting that combined mixing may become the preferred strategy in higher\-compute regimes where the benefits of graceful scaling outweigh the sequential pipeline’s advantage at smaller budgets\.

## Appendix MStage 3b Recipe Ablations at 1\.7B

To test whether the Stage 3b recipe conclusions extend beyond the 0\.6B scale used for the systematic study, we repeat the three highest\-leverage ablations with a Qwen3\-1\.7B teacher\. We distil into the same Hybrid\-KDA architecture and keep every training setting other than the ablated axis fixed\. Stage 3b uses 500M tokens, as at 0\.6B, and all numbers are averaged over two seeds\. Table[19](https://arxiv.org/html/2603.26556#A13.T19)reports the training\-objective and loss\-masking comparisons, and Table[20](https://arxiv.org/html/2603.26556#A13.T20)reports the attention\-freezing comparison\.

Table 19:Stage 3b recipe ablation at 1\.7B \(training objective and loss masking\), Hybrid\-KDA student distilled from a Qwen3\-1\.7B teacher, 500M Stage 3b tokens, frozen attention, seed\-averaged\. Generation\-based results \(EvalScope, left\) and perplexity\-based results \(LM\-eval, right\)\. Masking: P\+C = full\-sequence \(prompt \+ completion\); C = completion\-only\. Best student result per column inbold\.Generation\-based \(EvalScope\)Perplexity\-based \(LM\-eval\)LossMaskTokensC\-EvalMMLU\-RCMMLUWGHSARC\-EARC\-CBBHGSM8KHEvalIFEvalWGHSARC\-EARC\-CPIQALAM\.↓\\downarrowMMLUTeacher \(Qwen3\-1\.7B\)62\.866\.962\.054\.259\.884\.873\.634\.380\.958\.566\.161\.660\.569\.742\.872\.612\.255\.4SFTC500M33\.145\.746\.149\.531\.474\.860\.69\.554\.00\.036\.261\.260\.363\.938\.771\.411\.651\.5KDC500M56\.555\.054\.033\.223\.568\.558\.221\.260\.134\.858\.660\.958\.268\.541\.371\.613\.453\.0KDP\+C500M56\.552\.254\.851\.232\.380\.568\.119\.658\.07\.356\.460\.758\.968\.041\.772\.112\.953\.4

The objective conclusion reproduces\. KD leads SFT on generation by 7\.5pp on average, with the largest gaps on instruction following, code, and Chinese knowledge, while SFT attains the lower perplexity on the language\-modeling benchmarks \(LAMBADA, HellaSwag, WinoGrande\)\. The protocol disagreement that motivates the paper is therefore present at 1\.7B in the same direction as at 0\.6B\.

The masking conclusion reproduces only in part\. Completion\-only retains a clear advantage on instruction following, code, and reasoning, but full\-sequence masking overtakes it on the commonsense multiple\-choice benchmarks \(WinoGrande, HellaSwag, ARC\) and so attains a slightly higher generation average\. On those benchmarks the completion\-only student generates answers at close to random accuracy, near 33 on the binary WinoGrande and near 23 on the four\-way HellaSwag, while scoring normally under perplexity\-based evaluation\. This pattern matches the multiple\-choice extraction failures characterised in Appendix[N](https://arxiv.org/html/2603.26556#A14)rather than a loss of commonsense ability, and it makes completion\-only masking the most evaluation\-sensitive of the three recipe choices at scale\.

Table 20:Effect of freezing attention layers during Stage 3b at 1\.7B\. Both runs use KD with completion\-only masking and 500M Stage 3b tokens, differing only in whether the attention layers are frozen, seed\-averaged\. Benchmarks match Table[19](https://arxiv.org/html/2603.26556#A13.T19): generation\-based \(EvalScope, left\) and perplexity\-based \(LM\-eval, right\)\. Best result per column inbold\.Generation\-based \(EvalScope\)Perplexity\-based \(LM\-eval\)Frozen?C\-EvalMMLU\-RCMMLUWGHSARC\-EARC\-CBBHGSM8KHEvalIFEvalWGHSARC\-EARC\-CPIQALAM\.↓\\downarrowMMLUNo56\.854\.955\.032\.118\.063\.753\.221\.958\.735\.158\.761\.257\.969\.042\.271\.512\.953\.3Yes56\.555\.054\.033\.223\.568\.558\.221\.260\.134\.858\.660\.958\.268\.541\.371\.613\.453\.0

The freezing conclusion reproduces\. The gains are large on the commonsense multiple\-choice benchmarks \(HellaSwag and ARC, around 5pp each in Table[20](https://arxiv.org/html/2603.26556#A13.T20)\)\. On the remaining benchmarks the two runs lie within their seed\-to\-seed variation, and the few on which the unfrozen run scores higher do so by margins smaller than that variation, so freezing never significantly underperforms\. Averaged over all eleven generation benchmarks freezing improves by 1\.4pp, rising to 2\.7pp over the knowledge and commonsense subset \(C\-Eval, MMLU\-R, WinoGrande, HellaSwag, ARC\)\. Perplexity\-based scores move by less than half a point and, if anything, slightly favour the unfrozen run\. This is the same asymmetry reported at 0\.6B in Table[2](https://arxiv.org/html/2603.26556#S5.T2), where freezing improves generation while leaving perplexity\-based evaluation almost unchanged\.

## Appendix NQualitative KD vs\. SFT Failure Patterns

The Stage\-3b ablation in Section[5\.1](https://arxiv.org/html/2603.26556#S5.SS1)reports a consistent advantage for knowledge distillation over supervised fine\-tuning on generation\-based evaluation, with the opposite pattern on several perplexity\-based benchmarks\. To complement the aggregate numbers, we inspected the outputs of two same\-recipe Hybrid\-KDA 0\.6B Stage\-3b checkpoints \(completion\-only, 500M tokens\) that differ only in the loss type, on two unrelated tasks \(HellaSwag and GSM8K\)\. Both tasks show an identical\+7\.2\+7\.2pp accuracy gap in KD’s favour \(N=1000N=1000andN=1319N=1319respectively, greedy bf16 generation, seed=1=1\)\. The output traces reveal four behaviours that recur in the SFT generations but are essentially absent from the KD generations on the same prompts\. Table[21](https://arxiv.org/html/2603.26556#A14.T21)summarises the rates\.

##### Template overfitting\.

SFT wraps100%100\\%of its outputs in literal<answer\>\.\.\.</answer\>tags on both tasks, including on GSM8K where the 4\-shot prompt demonstrates the plain “Reasoning: … ANSWER:\\\\backslashboxed\{N\}” format fromlm\-evaluation\-harness\. KD wraps none\. The tags appear in part of the upstream Stage\-3b instruct corpus \(AM\-Qwen3\-Distilled\); SFT has learned them as a literal output prefix and suffix, whereas KD trains against the teacher’s full output distribution rather than the labels themselves and the Qwen3 teacher does not emit the tag\.

##### Multiple\-choice positional collapse\.

On HellaSwag, SFT picks option “A” on79\.5%79\.5\\%of all questions \(against KD’s54%54\\%\), and on86%86\\%of gold\-A questions specifically\. Conditional accuracy on the non\-A buckets collapses to5\.1%5\.1\\%on gold\-D and9\.1%9\.1\\%on gold\-B for SFT, while KD retains discrimination across the four options\.

##### Wordier and less accurate chains of thought\.

On GSM8K, SFT generates a mean of404\.3404\.3output tokens against KD’s302\.5302\.5, with twice the55\-gram local repetition rate and4\.7×4\.7\\timesthe rate of outputs that fail to emit\\\\backslashboxed\{⋅\\cdot\}\. Inspection of per\-prompt disagreement cases \(KD\-correct, SFT\-wrong\) shows the SFT chains restating the problem with extra ceremony, introducing ill\-defined intermediate variables, and committing arithmetic errors that KD avoids\.

##### Few\-shot recency mimicry\.

On the 4\-shot GSM8K prompt, SFT regurgitates tokens lifted directly from the recency\-most demonstration into answers to unrelated questions roughly three times as often as KD \(strict\-trigger rate3\.0%3\.0\\%vs\.1\.0%1\.0\\%\), leading to incorrect answers\. Re\-running the same benchmark with no in\-context demonstrations \(few\_shot\_num=0\) drops the mimicry rate to zero on both models and shrinks the KD\-vs\-SFT accuracy gap from\+7\.2\+7\.2to\+3\.4\+3\.4pp, confirming that the demonstrations themselves are the trigger and that SFT is the more susceptible of the two models to anchoring on irrelevant phrasing in the prompt\.

Table 21:Rates of the four SFT\-specific behaviours described in this appendix\. Both checkpoints are Hybrid\-KDA 0\.6B Stage\-3b \(completion\-only, 500M tokens\), identical generation config, differing only in the Stage\-3b loss type\.These four behaviours are consistent with a single hypothesised mechanism: KL distillation transmits properties of the teacher’s full output distribution to the student, including its calibration over option tokens on multiple choice\(Zhenget al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib66)\)and its resistance to copying labels from in\-context demonstrations\(Aliet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib67)\), whereas supervised fine\-tuning against arg\-max labels discards those properties\. Instruction\-tuned models additionally remain sensitive to the surface formatting of few\-shot prompts\(Sclaret al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib68)\), and SFT is documented to emit training templates verbatim when the instruction\-template diversity of the training set is low\(Linet al\.,[2025a](https://arxiv.org/html/2603.26556#bib.bib69)\), consistent with the<answer\>\.\.\.</answer\>wrap we observe\. The wordier\-but\-less\-accurate chain pattern on GSM8K is reminiscent of the “flaw repetition” failure modeYaoet al\.\([2025](https://arxiv.org/html/2603.26556#bib.bib70)\)document under SFT\-only post\-training of reasoning models, alongside their finding that SFT\-only pipelines degrade calibration relative to the base distribution \(their analysis is at77B–7070B reasoning\-model scale with reasoning\-specific recipes, so the parallel is qualitative\)\.

### Representative example traces

The four behaviours described above are illustrated below on four concrete prompts\. Each block shows the question \(4\-shot in\-context demonstrations stripped for readability\), the KD checkpoint’s full output, and the SFT checkpoint’s output verbatim\. Outputs are shown in monospace to preserve the literal characters emitted by the model, including the spurious<answer\>\.\.\.</answer\>wrap\.

##### Example 1: Template wrap \(GSM8K, gold=8=8\)\.

Q:Brandon’s iPhone is four times as old as Ben’s iPhone\. Ben’s iPhone is two times older than Suzy’s iPhone\. If Suzy’s iPhone is 1 year old, how old is Brandon’s iPhone?

KD output:

```
- Suzy’s age: 1 year.
- Ben’s age: 2 x 1 = 2 years.
- Brandon’s age: 4 x 2 = 8 years.
\boxed{8}
```

SFT output:

```
<answer>Brandon’s iPhone is four times as old as Ben’s iPhone.
Let x = age of Ben’s iPhone, so Brandon’s iPhone is 4x years old.
Suzy’s iPhone is 1 year old.
Since Suzy’s iPhone is two times older than Ben’s iPhone:
    2 * x = 1   =>   x = 1/2
Thus Ben’s iPhone is 1/2 years old, and Brandon’s is 4 * 1/2 = 2.
The final answer is \boxed{2}.</answer>
```

SFT wraps every output in literal<answer\>\.\.\.</answer\>tags and additionally inverts the “two times older than” relation\. KD never produces the wrap and computes the relation correctly\.

##### Example 2: Multiple\-choice positional collapse \(HellaSwag, gold==D, activity “Clean and jerk”\)\.

Context:A lady walks to a barbell\. She bends down and grabs the pole\. The lady…

Candidate continuations:

- •A\.swings and lands in her arms\.
- •B\.pulls the barbell forward\.
- •C\.pulls a rope attached to the barbell\.
- •D\.stands and lifts the weight over her head\.

KD answer:D \(correct\)\.SFT answer:A\.

Even though option A is semantically incongruous with a clean\-and\-jerk lift, SFT collapses to A; KD with the same base recipe correctly identifies D\.

##### Example 3: Wordier but less accurate chain of thought \(GSM8K, gold=50=50\)\.

Q:A car is on a road trip and drives 60 mph for 2 hours, and then 30 mph for 1 hour\. What is the car’s average speed in mph during this trip?

KD output:

```
1. Distance in first 2 hours: 60 x 2 = 120 miles
2. Distance in next 1 hour:   30 x 1 = 30 miles
3. Total distance: 120 + 30 = 150 miles
4. Total time:     2 + 1 = 3 hours
5. Average speed:  150 / 3 = 50 mph
\boxed{50}
```

SFT output:

```
<answer>To determine the car’s average speed during the trip ...
### Step 1: Understand the Given Information
### Step 2: Calculate the Distance Traveled in Each Segment
  First segment:  60 mph * 2 h = 120 miles
  Second segment: 30 mph * 1 h = 30  miles
### Step 3: Total Distance Traveled
  120 + 30 = 150 miles
### Step 4: Calculate the Average Speed
  Average Speed = 150 miles / 2 hours = 75 mph
### Final Answer
\boxed{75}</answer>
```

SFT produces a longer, more heavily formatted answer but divides total distance by the duration of one segment \(22h\) rather than total trip time \(33h\)\. Surface structure is not understanding\.

##### Example 4: Few\-shot recency mimicry \(GSM8K, gold=32=32\)\.

Q:Anakin and Locsin went to the beach today\. Anakin caught 10 starfish, 6 sea horses, and 3 clownfish\. Locsin caught 5 fewer starfish than Anakin, 3 fewer sea horses than Anakin, and 2 more clownfish than Anakin\. How many fish were they able to catch in total?

KD output:

```
Anakin: 10 + 6 + 3 = 19 fish.
Locsin: (10-5) + (6-3) + (3+2) = 5 + 3 + 5 = 13 fish.
Total: 19 + 13 = 32.   \boxed{32}
```

SFT output \(trimmed; the relevant intrusion is at steps 5–6\):

```
<answer>**Step-by-Step Explanation:**
1. Anakin’s Catch: 10 starfish, 6 sea horses, 3 clownfish.
2. Locsin’s Catch: 10-5=5 starfish, 6-3=3 sea horses, 3+2=5 clownfish.
3. Total Catch by Anakin: 10 + 6 + 3 = 19 fish.
4. Total Catch by Locsin: 5 + 3 + 5 = 13 fish.
...
5. Remaining Pages for Tomorrow:
   - Total pages: 120
   - Pages already read: 19
   - Remaining pages: 120 - 19 = 101
   - Pages to read tomorrow: 101 / 2 = 50.5
6. Final Answer: 50.5 ~= 50 pages.
**Answer:** \boxed{50}</answer>
```

SFT correctly assembles the per\-person fish counts in steps 1–4 and then silently switches into the “120\-page book / half tomorrow” template lifted from the recency\-most 4\-shot demonstration \(Julie reading a 120\-page book\), abandons the actual question, and reports5050\. The intrusion is the prompt’s fault rather than the training set’s: atfew\_shot\_num=0the mimicry rate falls to zero on both models \(Table[21](https://arxiv.org/html/2603.26556#A14.T21)\)\.

## Appendix OPerplexity\-Based Architecture Comparison

Table[4](https://arxiv.org/html/2603.26556#S5.T4)in the main text reports generation\-based results\. Here we present the corresponding perplexity\-based \(LM Evaluation Harness\) results for the same eight architectures\.

Table 22:Architecture comparison under perplexity\-based evaluation \(LM Evaluation Harness\)\. Same models as Table[4](https://arxiv.org/html/2603.26556#S5.T4)\. We report accuracy \(%\) for all tasks except LAMBADA, where we report perplexity \(↓\\downarrow\)\. Best student result per column inbold\.Architecture rankings under perplexity\-based evaluation are broadly consistent with generation\-based results: Hybrid KDA leads on most metrics and Pure Mamba is weakest\. However, inter\-model differences are markedly compressed: PPL\-based scoring narrows the architecture gaps substantially compared to generation\-based evaluation, with the exception of Pure Mamba’s catastrophic LAMBADA perplexity\. This reinforces the paper’s central finding that perplexity\-based metrics underestimate quality differences between architectures\.

## Appendix PMaximum Throughput Under GPU Saturation

The batch\-size\-1 measurements in Section[5\.3\.1](https://arxiv.org/html/2603.26556#S5.SS3.SSS1)do not capture the primary deployment benefit of reduced KV cache memory: fitting larger batches\. Figure[8](https://arxiv.org/html/2603.26556#A16.F8)reports peak throughput at each context length by sweeping batch sizes up to OOM\.

![Refer to caption](https://arxiv.org/html/2603.26556v2/x8.png)Figure 8:\(a\)Theoretical cache memory vs\. sequence length\. Attention cache grows linearly with context; SSM state is constant\.\(b\)Peak throughput \(sweeping batch size to OOM\) at four context lengths\. Annotations show relative throughput vs\. the Teacher\. At 32K\+ tokens, SSM variants achieve 1\.9–2\.5×\\timesthe teacher’s peak throughput by fitting larger batches\.At short contexts \(2K\), KV cache memory is small relative to model weights, so all architectures achieve similar peak throughput\. As context length grows, KV cache dominates memory allocation for the Transformer, limiting its maximum batch size\. The SSM and hybrid variants, with∼\{\\sim\}75% smaller cache, sustain larger batches and achieve up to 5×\\timeshigher peak throughput at 128K tokens\.

## Appendix QLong\-Context Quality

The efficiency gains reported in Section[5\.3\.1](https://arxiv.org/html/2603.26556#S5.SS3.SSS1)are meaningful only if hybrid models preserve functional quality at long contexts\. We evaluate on LongBench\(Baiet al\.,[2023](https://arxiv.org/html/2603.26556#bib.bib1)\)\(real\-world QA and summarization, 8K–22K tokens\) and RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2603.26556#bib.bib2)\)\(synthetic needle\-in\-a\-haystack retrieval, 4K–128K tokens\)\. All students were distilled on sequences of at most 4K tokens with no dedicated long\-context extension stage; the gap to the teacher reported below should be read in that light \(see Section[6](https://arxiv.org/html/2603.26556#S6)\)\.

Table 23:LongBench results \(F1 or ROUGE,×\\times100; contexts 8K–22K tokens\)\. DuR = DuReader, HQA = HotpotQA, MuS = MuSiQue, NQA = NarrativeQA, QMS = QMSum, TQA = TriviaQA\. “Hybrid” models retain 7 attention layers; “Pure” models use 0\. Best student per column inbold\.On LongBench \(Table[23](https://arxiv.org/html/2603.26556#A17.T23)\), all students fall substantially short of the teacher\. The architecture hierarchy mirrors the short\-context results \(Table[4](https://arxiv.org/html/2603.26556#S5.T4)\): Pure Mamba collapses \(Avg 3\.0\), Pure KDA is moderate \(12\.6\), and both hybrid variants lead, with Hybrid KDA \(14\.5\) narrowly ahead of Hybrid Mamba \(14\.0\)\. Individual task rankings vary \(Hybrid Mamba leads on DuReader and TriviaQA, while Hybrid KDA is stronger on HotpotQA, MuSiQue, and NarrativeQA\), but the overall gap to the teacher remains large, underscoring that short\-context distillation alone does not fully transfer long\-range comprehension\.

Table 24:RULER NIAH single\-needle retrieval accuracy \(%\) across context lengths\. The three Single variants differ in haystack complexity and needle value type:Single\-1embeds a numeric needle in a short repeated sentence,Single\-2embeds a numeric needle in natural prose \(Paul Graham essays\), andSingle\-3embeds a UUID string in the same natural prose\. “Hybrid” models retain 7 attention layers; “Pure” models use 0\. Best student per column inbold\.The NIAH results \(Table[24](https://arxiv.org/html/2603.26556#A17.T24)\) provide a finer\-grained view\. Pure Mamba fails completely across all needle types and context lengths \(0\.0%\), confirming that Mamba2 layers lack the precise token\-level retrieval mechanism required for needle\-in\-a\-haystack tasks\. Pure KDA retains some retrieval ability on the simplest needle type \(Single\-1: 96\.4% at 4K\) but collapses to near\-zero on Single\-2 \(0\.6%\) and Single\-3 \(0\.0%\)\. The dramatic difference is explained by the task structure: in Single\-1, the haystack is a short sentence repeated many times, making any inserted needle highly salient against a trivially predictable background; in Single\-2/3, the needle must be located within coherent natural prose, requiring precise positional attention that linear sequence mixers fundamentally lack\. Single\-3 adds further difficulty by requiring exact reproduction of 36\-character UUID strings rather than short numbers, a task that demands character\-level fidelity from the retrieval mechanism\. This progression exposes a fundamental limitation of KDA layers: their success on Single\-1 reflects pattern\-matching against a predictable context rather than genuine retrieval capability\. Both hybrid models perform dramatically better, with near\-perfect accuracy on Single\-1 and Single\-2 at short contexts\. Hybrid Mamba slightly outperforms Hybrid KDA at short contexts on the easiest tasks, but Hybrid KDA degrades more gracefully at longer contexts, maintaining 92\.6% at 64K on Single\-1 versus 75\.8% for Hybrid Mamba\. On the hardest needle type \(Single\-3\), both hybrid models struggle beyond 16K tokens, with Hybrid KDA retaining a slight edge at 8K–32K\.

Taken together, the long\-context results confirm that hybrid architectures enable meaningful efficiency gains \(Section[5\.3\.1](https://arxiv.org/html/2603.26556#S5.SS3.SSS1)\) while retaining basic retrieval capability\. Retained attention layers are essential for non\-trivial retrieval, with both pure architectures failing on harder needle types\. Among hybrid variants, KDA maintains a slight overall edge, particularly at longer contexts\. A substantial gap to the teacher persists at harder retrieval patterns and longer contexts; we discuss in Section[6](https://arxiv.org/html/2603.26556#S6)why this is expected given the absence of a dedicated long\-context training stage, and how concurrent work\(Chenet al\.,[2026](https://arxiv.org/html/2603.26556#bib.bib61); Teamet al\.,[2026](https://arxiv.org/html/2603.26556#bib.bib60)\)closes the gap with such a stage\.