A Study on Hidden Layer Distillation for Large Language Model Pre-Training

# A Study on Hidden Layer Distillation for Large Language Model Pre-Training
Source: [https://arxiv.org/html/2605.11513](https://arxiv.org/html/2605.11513)
Maxime Guigon Google DeepMind guimax@google\.com &Lucas Dixon Google DeepMind ldixon@google\.com &Michaël E\. Sander Google DeepMind michaelsander@google\.com

###### Abstract

Knowledge Distillation \(KD\) is a critical tool for training Large Language Models \(LLMs\), yet the majority of research focuses on approaches that rely solely on output logits, neglecting semantic information in the teacher’s intermediate representations\. While Hidden Layer Distillation \(HLD\) showed potential for encoder architectures, its application to decoder\-only pre\-training at scale remains largely unexplored\. Through compute\-controlled experiments, we benchmark HLD against logit\-based KD and self\-supervised baselines with Gemma3 3\.4B as teacher and 123M and 735M students trained on up to 168B tokens from the C4 dataset\. Our experiments show that HLD does not consistently outperform standard KD on downstream evaluation tasks\. Nevertheless, we show that HLD can yield a systematic perplexity gain over KD across all shared\-hyperparameter configurations, suggesting that a latent signal can be extracted, but a breakthrough may be needed for it to play a more significant role in LLM pre\-training\.

## 1Introduction

Large Language Models \(LLMs\) have achieved remarkable success across numerous Natural Language Processing \(NLP\) tasks, a phenomenon largely driven by the empirical observation of “scaling laws”, which link model size and number of training tokens to performance gains\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib20); Rosenfeldet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib21); Hestnesset al\.,[2017](https://arxiv.org/html/2605.11513#bib.bib22)\)\. However, this trend of continuously increasing scale introduces significant practical challenges, including soaring deployment costs, increased inference latency, and substantial energy consumption\. Reducing the computational footprint of LLMs, particularly during inference, has therefore become a critical area of research\.

This paper focuses on Knowledge Distillation \(KD\)\(Hintonet al\.,[2015](https://arxiv.org/html/2605.11513#bib.bib1)\), a robust and widely\-adopted paradigm for transferring knowledge from a large, powerful “teacher” model to a smaller, more efficient “student” model by training the latter to reproduce the teacher’s behavior\. KD has proven to be an enduring and highly effective technique for model compression across a decade of research, demonstrating utility in computer vision\(Ahnet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib23)\), speech processing\(Tan and Wang,[2021](https://arxiv.org/html/2605.11513#bib.bib24)\), and natural language processing for diverse architectures, including encoder\-only LLMs\(Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8)\)and various modern decoder\-only LLMs spanning both pre\-training and post\-training stages\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18); DeepSeek\-AI,[2024](https://arxiv.org/html/2605.11513#bib.bib19)\)\. Nevertheless, current distillation methods struggle to close the performance gap between students and teachers\(Penget al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib4); Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8); Wanget al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib9); Liuet al\.,[2024b](https://arxiv.org/html/2605.11513#bib.bib3)\), yet teacher models frequently exhibit internal redundancy\(Durraniet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib25); Dalviet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib26)\)that foresees lossless compression\. Furthermore, academic exploration of distillation techniques within the context of massive, cost\-intensive LLM pre\-training remains limited, mainly exploring classic, logits\-level KD\(Penget al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib4); Busbridgeet al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib5)\)\.

Hidden Layer Distillation \(HLD\), an encouraging early extension to KD introduced for CNNs in\(Romeroet al\.,[2015](https://arxiv.org/html/2605.11513#bib.bib2)\)\(FitNets\), which leverages a teacher’s hidden states, has been largely applied to smaller\-scale encoder\(\-decoder\) language models showing potential advantage compared to KD\(Jiaoet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib6); Lianget al\.,[2023b](https://arxiv.org/html/2605.11513#bib.bib7); Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8); Wanget al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib9),[2021](https://arxiv.org/html/2605.11513#bib.bib10); Yuet al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib11); Zuoet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib12); Dasgupta and Cohn,[2025](https://arxiv.org/html/2605.11513#bib.bib13); Sunet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib14); Lianget al\.,[2023a](https://arxiv.org/html/2605.11513#bib.bib15)\)\. Despite this potential, the application of HLD has never been scaled to the magnitude of current state\-of\-the\-art LLMs\.

Motivated by this, we propose an exploration of HLD specifically for decoder\-only models during the pre\-training phase\. Evaluating LLMs is sensitive to minor hyperparameter changes that can disproportionately affect final performances; we therefore establish rigorous baselines ensuring that any performance gains are attributable to the evaluated distillation methodology rather than artifacts\. Our experiments are built upon the open\-source NanoDo codebase\(Liuet al\.,[2024a](https://arxiv.org/html/2605.11513#bib.bib39)\), the C4 dataset\(Raffelet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib27)\), and open\-weight models from the Gemma family\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18)\)\.

TeacherEmbedLayer 1Layer 2Layerkk⋮\\vdotsLayerLLStudentEmbedLayer 1Layermm⋮\\vdotsLayerNNxxxxDe\-embedDe\-embedℒKD\\mathcal\{L\}\_\{\\mathrm\{KD\}\}Regressorℒemb\\mathcal\{L\}\_\{\\mathrm\{emb\}\}hkTh\_\{k\}^\{T\}hmSh\_\{m\}^\{S\}

Figure 1:Overview of HLD\. The student receives two training signals:ℒKD\\mathcal\{L\}\_\{\\mathrm\{KD\}\}matches the teacher’s output logits, whileℒemb\\mathcal\{L\}\_\{\\mathrm\{emb\}\}aligns a student hidden statehmSh\_\{m\}^\{S\}with a teacher hidden statehkTh\_\{k\}^\{T\}through a learned regressor\.More precisely, we make the following contributions:

- •We provide a systematic evaluation of HLD applied specifically to the pre\-training of decoder\-only LLMs by training 123M and 735M gemma students models\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18)\)on 8B and 168B tokens with a gemma 3\.4B teacher\.
- •We establish a rigorous evaluation framework that benchmarks methods based on strict computational equivalence \(FLOPs\) rather than token count, explicitly accounting for the non\-negligible cost of de\-embedding and loss computation in sub\-billion parameter students\.
- •We demonstrate through extensive compute\-matched experiments and evaluation over Wikitext 103, HellaSwag, WinoGrande, LAMBADA, PIQA, and ARC\-E that, compared to KD: - –\(i\) joint optimization of hidden layer alignment and logit distillation \(HLDC\) performs on par with KD;

- –\(ii\) sequential optimization \(HLDF\) yields modest improvements in C4 perplexity across all shared\-hyperparameter configurations, while performance across downstream evaluation remains comparable\.

## 2Background and Related Works

#### Transformers\.

The Transformer\(Vaswaniet al\.,[2017](https://arxiv.org/html/2605.11513#bib.bib37)\)is the main neural network architecture for the modeling of sequential data\. Current generative LLMs use the decoder\-only version\. Given an input stringss, the tokenizer produces a sequence of discrete token indices𝐓=\[t1,…,tn\]∈\{0,…,V−1\}n\\mathbf\{T\}=\[t\_\{1\},\\dots,t\_\{n\}\]\\in\\\{0,\\dots,V\-1\\\}^\{n\}, whereVVis the vocabulary size\. These are mapped to continuous embeddings𝐇0=\[𝐡10,…,𝐡n0\]∈ℝde​m​b×n\\mathbf\{H\}^\{0\}=\[\\mathbf\{h\}\_\{1\}^\{0\},\\dots,\\mathbf\{h\}\_\{n\}^\{0\}\]\\in\\mathbb\{R\}^\{d\_\{emb\}\\times n\}\. The backbone consists ofDDresidual layers, with update rule𝐇k\+1=𝐇k\+fl​a​y​e​rk\+1​\(𝐇k\)\\mathbf\{H\}^\{k\+1\}=\\mathbf\{H\}^\{k\}\+f\_\{layer\}^\{k\+1\}\(\\mathbf\{H\}^\{k\}\)\. Finally, a de\-embedding layer projects the final representations𝐇D\\mathbf\{H\}^\{D\}to logits𝐙∈ℝV×n\\mathbf\{Z\}\\in\\mathbb\{R\}^\{V\\times n\}\. The probability distribution for the next token is𝐩n=softmax​\(𝐳n/τ\)\\mathbf\{p\}\_\{n\}=\\text\{softmax\}\(\\mathbf\{z\}\_\{n\}/\\tau\), whereτ\\tauis the temperature parameter\.

#### Self\-Supervised Training\.

In the absence of a teacher model, the conventional approach for training a Transformer involves self\-supervised learning on a comprehensive text corpus\. For decoder\-only models designed for generative tasks, the optimization objective typically employs a causal language modeling loss\. If𝜽\\bm\{\\theta\}represents the model parameters, this loss is defined for a single input sequence as:

ℒdata​\(𝜽\)=−1n−1​∑i=1n−1log⁡𝐩i​\[ti\+1\]\.\\mathcal\{L\}\_\{\\text\{data\}\}\(\\bm\{\\theta\}\)=\-\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\-1\}\\log\\mathbf\{p\}\_\{i\}\[t\_\{i\+1\}\]\.\(1\)We refer to this objective as NLL \(Negative Log\-Likelihood\)\.

#### Knowledge Distillation\.

In contexts where a teacher model is available, the Knowledge Distillation \(KD\)\(Hintonet al\.,[2015](https://arxiv.org/html/2605.11513#bib.bib1)\)framework can be applied to any classification model\. This optimization objective is designed to align the student model’s output distribution with that of the teacher\. To quantify the discrepancy between these distributions \(denoted here asppandqq\) and calculate the associated loss, the Kullback\-Leibler \(KL\) divergence is commonly utilized with a temperature term within the softmax function:

KL⁡\(p∥q\)=∑x∈𝒳p​\(x\)​log⁡p​\(x\)q​\(x\)\.\\operatorname\{KL\}\(p\\,\\\|\\,q\)=\\sum\_\{x\\in\\mathcal\{X\}\}p\(x\)\\log\\frac\{p\(x\)\}\{q\(x\)\}\.This approach is recognized for its superior performance compared to NLL, as the teacher’s logits encapsulate more comprehensive information than one\-hot ground\-truth labels\. Denoting the output probabilities of the student as𝐩iS\\mathbf\{p\}\_\{i\}^\{S\}and the teacher as𝐩iT\\mathbf\{p\}\_\{i\}^\{T\}, the associated objective is defined as follows:

ℒlogits\(𝜽S\)=τ2n∑i=1nKL\(𝐩iT\|\|𝐩iS\)\.\\mathcal\{L\}\_\{\\text\{logits\}\}\(\\bm\{\\theta\}\_\{S\}\)=\\frac\{\\tau^\{2\}\}\{n\}\\sum\_\{i=1\}^\{n\}\\text\{KL\}\(\\mathbf\{p\}\_\{i\}^\{T\}\|\|\\mathbf\{p\}\_\{i\}^\{S\}\)\.In practice, the loss used is a composite of the NLL loss and the knowledge distillation loss:

ℒKD​\(𝜽S\)=\(1−α\)​ℒdata​\(𝜽S\)\+α​ℒlogits​\(𝜽S\),\\mathcal\{L\}\_\{\\text\{KD\}\}\(\\bm\{\\theta\}\_\{S\}\)=\(1\-\\alpha\)\\mathcal\{L\}\_\{\\text\{data\}\}\(\\bm\{\\theta\}\_\{S\}\)\+\\alpha\\mathcal\{L\}\_\{\\text\{logits\}\}\(\\bm\{\\theta\}\_\{S\}\),\(2\)

#### Distillation in open\-source research\.

The open\-source literature on distillation techniques for decoder\-only models is relatively sparse regarding large scale models\.Penget al\.\([2025](https://arxiv.org/html/2605.11513#bib.bib4)\)make a similar observation, motivating their thorough exploration of the design space using transparent, open\-weight teachers of 9B and 32B parameters and student scales ranging from 330M to 6\.8B parameters\. They observe that knowledge distillation consistently yields improvements compared to NLL and explore the hyperparameter space to identify optimal configurations\. Similarly,Busbridgeet al\.\([2025](https://arxiv.org/html/2605.11513#bib.bib5)\)propose a distillation scaling law to predict the performance of a distilled model based on its compute budget and the allocation between student and teacher models\. They scale teachers and students from 143M to 12\.6B parameters, trained on up to 16 times the Chinchilla\-optimal token budget\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib20)\)\.

#### Distillation in proprietary development\.

Parallel to these academic efforts, major industry laboratories have widely adopted distillation for the training of state\-of\-the\-art models\(Yanget al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib17); Gemini Team,[2025](https://arxiv.org/html/2605.11513#bib.bib16); Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18); DeepSeek\-AI,[2024](https://arxiv.org/html/2605.11513#bib.bib19)\)\. While the specific training pipelines and hyperparameters often remain proprietary, the high performance of these releases—some of which are open\-weights models—serves as strong empirical evidence for the efficacy of distillation in the pre\-training phase\.

#### Hidden Layer Distillation\.

Introduced byRomeroet al\.\([2015](https://arxiv.org/html/2605.11513#bib.bib2)\)\(FitNets\), HLD employs a two\-phase training strategy\. The first phase aligns the student’s intermediate hidden activations with the teacher’s, using a learned regressor\. The second phase then applies standard knowledge distillation across the full architecture\.

#### HLD for Transformers\.

Contemporary research into extracting value from the latent representations of Transformer models has introduced various modifications to the initial framework, resulting in a diverse array of loss functions\. Examples of these experimented variations include the alignment of attention maps\(Jiaoet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib6)\)and value\-relation\(Wanget al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib9),[2021](https://arxiv.org/html/2605.11513#bib.bib10)\), multi\-layer matching strategies using distinct layer allocation schemes\(Yuet al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib11)\), the prior training of task\-aware filters to match the student and teacher hidden dim\(Lianget al\.,[2023b](https://arxiv.org/html/2605.11513#bib.bib7)\), the application of non\-learned transformation\-invariant techniques to reconcile disparate hidden activation dimensions\(Dasgupta and Cohn,[2025](https://arxiv.org/html/2605.11513#bib.bib13)\), direct RMS matching\(Sunet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib14); Zuoet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib12)\)or cosine distance\(Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8)\)of normalized activations for teacher and student models of identical width\.

Notably, current implementations typically favor simple linear regressors over the multi\-layer models originally proposed byRomeroet al\.\([2015](https://arxiv.org/html/2605.11513#bib.bib2)\)\. Furthermore, the two\-phase training protocol described in earlier literature is frequently replaced by a single\-phase process that minimizes an aggregated objective function:

ℒH​L​D=β​ℒd​a​t​a\+α​ℒl​o​g​i​t​s\+γ​ℒe​m​b\.\\mathcal\{L\}\_\{HLD\}=\\beta\\mathcal\{L\}\_\{data\}\+\\alpha\\mathcal\{L\}\_\{logits\}\+\\gamma\\mathcal\{L\}\_\{emb\}\.\(3\)Unfortunately, most existing studies focus on the post\-training phase and on encoder or encoder\-decoder models\. To our knowledge,Lianget al\.\([2023b](https://arxiv.org/html/2605.11513#bib.bib7)\)are the only ones to experiment with HLD on decoder\-only models and only with continual pre\-training\. They worked withRadfordet al\.\([2019](https://arxiv.org/html/2605.11513#bib.bib38)\)’s GPT\-212\(120M parameters\) as the teacher and GPT\-26\(82M parameters\) as the student, which is initialized with a subset of the teacher’s layers\. To the best of our knowledge, our work is the first open\-source study of HLD during pre\-training for causal LLMs\.

## 3Proposed Method

To investigate both the original HLD formulations and contemporary approaches in the Transformer literature, we evaluate the following two methods\.

#### Sequential Optimization \(HLDF\)\.

We adaptRomeroet al\.\([2015](https://arxiv.org/html/2605.11513#bib.bib2)\)’s two\-phase protocol as a minimal extension to standard distillation that isolates the value of intermediate feature guidance\. The regressorfr​e​g\(\.;𝜽𝑹\)f\_\{reg\}\(\.;\\bm\{\\theta\_\{R\}\}\)is a one\-layer dense perceptron\. Phase 1 minimizes a normalized hint\-training \(HT\) loss:

ℒH​T​\(𝜽𝑺,𝜽𝑹\)=MeanSquaredError​\(𝐇𝐓𝐃𝐓/𝟐‖𝐇𝐓𝐃𝐓/𝟐‖,fr​e​g​\(𝐇𝐒𝐃𝐒/𝟐;𝜽𝑹\)‖fr​e​g​\(𝐇𝐒𝐃𝐒/𝟐;𝜽𝑹\)‖\)\.\\mathcal\{L\}\_\{HT\}\(\\bm\{\\theta\_\{S\}\},\\bm\{\\theta\_\{R\}\}\)=\\text\{MeanSquaredError\}\\\!\\left\(\\frac\{\\mathbf\{H^\{D\_\{T\}/2\}\_\{T\}\}\}\{\|\|\\mathbf\{H^\{D\_\{T\}/2\}\_\{T\}\}\|\|\},\\frac\{f\_\{reg\}\(\\mathbf\{H^\{D\_\{S\}/2\}\_\{S\}\};\\bm\{\\theta\_\{R\}\}\)\}\{\|\|f\_\{reg\}\(\\mathbf\{H^\{D\_\{S\}/2\}\_\{S\}\};\\bm\{\\theta\_\{R\}\}\)\|\|\}\\right\)\\\!\.\(4\)Phase 2 runs standard KD withℒK​D\\mathcal\{L\}\_\{KD\}as objective\. In short:

- •Phase 1: The objective isℒH​T​\(𝜽𝑺,𝜽𝑹\)\\mathcal\{L\}\_\{HT\}\(\\bm\{\\theta\_\{S\}\},\\bm\{\\theta\_\{R\}\}\)
- •Phase 2: The objective isℒK​D​\(𝜽𝑺\)\\mathcal\{L\}\_\{KD\}\(\\bm\{\\theta\_\{S\}\}\)

We use HLDF \(F stands for FitNet\) to refer to this training method\.

#### Joint Optimization \(HLDC\)\.

To align with loss functions previously explored in the literature for decoder architectures, we also implement a single\-stage training usingℒHLD\\mathcal\{L\}\_\{\\text\{HLD\}\}\([3](https://arxiv.org/html/2605.11513#S2.E3)\), with the embedding loss defined as:

ℒe​m​b​\(𝜽𝑺,𝜽𝑹\)=MeanSquaredError​\(𝐇𝐓𝐃𝐓/𝟐‖𝐇𝐓𝐃𝐓/𝟐‖,𝐖𝐑​𝐇𝐒𝐃𝐒/𝟐‖𝐖𝐑​𝐇𝐒𝐃𝐒/𝟐‖\)\.\\mathcal\{L\}\_\{emb\}\(\\bm\{\\theta\_\{S\}\},\\bm\{\\theta\_\{R\}\}\)=\\text\{MeanSquaredError\}\\\!\\left\(\\frac\{\\mathbf\{H^\{D\_\{T\}/2\}\_\{T\}\}\}\{\|\|\\mathbf\{H^\{D\_\{T\}/2\}\_\{T\}\}\|\|\},\\frac\{\\mathbf\{W\_\{R\}\}\\mathbf\{H^\{D\_\{S\}/2\}\_\{S\}\}\}\{\|\|\\mathbf\{W\_\{R\}\}\\mathbf\{H^\{D\_\{S\}/2\}\_\{S\}\}\|\|\}\\right\)\\\!\.We use HLDC \(C stands for composite\) to refer to this training method\.

#### Normalization Strategy\.

Unlike FitNets, we compare activations after normalization, consistent with prior work\(Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8); Sunet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib14)\)\. This approach is essential for Gemma 3 architectures in which we observed that the unnormalized residual stream can grow significantly in magnitude across layers\. Since the student model possesses fewer layers than the teacher, it cannot naturally match the teacher’s internal activation scale\. For instance, the mean squared magnitude of the teacher \(𝐇𝐓𝐃𝐓/𝟐\\mathbf\{H^\{D\_\{T\}/2\}\_\{T\}\}\) reaches approximately3×1053\\times 10^\{5\}, whereas the pre\-trained Gemma3 270M student \(𝐇𝐒𝐃𝐒/𝟐\\mathbf\{H^\{D\_\{S\}/2\}\_\{S\}\}\) only reaches≈1×105\\approx 1\\times 10^\{5\}\.

#### Data\.

We train the models on the English subset of the C4 dataset\(Raffelet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib27)\)\. This is a well\-established corpus, also used byBusbridgeet al\.\([2025](https://arxiv.org/html/2605.11513#bib.bib5)\)\. With around 150B tokens, it is sufficiently large to ensure that few sample are seen more than once during training\.

#### Teacher model\.

We pretrain a Gemma3 4B\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18)\)as teacher on 106B unique tokens with an alternative SentencePiece unigram model with a vocabulary size of 32k \(making it a 3\.4B model\)\. This choice is motivated by the large original Gemma vocabulary \(256k tokens\), which is multilingual while we train exclusively on English data\. Gemma3 4B is a 34\-layer pretrained decoder\-only Transformer with an embedding dimensiondT=2560d\_\{T\}=2560\.

#### Student models\.

As student models, we use Gemma3 270M and 1B\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18)\)with standard random initialization\. They are respectively 18 and 26\-layer decoder\-only Transformers sharing the architecture pattern of the teacher, with embedding dimensionsdSd\_\{S\}of 640 and 1152\. With the 32k vocabulary, they become 123M \(∼\\sim100M backbone\) and 735M \(∼\\sim700M backbone\) total parameters, yielding 27:1 and 4\.5:1 compression ratios\. The investigation of alternative teacher\-student pairs and different compression ratios remains a priority for future research\.

#### Logits\.

A common industry practice for optimizing offline storage is to retain only the top\-kklogits, wherekkis significantly smaller than the total vocabulary size; this method reduces storage requirements by several orders of magnitude\. For instance, the Gemma 3 tokenizer possesses a vocabulary of approximately 256,000 tokens, and a standard value ofk=128k=128is typically employed\. This threshold also functions as a regularizer, shielding the student model from the noise inherent in low\-probability logits\. In this study,kkis fixed at 128 and is not subject to further analysis, asPenget al\.\([2025](https://arxiv.org/html/2605.11513#bib.bib4)\)observe that the choice ofkkmakes little difference\.

#### Intermediate activations\.

However, the application of HLD necessitates the storage of full activations as vectors of the teacher embedding space, preventing the use of the top\-kktruncation technique\. In order to fit storage constraints, minimize the number of hyperparameters, we consider that evaluating the efficacy of HLD using the activations from a single teacher layer presents a significant challenge and a robust initial step for investigating hidden layer distillation\. To stay align withRomeroet al\.\([2015](https://arxiv.org/html/2605.11513#bib.bib2)\), we select the median layer of the teacher model for data retention and subsequent distillation\.

#### Optimization\.

Our experiments are built upon the NanoDo codebase\(Liuet al\.,[2024a](https://arxiv.org/html/2605.11513#bib.bib39)\), an open\-source vanilla decoder\-only pre\-training code base\. We use the AdamW optimizer\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2605.11513#bib.bib40)\)withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.98\\beta\_\{2\}=0\.98,ϵ=10−9\\epsilon=10^\{\-9\}, a weight decay of0\.10\.1, a batch size of 256 and a context length of 2048\. The learning rate follows a WSD schedule\(Huet al\.,[2024](https://arxiv.org/html/2605.11513#bib.bib45)\)with 1,000 warmup steps and a 10% decay phase ending atη/100\\eta/100\.

#### Hardware\.

All experiments, including the storage of the teacher’s logits and activations, as well as the training and evaluation of the models, are conducted on TPUv4 and TPUv7 chips\.

#### Evaluation\.

Since our study focuses exclusively on the pre\-training phase, we restricted performance assessments to perplexity or score\-based evaluations across:Wikitext 103\(Merityet al\.,[2017](https://arxiv.org/html/2605.11513#bib.bib28)\),HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib30)\),WinoGrande\(Sakaguchiet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib31)\),LAMBADA\(Papernoet al\.,[2016](https://arxiv.org/html/2605.11513#bib.bib29)\),PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib32)\),ARC\-E\(Chollet,[2019](https://arxiv.org/html/2605.11513#bib.bib33)\), and a held\-out validation split ofC4\(Raffelet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib27)\)\.

## 4Experiments

#### Hyperparameter Considerations\.

Evaluating the efficacy of a novel distillation methodology is complex, as subtle hyperparameter sensitivities can disproportionately influence both convergence and terminal performance\. A fair comparison must verify that the baseline is well\-tuned: would a practitioner be better served by adopting the new method or simply by performing a learning rate sweep on conventional KD? Rather than tuning HLD to beat KD, we characterize its general behavior over the peak learning ratesηK​D\\eta\_\{KD\}andηH​T\\eta\_\{HT\}, the temperatureτ\\tau, and the NLL weightα\\alphain the global objective \([2](https://arxiv.org/html/2605.11513#S2.E2)\) and \([3](https://arxiv.org/html/2605.11513#S2.E3)\)\.

#### Compute\-matched comparisons\.

Following the establishment of scaling laws for LLMs\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib20)\), compute\-matched comparisons have become a methodological necessity\. Consequently,all experimental evaluations in this work are conducted between models trained using equivalent computational budgets\.

#### Overtraining Unit\.

We quantify computational expenditure using an “overtraining unit”, denoted asOTk\\text\{OT\}\_\{k\}\. This unit representskktimes the compute required to reach the Chinchilla\-optimal point for our student model with NLL\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib20)\)\. For instance,OT1\\text\{OT\}\_\{1\}corresponds to the budget required to optimally train the 123M\-parameter student on 2B tokens \(20×backbone parameters20\\times\\text\{backbone parameters\}\)\. However,Hoffmannet al\.\([2022](https://arxiv.org/html/2605.11513#bib.bib20)\)’s compute\-optimal models do not account for inference costs during the model’s serving lifetime\. A newer paradigm,overtraining, suggests training models well beyond compute optimality to amortize serving costs\(Gadreet al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib42)\)\. To maintain relevance within this paradigm we train:

- •the 123M student up toOT54\\text\{OT\}\_\{54\}\(108B tokens\)
- •the 735M student up toOT12\\text\{OT\}\_\{12\}\(168B tokens\)\.

#### FLOPs accounting\.

We refine the standard6​N​D6NDapproximation for training compute, as the conventional formula neglects embedding and de\-embedding layers \([table˜3](https://arxiv.org/html/2605.11513#A1.T3)\)\. While these layers are negligible in large\-scale Transformers, they represent a significant portion of the total FLOPs for our student models due to the large vocabulary size \(V=32,000V=32\{,\}000\)\. The backward pass cost is equal to the forward pass for non\-parametric layers and twice the forward pass for layers with learnable parameters \(seee\.g\.\(Blondel and Roulet,[2024](https://arxiv.org/html/2605.11513#bib.bib41)\)\)\. LetNNdenote the number of student backbone parameters,dSd\_\{S\}the internal dimension,dTd\_\{T\}the teacher’s hidden dimension, andNregN\_\{\\text\{reg\}\}the parameters in the mapping regressor\. FollowingAustinet al\.\([2025](https://arxiv.org/html/2605.11513#bib.bib43)\)and[table˜3](https://arxiv.org/html/2605.11513#A1.T3), the estimated compute cost𝒞d​a​t​a\\mathcal\{C\}\_\{data\}per token of self\-supervised learning \([1](https://arxiv.org/html/2605.11513#S2.E1)\), the cost𝒞K​D\\mathcal\{C\}\_\{KD\}of conventional KD \([2](https://arxiv.org/html/2605.11513#S2.E2)\), the cost𝒞H​T\\mathcal\{C\}\_\{HT\}of the first phase of the HLDF that train only up to the matched layer \([4](https://arxiv.org/html/2605.11513#S3.E4)\) and the cost𝒞H​L​D​C\\mathcal\{C\}\_\{HLDC\}of HLDC \([3](https://arxiv.org/html/2605.11513#S2.E3)\) are:

𝒞d​a​t​a≈6​N⏟backbone\+6​dS​V⏟logits,𝒞K​D≈𝒞d​a​t​a\+𝒞Teacher,\\displaystyle\\mathcal\{C\}\_\{data\}\\approx\\underbrace\{6N\}\_\{\\text\{backbone\}\}\+\\underbrace\{6d\_\{S\}V\}\_\{\\text\{logits\}\},\\quad\\mathcal\{C\}\_\{KD\}\\approx\\mathcal\{C\}\_\{data\}\+\\mathcal\{C\}\_\{\\text\{Teacher\}\},𝒞H​T≈3​N⏟half backbone\+6​Nr​e​g⏟Mapping\+𝒞Teacher/2and𝒞H​L​D​C≈𝒞K​D\+6​Nr​e​g⏟Mapping\\displaystyle\\mathcal\{C\}\_\{HT\}\\approx\\underbrace\{3N\}\_\{\\text\{half backbone\}\}\+\\underbrace\{6N\_\{reg\}\}\_\{\\text\{Mapping\}\}\+\\mathcal\{C\}\_\{\\text\{Teacher\}\}/2\\quad\\text\{and\}\\quad\\mathcal\{C\}\_\{HLDC\}\\approx\\mathcal\{C\}\_\{KD\}\+\\underbrace\{6N\_\{reg\}\}\_\{\\text\{Mapping\}\}Since teacher logits are already pre\-computed,𝒞Teacher\\mathcal\{C\}\_\{\\text\{Teacher\}\}is negligible\. For the 123M student, the relative costs are

𝒞K​D𝒞d​a​t​a≈1\.000,𝒞H​T𝒞d​a​t​a≈0\.442,𝒞H​L​D​C𝒞d​a​t​a≈1\.027\.\\frac\{\\mathcal\{C\}\_\{KD\}\}\{\\mathcal\{C\}\_\{data\}\}\\approx 1\.000,\\quad\\frac\{\\mathcal\{C\}\_\{HT\}\}\{\\mathcal\{C\}\_\{data\}\}\\approx 0\.442,\\quad\\frac\{\\mathcal\{C\}\_\{HLDC\}\}\{\\mathcal\{C\}\_\{data\}\}\\approx 1\.027\.Therefore, while Phase 1 of HLDF updates approximately half the model’s backbone parameters, its computational cost is significantly less than 50% of the KD baseline\. This efficiency is amplified by the high ratio of vocabulary size to model parameters in the 123M architecture, where the bypassed output projection accounts for a large fraction of FLOPs\. As model size scales and backbone computations dominate, this advantage vanishes: a Gemma3 27B student\(Gemma Team,[2025](https://arxiv.org/html/2605.11513#bib.bib18)\)with a2×2\\timeswider teacher yields𝒞H​T𝒞d​a​t​a≈0\.5001\\frac\{\\mathcal\{C\}\_\{HT\}\}\{\\mathcal\{C\}\_\{data\}\}\\approx 0\.5001\.

#### HLD\-specific hyperparameters\.

To emulate a “no\-tuning” scenario, we fix HLDF\-specific hyperparameters to heuristic values\. We employ a single\-layer MLP regressor with an expansion factor of 4 \(relative to the student’s hidden dimension\), mirroring the standard architecture of the student’s internal feed\-forward layers\. For the Phase 1 schedule, we use a constant learning rateηH​T\\eta\_\{HT\}following a 1,000\-step linear warmup, with no subsequent decay\. For HLDC, the only exclusive hyperparameter isγ\\gamma, the weight of the embedding loss\.

#### Two evaluation methods\.

We compare HLD against KD in two complementary ways\. First \(§[4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px7)\), we perform pointwise comparisons at shared hyperparameter configurations, evaluating robustness across a broad grid without any method\-specific tuning\. Second \(§[4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px8)\), we give each method its independently selected best hyperparameters, enabling a ceiling\-to\-ceiling comparison\.

Table 1:Hyperparameter grid\.
#### Shared hyperparameters\.

For each set of hyperparameters\(η=ηK​D=ηH​T,τ,α\)\(\\eta=\\eta\_\{KD\}=\\eta\_\{HT\},\\tau,\\alpha\), we run one NLL training and one KD training as baselines\. For HLDF, we useηH​T=η\\eta\_\{HT\}=\\etain Phase 1 and\(η,τ,α\)\(\\eta,\\tau,\\alpha\)in Phase 2, varying the budget splitP1P\_\{1\}between phases\. For HLDC, we explore small values ofγ\\gamma\(Lianget al\.,[2023b](https://arxiv.org/html/2605.11513#bib.bib7)\)with the same\(η,τ,α\)\(\\eta,\\tau,\\alpha\)as the baseline\. We select values forη,τ,\\eta,\\tau,andα\\alphafrom regimes where KD is known to perform well\(Penget al\.,[2025](https://arxiv.org/html/2605.11513#bib.bib4)\), adaptingη\\etato our batch size via square\-root scaling\(Hofferet al\.,[2017](https://arxiv.org/html/2605.11513#bib.bib34)\)\. The complete set of explored hyperparameters is summarized in[table˜1](https://arxiv.org/html/2605.11513#S4.T1)\. To evaluate the general performance of HLD over KD, we perform a pointwise comparison using identical hyperparameter sets\(η,τ,α\)\(\\eta,\\tau,\\alpha\)\. Detailed results are provided in[figs\.˜2](https://arxiv.org/html/2605.11513#S5.F2),[3](https://arxiv.org/html/2605.11513#S5.F3),[4](https://arxiv.org/html/2605.11513#S5.F4),[5](https://arxiv.org/html/2605.11513#A1.F5),[6](https://arxiv.org/html/2605.11513#A1.F6)and[7](https://arxiv.org/html/2605.11513#A1.F7)\.

#### Best hyperparameters\.

We ask whether HLD outperforms KD when each method uses its best configuration\. We select for each method the best\-performing\(η∗,τ∗,α∗\)\(\\eta^\{\\ast\},\\tau^\{\\ast\},\\alpha^\{\\ast\}\)from the shared\-hyperparameter grid[table˜1](https://arxiv.org/html/2605.11513#S4.T1), and compare the resulting peak performances head\-to\-head in[table˜2](https://arxiv.org/html/2605.11513#S5.T2)\.

## 5Results

We present our results in four complementary views:

- •[Figures˜2](https://arxiv.org/html/2605.11513#S5.F2),[4](https://arxiv.org/html/2605.11513#S5.F4)and[3](https://arxiv.org/html/2605.11513#S5.F3)show histograms of the improvement of HLD over KD at fixed hyperparameter sets, illustrating the distribution of gains of each evaluated method against the KD baseline\.
- •[Figures˜5](https://arxiv.org/html/2605.11513#A1.F5),[6](https://arxiv.org/html/2605.11513#A1.F6)and[7](https://arxiv.org/html/2605.11513#A1.F7)in the appendix are scatter plots in which each point corresponds to one shared hyperparameter set: thexx\-axis reports the KD score and theyy\-axis the score of the compared method \(HLD or NLL\)\. These plots reveal the general behavior of each method relative to its baseline, as well as the behavior of KD relative to its NLL baseline\.
- •[Table˜2](https://arxiv.org/html/2605.11513#S5.T2)reports the best score achieved by each method, alongside the teacher and the NLL baseline\.
- •Finally,LABEL:tab:xps\_123andLABEL:tab:xps\_735in the appendix list the full set of training runs together with their evaluation results\.

![Refer to caption](https://arxiv.org/html/2605.11513v1/x1.png)Figure 2:C4 perplexity improvement over KD\. Distribution of pointwise improvements across all hyperparameter configurations for both sizes\. Positive values indicate lower perplexity than KD\.Table 2:Best Performance Comparison\.Test log\-perplexity \(↓\\downarrow\) or the error rate \(↓\\downarrow\) for the optimal hyperparameters found within the compute\-matched search space\. Results are presented as123M / 735Mstudent performances\.Boldindicates the best result for a given student size\.#### Baseline\.

A prerequisite for any meaningful comparison is that the KD baseline itself is sufficiently well\-tuned to serve as a competitive reference for HLD\. Several elements support this\. First, as shown in[table˜2](https://arxiv.org/html/2605.11513#S5.T2), our best KD configuration outperforms NLL by a clear margin on every benchmark, even on noisier evaluations such as WinoGrande\. Second, on the perplexity scatter plots \([fig\.˜5](https://arxiv.org/html/2605.11513#A1.F5)\), all but one KD run beats NLL for the 735M student, and this trend is corroborated by the downstream evaluations \([fig\.˜7](https://arxiv.org/html/2605.11513#A1.F7)\)\. The picture is more mixed for the 123M student: roughly half of the KD runs fail to beat NLL on C4 perplexity \([fig\.˜5](https://arxiv.org/html/2605.11513#A1.F5)\), but the downstream results \([fig\.˜6](https://arxiv.org/html/2605.11513#A1.F6)\) again show KD outperforming NLL on nearly all evaluation points\. Taken together, these observations indicate that our KD baseline is genuinely competitive, and that any comparison with HLD is being made against a strong reference rather than a weak one—a prerequisite that, in our knowledge, is not always met in the HLD literature\.

#### Shared hyperparameters\.

On perplexity, HLDC performs on par with KD on C4, while HLDF achieves a modest but systematic improvement \([fig\.˜2](https://arxiv.org/html/2605.11513#S5.F2)\)\. The effect size is small and, given the computational cost of pre\-training, we lack the statistical power to claim significance from independent seeds; we instead rely on the consistency of the sign of the improvement across the entire shared\-hyperparameter grid\. Downstream evaluations tell a different story: as shown in[figs\.˜3](https://arxiv.org/html/2605.11513#S5.F3)and[4](https://arxiv.org/html/2605.11513#S5.F4), the score distributions are more spread out and roughly centered around zero on all benchmarks, indicating that the potential hidden\-layer signal does not translate into consistent gains\. A closer look at the perplexity scatter plot \([fig\.˜5](https://arxiv.org/html/2605.11513#A1.F5)\) further reveals that HLDF’s improvement over KD occurs predominantly in regimes where KD itself performs poorly, while in regimes where KD is already efficient the gain becomes negligible\.

![Refer to caption](https://arxiv.org/html/2605.11513v1/x2.png)Figure 3:Shared\-HP — downstream score improvement over KD for the735M student\. Distribution of pointwise score improvements across all hyperparameter configurations on benchmarks\.![Refer to caption](https://arxiv.org/html/2605.11513v1/x3.png)Figure 4:Shared\-HP — downstream score improvement over KD for the123M student\. Distribution of pointwise score improvements across all hyperparameter configurations on benchmarks\. Green indicates HLD beats KD\.
#### Best hyperparameters\.

[Table˜2](https://arxiv.org/html/2605.11513#S5.T2)report the best\-configuration performance for each method on both students\. No method dominates KD across the seven evaluations: differences are small and split between methods, reinforcing the conclusion that, in our setup, HLD does not provide a consistent, reliable improvement over logit\-based distillation that is tuned enough and under compute\-matched conditions\.

#### Reconciling with prior work\.

Our findings appear to contradict a body of work reporting consistent gains from intermediate\-layer matching\(Jiaoet al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib6); Sanhet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib8); Wanget al\.,[2020](https://arxiv.org/html/2605.11513#bib.bib9),[2021](https://arxiv.org/html/2605.11513#bib.bib10); Sunet al\.,[2019](https://arxiv.org/html/2605.11513#bib.bib14); Lianget al\.,[2023b](https://arxiv.org/html/2605.11513#bib.bib7); Zuoet al\.,[2022](https://arxiv.org/html/2605.11513#bib.bib12); Dasgupta and Cohn,[2025](https://arxiv.org/html/2605.11513#bib.bib13)\)\. We see three methodological factors that plausibly account for the discrepancy\. \(i\)*Compute accounting\.*Most prior comparisons match token counts rather than FLOPs and some variations of HLD cost more by token than KD\. \(ii\)*Architecture and training phase\.*The HLD literature is dominated by encoder and encoder\-decoder models in post\-training or task\-specific distillation regimes, where representations are already shaped by a pre\-training objective and the alignment target is correspondingly well\-defined\. Causal decoders trained from scratch present a fundamentally different optimization landscape, and our results suggest the inductive bias HLD provides is less valuable in this setting\. \(iii\)*Baseline tuning\.*When HLD is introduced as the contribution, the KD baseline is rarely swept as carefully as the proposed method\. Our shared\-hyperparameter protocol explicitly avoids this asymmetry, and the resulting KD baseline is strong enough to absorb most of HLD’s apparent advantage\. \(iv\)*Methodology\.*Adapting the HLD methodology from few millions parameters CNN models to almost billions parameters transformers is not straightforward and would certainly need more profound design choices \(losses, adapter architecture, protocol\) to extract the information from the latent representations of the teacher\.

#### On the residual HLDF signal\.

The systematic perplexity gain from HLDF is small but does not vanish under our protocol\. One interpretation is that hint\-training acts as warm\-start mechanism — aligning the student’s mid\-network representations early may provide a better initialization for the subsequent KD phase, without fundamentally changing what the student can learn\. This would explain why the effect appears in C4 perplexity \(sensitive to fine\-grained distributional fit\) but not in downstream benchmarks \(sensitive to coarser capabilities that converge similarly under both protocols\) and why the difference with KD vanishes in regimes where KD performs the best\.

## Conclusion

This work presents a compute\-controlled evaluation of Hidden Layer Distillation \(HLD\) for the pre\-training of causal LLMs on the English version of C4\. Despite the appeal of aligning intermediate representations to capture the teacher’s deep internal knowledge, our results indicate that HLD does not consistently outperform a well\-tuned logit\-based KD baseline when training budgets are strictly equalized\. Furthermore, the complexity created by HLD introduces hyperparameter sensitivity that may outweigh its potential benefits in practical deployment scenarios\. These findings underscore the resilience of standard logit\-based distillation and highlight the necessity of exact FLOPs accounting when evaluating novel supervision techniques\.

That said, HLDF yields a systematic perplexity gain over KD, suggesting a latent signal can be extracted that \(i\) need to be confirmed through stronger statistical evidence and a broader context \(larger scales, different compression ratios, alternative model families\) and \(ii\) would likely require a significant breakthrough or substantial design choices \(losses, adapter architecture, protocol\) to thoroughly extract the information from the teacher’s latent representations\.

## References

- Variational information distillation for knowledge transfer\.InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019,pp\. 9163–9171\.External Links:[Link](http://openaccess.thecvf.com/content%5C_CVPR%5C_2019/html/Ahn%5C_Variational%5C_Information%5C_Distillation%5C_for%5C_Knowledge%5C_Transfer%5C_CVPR%5C_2019%5C_paper.html)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1)\.
- J\. Austin, S\. Douglas, R\. Frostig, A\. Levskaya, C\. Chen, S\. Vikram, F\. Lebron, P\. Choy, V\. Ramasesh, A\. Webson, and R\. Pope \(2025\)How to scale your model\.Google DeepMind\.Note:OnlineRetrieved from https://jax\-ml\.github\.io/scaling\-book/Cited by:[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px4.p1.10)\.
- Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.InThirty\-Fourth AAAI Conference on Artificial Intelligence,Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- M\. Blondel and V\. Roulet \(2024\)The elements of differentiable programming\.CoRRabs/2403\.14606\.External Links:[Link](https://doi.org/10.48550/arXiv.2403.14606)Cited by:[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px4.p1.10)\.
- D\. Busbridge, A\. Shidani, F\. Weers, J\. Ramapuram, E\. Littwin, and R\. Webb \(2025\)Distillation scaling laws\.InForty\-second International Conference on Machine Learning, ICML 2025,External Links:[Link](https://openreview.net/forum?id=1nEBAkpfb9)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px4.p1.1)\.
- F\. Chollet \(2019\)On the measure of intelligence\.External Links:1911\.01547,[Link](https://arxiv.org/abs/1911.01547)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- F\. Dalvi, H\. Sajjad, N\. Durrani, and Y\. Belinkov \(2020\)Analyzing redundancy in pretrained transformer models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,pp\. 4908–4926\.External Links:[Link](https://doi.org/10.18653/v1/2020.emnlp-main.398)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1)\.
- S\. Dasgupta and T\. Cohn \(2025\)Improving language model distillation through hidden state matching\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025,External Links:[Link](https://openreview.net/forum?id=IcVSKhVpKu)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-v3 technical report\.CoRRabs/2412\.19437\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.19437)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px5.p1.1)\.
- N\. Durrani, H\. Sajjad, F\. Dalvi, and Y\. Belinkov \(2020\)Analyzing individual neurons in pre\-trained language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,pp\. 4865–4880\.External Links:[Link](https://doi.org/10.18653/v1/2020.emnlp-main.395)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1)\.
- S\. Y\. Gadre, G\. Smyrnis, V\. Shankar, S\. Gururangan, M\. Wortsman, R\. Shao, J\. Mercat, A\. Fang, J\. Li, S\. Keh, R\. Xin, M\. Nezhurina, I\. Vasiljevic, L\. Soldaini, J\. Jitsev, A\. Dimakis, G\. Ilharco, P\. W\. Koh, S\. Song, and T\. Kollar \(2025\)Language models scale reliably with over\-training and on downstream tasks\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025,External Links:[Link](https://openreview.net/forum?id=iZeQBqJamf)Cited by:[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px3.p1.4)\.
- Gemini Team \(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.CoRRabs/2507\.06261\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.06261)Cited by:[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px5.p1.1)\.
- Gemma Team \(2025\)Gemma 3 technical report\.CoRRabs/2503\.19786\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.19786)Cited by:[1st item](https://arxiv.org/html/2605.11513#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§1](https://arxiv.org/html/2605.11513#S1.p4.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px5.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px5.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px6.p1.3),[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px4.p1.13)\.
- J\. Hestness, S\. Narang, N\. Ardalani, G\. F\. Diamos, H\. Jun, H\. Kianinejad, Md\. M\. A\. Patwary, Y\. Yang, and Y\. Zhou \(2017\)Deep learning scaling is predictable, empirically\.CoRRabs/1712\.00409\.External Links:[Link](http://arxiv.org/abs/1712.00409)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p1.1)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.CoRRabs/1503\.02531\.External Links:[Link](http://arxiv.org/abs/1503.02531),1503\.02531Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px3.p1.2)\.
- E\. Hoffer, I\. Hubara, and D\. Soudry \(2017\)Train longer, generalize better: closing the generalization gap in large batch training of neural networks\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017,pp\. 1731–1741\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/a5e0ff62be0b08456fc7f1e88812af3d-Abstract.html)Cited by:[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px7.p1.10)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre \(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p1.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px3.p1.4)\.
- S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao, X\. Zhang, Z\. L\. Thai, K\. Zhang, C\. Wang, Y\. Yao, C\. Zhao, J\. Zhou, J\. Cai, Z\. Zhai, N\. Ding, C\. Jia, G\. Zeng, D\. Li, Z\. Liu, and M\. Sun \(2024\)MiniCPM: unveiling the potential of small language models with scalable training strategies\.External Links:2404\.06395,[Link](https://arxiv.org/abs/2404.06395)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px9.p1.5)\.
- X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu \(2020\)TinyBERT: distilling BERT for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4163–4174\.External Links:[Link](https://doi.org/10.18653/v1/2020.findings-emnlp.372)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- C\. Liang, H\. Jiang, Z\. Li, X\. Tang, B\. Yin, and T\. Zhao \(2023a\)HomoDistil: homotopic task\-agnostic distillation of pre\-trained transformers\.InThe Eleventh International Conference on Learning Representations, ICLR 2023,External Links:[Link](https://openreview.net/forum?id=D7srTrGhAs)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1)\.
- C\. Liang, S\. Zuo, Q\. Zhang, P\. He, W\. Chen, and T\. Zhao \(2023b\)Less is more: task\-aware layer\-wise distillation for language model compression\.InInternational Conference on Machine Learning, ICML 2023,Vol\.202,pp\. 20852–20867\.External Links:[Link](https://proceedings.mlr.press/v202/liang23j.html)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p2.2),[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px7.p1.10),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- P\. J\. Liu, R\. Novak, J\. Lee, M\. Wortsman, L\. Xiao, K\. Everett, A\. A\. Alemi, M\. Kurzeja, P\. Marcenac, I\. Gur, S\. Kornblith, K\. Xu, G\. Elsayed, I\. Fischer, J\. Pennington, B\. Adlam, and J\. Dickstein \(2024a\)NanoDO: a minimal transformer decoder\-only language model implementation in JAX\.\.External Links:[Link](http://github.com/google-deepmind/nanodo)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p4.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px9.p1.5)\.
- Z\. Liu, C\. Zhao, F\. N\. Iandola, C\. Lai, Y\. Tian, I\. Fedorov, Y\. Xiong, E\. Chang, Y\. Shi, R\. Krishnamoorthi, L\. Lai, and V\. Chandra \(2024b\)MobileLLM: optimizing sub\-billion parameter language models for on\-device use cases\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=EIGbXbxcUQ)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.In7th International Conference on Learning Representations, ICLR 2019,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px9.p1.5)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.In5th International Conference on Learning Representations, ICLR 2017,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016,External Links:[Link](https://doi.org/10.18653/v1/p16-1144)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- H\. Peng, X\. Lv, Y\. Bai, Z\. Yao, J\. Zhang, L\. Hou, and J\. Li \(2025\)Pre\-training distillation for large language models: A design space exploration\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025,pp\. 3603–3618\.External Links:[Link](https://aclanthology.org/2025.acl-long.181/)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px7.p1.5),[§4](https://arxiv.org/html/2605.11513#S4.SS0.SSS0.Px7.p1.10)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.External Links:[Link](https://api.semanticscholar.org/CorpusID:160025533)Cited by:[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p2.2)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.J\. Mach\. Learn\. Res\.21,pp\. 140:1–140:67\.External Links:[Link](https://jmlr.org/papers/v21/20-074.html)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p4.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px4.p1.1)\.
- A\. Romero, N\. Ballas, S\. E\. Kahou, A\. Chassang, C\. Gatta, and Y\. Bengio \(2015\)FitNets: hints for thin deep nets\.In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\-9, 2015,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1412.6550)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p2.3),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px8.p1.1)\.
- J\. S\. Rosenfeld, A\. Rosenfeld, Y\. Belinkov, and N\. Shavit \(2020\)A constructive prediction of the generalization error across scales\.In8th International Conference on Learning Representations, ICLR 2020,External Links:[Link](https://openreview.net/forum?id=ryenvpEKDr)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2019\)WinoGrande: an adversarial winograd schema challenge at scale\.External Links:1907\.10641,[Link](https://arxiv.org/abs/1907.10641)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.CoRRabs/1910\.01108\.External Links:[Link](http://arxiv.org/abs/1910.01108)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px3.p1.4),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- S\. Sun, Y\. Cheng, Z\. Gan, and J\. Liu \(2019\)Patient knowledge distillation for BERT model compression\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP\-IJCNLP 2019,pp\. 4322–4331\.External Links:[Link](https://doi.org/10.18653/v1/D19-1441)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px3.p1.4),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- K\. Tan and D\. Wang \(2021\)Towards model compression for deep learning based speech enhancement\.IEEE ACM Trans\. Audio Speech Lang\. Process\.29,pp\. 1785–1794\.External Links:[Link](https://doi.org/10.1109/TASLP.2021.3082282)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017,pp\. 5998–6008\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px1.p1.10)\.
- W\. Wang, H\. Bao, S\. Huang, L\. Dong, and F\. Wei \(2021\)MiniLMv2: multi\-head self\-attention relation distillation for compressing pretrained transformers\.InFindings of the Association for Computational Linguistics: ACL/IJCNLP 2021,pp\. 2140–2151\.External Links:[Link](https://doi.org/10.18653/v1/2021.findings-acl.188)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou \(2020\)MiniLM: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020,External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p2.1),[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.09388)Cited by:[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px5.p1.1)\.
- Z\. Yu, Y\. Wen, and L\. Mou \(2025\)Revisiting intermediate\-layer matching in knowledge distillation: layer\-selection strategy doesn’t matter \(much\)\.External Links:2502\.04499,[Link](https://arxiv.org/abs/2502.04499)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019,pp\. 4791–4800\.External Links:[Link](https://doi.org/10.18653/v1/p19-1472)Cited by:[§3](https://arxiv.org/html/2605.11513#S3.SS0.SSS0.Px11.p1.1)\.
- S\. Zuo, Q\. Zhang, C\. Liang, P\. He, T\. Zhao, and W\. Chen \(2022\)MoEBERT: from BERT to mixture\-of\-experts via importance\-guided adaptation\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022,pp\. 1610–1623\.External Links:[Link](https://doi.org/10.18653/v1/2022.naacl-main.116)Cited by:[§1](https://arxiv.org/html/2605.11513#S1.p3.1),[§2](https://arxiv.org/html/2605.11513#S2.SS0.SSS0.Px7.p1.1),[§5](https://arxiv.org/html/2605.11513#S5.SS0.SSS0.Px4.p1.1)\.

## Appendix AAppendix

### A\.1Accounting

Table 3:Compute cost for forward pass, per token\.
### A\.2Scatter plots

![Refer to caption](https://arxiv.org/html/2605.11513v1/x4.png)Figure 5:C4 perplexity improvement over KD\. Scatter plot of pointwise score improvements across all hyperparameter configurations on C4 evaluation set\. Each point is one hyperparameter set\. For each point, the x axis is the KD log\-perplexity associated with the hyperparameter set, the y axis is the HLD&NLL log\-perplexity associated with the same hyperparameter set\. The lower is always the better so the green zone is where HLD or NLL beats KD\.![Refer to caption](https://arxiv.org/html/2605.11513v1/x5.png)Figure 6:Shared\-HP — downstream score improvement over KD for the123M student\. Scatter plot of pointwise score improvements across all hyperparameter configurations on benchmarks\. Each point is one hyperparameter set\. For each point, the x axis is the KD score associated with the hyperparameter set, the y axis is the HLD&NLL score associated with the same hyperparameter set\. The lower is always the better so the green zone is where HLD or NLL beats KD\.![Refer to caption](https://arxiv.org/html/2605.11513v1/x6.png)Figure 7:Shared\-HP — downstream score improvement over KD for the735M student\. Scatter plot of pointwise score improvements across all hyperparameter configurations on benchmarks\. Each point is one hyperparameter set\. For each point, the x axis is the KD score associated with the hyperparameter set, the y axis is the HLD&NLL score associated with the same hyperparameter set\. The lower is always the better so the green zone is where HLD or NLL beats KD\.
### A\.3All results

Table 4:All training runs for the 123M student\.Method𝜼\\bm\{\\eta\}𝑷𝟏\\bm\{P\_\{1\}\}𝜶\\bm\{\\alpha\}𝝉\\bm\{\\tau\}𝜸\\bm\{\\gamma\}C4WikitextHSwagPiqaWinoGLambadaArc\-ENLL0\.00010\.00–––3\.4583\.4220\.7190\.3790\.4510\.5720\.630NLL0\.00040\.00–––3\.0713\.0000\.6510\.3320\.4480\.5390\.543NLL0\.00160\.00–––3\.0633\.0620\.6430\.3380\.4400\.5270\.529KD0\.00010\.000\.70\.5–3\.5513\.5160\.7080\.3740\.4400\.5740\.625KD0\.00010\.000\.71\.0–3\.4813\.4690\.7170\.3840\.4380\.5730\.623KD0\.00010\.000\.90\.5–3\.7553\.8590\.6990\.3740\.4310\.5650\.628KD0\.00010\.000\.91\.0–3\.4923\.4690\.7160\.3740\.4460\.5740\.625KD0\.00040\.000\.70\.5–3\.0723\.0000\.6450\.3310\.4440\.5310\.530KD0\.00040\.000\.71\.0–3\.0362\.9380\.6300\.3340\.4400\.5210\.534KD0\.00040\.000\.90\.5–3\.1523\.1410\.6320\.3320\.4340\.5110\.545KD0\.00040\.000\.91\.0–3\.0452\.9380\.6290\.3260\.4400\.5290\.525KD0\.00160\.000\.70\.5–3\.0312\.9220\.6270\.3310\.4270\.5200\.535KD0\.00160\.000\.71\.0–3\.0052\.9840\.6220\.3210\.4330\.5130\.505KD0\.00160\.000\.90\.5–3\.1003\.0470\.6160\.3230\.4460\.5130\.542KD0\.00160\.000\.91\.0–3\.0142\.9060\.6190\.3220\.4390\.5090\.508HLDC0\.00010\.000\.70\.50\.053\.5513\.8120\.7080\.3840\.4330\.5710\.625HLDC0\.00010\.000\.70\.50\.103\.5553\.5160\.7070\.3760\.4400\.5690\.625HLDC0\.00010\.000\.71\.00\.053\.4823\.4690\.7160\.3850\.4490\.5740\.619HLDC0\.00010\.000\.71\.00\.103\.4853\.4530\.7140\.3760\.4330\.5650\.624HLDC0\.00010\.000\.90\.50\.053\.7564\.0000\.6970\.3740\.4400\.5610\.636HLDC0\.00010\.000\.90\.50\.103\.7554\.0310\.6980\.3810\.4290\.5580\.633HLDC0\.00010\.000\.91\.00\.053\.4933\.4840\.7230\.3790\.4470\.5730\.630HLDC0\.00010\.000\.91\.00\.103\.4923\.4530\.7180\.3790\.4440\.5720\.622HLDC0\.00040\.000\.70\.50\.053\.0723\.0780\.6400\.3260\.4460\.5230\.524HLDC0\.00040\.000\.70\.50\.103\.0723\.0000\.6320\.3340\.4490\.5260\.527HLDC0\.00040\.000\.71\.00\.053\.0372\.9220\.6390\.3330\.4400\.5290\.524HLDC0\.00040\.000\.71\.00\.103\.0372\.9380\.6320\.3360\.4370\.5270\.525HLDC0\.00040\.000\.90\.50\.053\.1543\.0940\.6360\.3310\.4330\.5120\.549HLDC0\.00040\.000\.90\.50\.103\.1543\.2030\.6360\.3370\.4340\.5180\.545HLDC0\.00040\.000\.91\.00\.053\.0462\.9380\.6360\.3290\.4530\.5240\.525HLDC0\.00040\.000\.91\.00\.103\.0452\.9380\.6350\.3280\.4440\.5190\.526HLDC0\.00160\.000\.70\.50\.053\.0322\.9380\.6160\.3200\.4360\.5140\.519HLDC0\.00160\.000\.70\.50\.103\.0312\.9220\.6210\.3250\.4210\.5180\.532HLDC0\.00160\.000\.71\.00\.053\.0052\.9380\.6300\.3230\.4230\.5050\.505HLDC0\.00160\.000\.71\.00\.103\.0072\.8910\.6220\.3230\.4360\.5240\.528HLDC0\.00160\.000\.90\.50\.053\.1003\.0310\.6230\.3250\.4250\.5040\.538HLDC0\.00160\.000\.90\.50\.103\.1023\.0620\.6210\.3290\.4250\.5180\.551HLDC0\.00160\.000\.91\.00\.053\.0152\.9220\.6180\.3230\.4310\.5160\.519HLDC0\.00160\.000\.91\.00\.103\.0152\.9380\.6230\.3250\.4360\.5140\.510HLDF0\.00010\.010\.70\.5–3\.5403\.8280\.7050\.3750\.4300\.5760\.628HLDF0\.00010\.050\.70\.5–3\.5423\.6410\.7000\.3640\.4320\.5760\.623HLDF0\.00010\.010\.71\.0–3\.4734\.0940\.7220\.3780\.4360\.5760\.617HLDF0\.00010\.050\.71\.0–3\.4703\.5780\.7110\.3740\.4360\.5820\.626HLDF0\.00010\.010\.90\.5–3\.7394\.0940\.7060\.3720\.4360\.5690\.623HLDF0\.00010\.050\.90\.5–3\.7333\.8590\.7020\.3620\.4450\.5720\.630HLDF0\.00010\.010\.91\.0–3\.4824\.0940\.7070\.3750\.4480\.5810\.631HLDF0\.00010\.050\.91\.0–3\.4803\.6090\.7140\.3740\.4510\.5740\.626HLDF0\.00040\.010\.70\.5–3\.0613\.0160\.6300\.3200\.4270\.5280\.535HLDF0\.00040\.050\.70\.5–3\.0603\.0160\.6240\.3320\.4190\.5200\.538HLDF0\.00040\.010\.71\.0–3\.0292\.9690\.6330\.3240\.4420\.5270\.520HLDF0\.00040\.050\.71\.0–3\.0282\.9690\.6340\.3250\.4320\.5270\.533HLDF0\.00040\.010\.90\.5–3\.1433\.1560\.6350\.3330\.4400\.5220\.550HLDF0\.00040\.050\.90\.5–3\.1403\.0940\.6260\.3190\.4330\.5160\.550HLDF0\.00040\.010\.91\.0–3\.0372\.9690\.6340\.3230\.4400\.5290\.524HLDF0\.00040\.050\.91\.0–3\.0362\.9690\.6270\.3240\.4320\.5210\.529HLDF0\.00160\.010\.70\.5–3\.0323\.0000\.6230\.3210\.4260\.5190\.528HLDF0\.00160\.050\.70\.5–3\.0272\.8910\.6340\.3180\.4370\.5200\.529HLDF0\.00160\.010\.71\.0–3\.0022\.9380\.6180\.3160\.4470\.5080\.518HLDF0\.00160\.050\.71\.0–3\.0002\.8750\.6250\.3220\.4370\.5070\.519HLDF0\.00160\.010\.90\.5–3\.0973\.0000\.6240\.3320\.4310\.5120\.535HLDF0\.00160\.050\.90\.5–3\.0933\.0000\.6220\.3290\.4330\.5080\.537HLDF0\.00160\.010\.91\.0–3\.0122\.9060\.6220\.3170\.4330\.5220\.516HLDF0\.00160\.050\.91\.0–3\.0102\.8910\.6290\.3160\.4290\.5090\.515Table 5:All training runs for the 735M student\.Method𝜼\\bm\{\\eta\}𝑷𝟏\\bm\{P\_\{1\}\}𝜶\\bm\{\\alpha\}𝝉\\bm\{\\tau\}𝜸\\bm\{\\gamma\}C4WikitextHSwagPiqaWinoGLambadaArc\-ENLL0\.00010\.00–––2\.7482\.7500\.5160\.2800\.4290\.4600\.473NLL0\.00040\.00–––2\.6392\.4530\.4400\.2610\.4230\.4450\.416NLL0\.00160\.00–––2\.6482\.4690\.4580\.2630\.4060\.4410\.419KD0\.00010\.000\.70\.5–2\.7512\.7500\.5030\.2820\.4190\.4480\.475KD0\.00010\.000\.71\.0–2\.7302\.7500\.5040\.2820\.4120\.4500\.466KD0\.00010\.000\.90\.5–2\.8082\.8120\.5030\.2920\.4010\.4530\.473KD0\.00010\.000\.91\.0–2\.7422\.7030\.5070\.2750\.4220\.4490\.463KD0\.00040\.000\.70\.5–2\.6182\.4840\.4300\.2600\.3980\.4340\.419KD0\.00040\.000\.71\.0–2\.6122\.4690\.4340\.2560\.4040\.4300\.409KD0\.00040\.000\.90\.5–2\.6392\.5310\.4350\.2650\.4140\.4310\.432KD0\.00040\.000\.91\.0–2\.6302\.4380\.4400\.2560\.4180\.4320\.403KD0\.00160\.000\.70\.5–2\.6172\.4220\.4410\.2540\.4130\.4240\.413KD0\.00160\.000\.71\.0–2\.6092\.4060\.4280\.2560\.4070\.4230\.402KD0\.00160\.000\.90\.5–2\.6332\.4380\.4280\.2560\.4060\.4210\.424KD0\.00160\.000\.91\.0–2\.6292\.4380\.4210\.2530\.4100\.4160\.412HLDC0\.00010\.000\.70\.50\.052\.7502\.7810\.4980\.2760\.4280\.4590\.467HLDC0\.00010\.000\.70\.50\.102\.7502\.7810\.5040\.2750\.4250\.4480\.473HLDC0\.00010\.000\.71\.00\.052\.7292\.7190\.4970\.2740\.4290\.4530\.464HLDC0\.00010\.000\.71\.00\.102\.7302\.7500\.5100\.2690\.4360\.4530\.462HLDC0\.00010\.000\.90\.50\.052\.8092\.7810\.4990\.2740\.4070\.4500\.473HLDC0\.00010\.000\.90\.50\.102\.8082\.8280\.4920\.2840\.4210\.4520\.475HLDC0\.00010\.000\.91\.00\.052\.7422\.7190\.5070\.2720\.4060\.4630\.472HLDC0\.00010\.000\.91\.00\.102\.7412\.7810\.5060\.2810\.4190\.4600\.466HLDC0\.00040\.000\.70\.50\.052\.6182\.4840\.4460\.2640\.4130\.4270\.423HLDC0\.00040\.000\.70\.50\.102\.6182\.4690\.4320\.2560\.4090\.4350\.425HLDC0\.00040\.000\.71\.00\.052\.6112\.4220\.4390\.2520\.4030\.4320\.406HLDC0\.00040\.000\.71\.00\.102\.6112\.4690\.4330\.2560\.4090\.4290\.409HLDC0\.00040\.000\.90\.50\.052\.6392\.5310\.4380\.2670\.4010\.4270\.423HLDC0\.00040\.000\.90\.50\.102\.6392\.5310\.4360\.2620\.3930\.4260\.439HLDC0\.00040\.000\.91\.00\.052\.6282\.4380\.4460\.2550\.4130\.4270\.410HLDC0\.00040\.000\.91\.00\.102\.6282\.4690\.4460\.2540\.4180\.4280\.396HLDC0\.00160\.000\.70\.50\.052\.6172\.4220\.4200\.2630\.4000\.4320\.407HLDC0\.00160\.000\.70\.50\.102\.6182\.4220\.4280\.2620\.4000\.4270\.412HLDC0\.00160\.000\.71\.00\.052\.6082\.4380\.4300\.2510\.4050\.4250\.415HLDC0\.00160\.000\.71\.00\.102\.6102\.4380\.4310\.2500\.4130\.4210\.397HLDC0\.00160\.000\.90\.50\.052\.6342\.4380\.4280\.2570\.4070\.4230\.429HLDC0\.00160\.000\.90\.50\.102\.6352\.4380\.4270\.2580\.3980\.4300\.418HLDC0\.00160\.000\.91\.00\.052\.6282\.4380\.4230\.2490\.4180\.4260\.393HLDC0\.00160\.000\.91\.00\.102\.6282\.4380\.4240\.2540\.3990\.4310\.409HLDF0\.00010\.010\.70\.5–2\.7342\.6560\.4980\.2760\.4150\.4430\.467HLDF0\.00010\.040\.70\.5–2\.7342\.6880\.4930\.2760\.4180\.4620\.470HLDF0\.00010\.010\.71\.0–2\.7162\.6560\.4900\.2830\.4200\.4630\.462HLDF0\.00010\.040\.71\.0–2\.7172\.6880\.4910\.2740\.4320\.4640\.456HLDF0\.00010\.010\.90\.5–2\.7872\.7500\.4870\.2840\.4300\.4540\.463HLDF0\.00010\.040\.90\.5–2\.7852\.7190\.4840\.2800\.4210\.4560\.465HLDF0\.00010\.010\.91\.0–2\.7292\.6880\.5000\.2700\.4270\.4560\.460HLDF0\.00010\.040\.91\.0–2\.7292\.6560\.5030\.2740\.4230\.4650\.459HLDF0\.00040\.010\.70\.5–2\.6142\.5000\.4420\.2620\.3980\.4330\.428HLDF0\.00040\.040\.70\.5–2\.6142\.5000\.4370\.2550\.4030\.4470\.412HLDF0\.00040\.010\.71\.0–2\.6082\.5000\.4300\.2560\.3990\.4390\.414HLDF0\.00040\.040\.71\.0–2\.6082\.4840\.4450\.2510\.3970\.4400\.408HLDF0\.00040\.010\.90\.5–2\.6342\.5000\.4400\.2580\.3990\.4290\.431HLDF0\.00040\.040\.90\.5–2\.6332\.5000\.4450\.2620\.4040\.4350\.412HLDF0\.00040\.010\.91\.0–2\.6262\.5000\.4380\.2550\.4010\.4270\.412HLDF0\.00040\.040\.91\.0–2\.6262\.5000\.4440\.2520\.4040\.4370\.403HLDF0\.00160\.010\.70\.5–2\.6172\.4380\.4260\.2510\.4020\.4290\.423HLDF0\.00160\.040\.70\.5–2\.6172\.4380\.4350\.2530\.4060\.4260\.408HLDF0\.00160\.010\.71\.0–2\.6072\.4220\.4260\.2610\.4040\.4280\.409HLDF0\.00160\.040\.71\.0–2\.6082\.4220\.4240\.2450\.3990\.4170\.396HLDF0\.00160\.010\.90\.5–2\.6332\.4530\.4350\.2570\.3910\.4280\.420HLDF0\.00160\.040\.90\.5–2\.6322\.4380\.4240\.2580\.4050\.4250\.423HLDF0\.00160\.010\.91\.0–2\.6272\.4380\.4270\.2550\.4180\.4300\.403HLDF0\.00160\.040\.91\.0–2\.6272\.4380\.4410\.2600\.4110\.4280\.408