Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

# Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing
Source: [https://arxiv.org/html/2610.20914](https://arxiv.org/html/2610.20914)
###### Abstract

Distillation of reasoning chains from Large Language Models \(LLMs\) into their smaller siblings is a standard training approach\. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance\. We propose a recursive offline distillation method combining two components: entropy\-based sampling for selective distillation and answer mixing\. Specifically, for three open models \(3\-4B\), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model \(70\-72B\) and, next, supplement thinking traces with single\-token ground truth answers\. Our approach consistently outperforms naive distillation with comparable data budgets \(57\.15±1\.0157\.15\\pm 1\.01vs 51\.61 on average\) and offers a much steeper learning curve\. Additionally, we present a comparative analysis of methods to generate and normalize distillation traces\. We publish our code and data111[https://github\.com/LabARSS/recursive\_offline\_distillation](https://github.com/LabARSS/recursive_offline_distillation)\.

## 1Introduction

Modern LLMs come in different sizes, and their performance is often correlated with the number of parameters\. The largest of them demonstrate exceptional results and span 1\-2 trillion parameters\. Further domain adaptation\([Cheng et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib6);[Lu et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib7)\)or even inference at this scale requires massive computational resources\([Wan et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib4)\)unavailable to most practitioners\. It prompts the need for capable smaller models\([Chen and Varoquaux, 2026](https://arxiv.org/html/2610.20914#bib.bib5)\)that can be fine\-tuned and deployed in resource\-constrained environments\.

![Refer to caption](https://arxiv.org/html/2610.20914v1/img/recursive_caft_scheme_v2.png)Figure 1:Recursive offline distillation schemeA prominent approach for small model training is distillation from a larger teacher model\([Hinton et al\., 2015](https://arxiv.org/html/2610.20914#bib.bib8);[Sanh et al\., 2020](https://arxiv.org/html/2610.20914#bib.bib10);[Xu et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib11)\)\. Many methods rely on having access to the online teacher\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib12);[Jiao et al\., 2020](https://arxiv.org/html/2610.20914#bib.bib13);[Gu et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib9)\), which is not always possible\([Zhou et al\., 2023](https://arxiv.org/html/2610.20914#bib.bib14);[Ryskulov et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib15)\)\. It forces practitioners to explore offline distillation, where the student model is trained on the teacher’s pre\-generated text output\([Li et al\., 2023](https://arxiv.org/html/2610.20914#bib.bib16);[Hsieh et al\., 2023](https://arxiv.org/html/2610.20914#bib.bib17)\)\.

We propose a fully automated recursive pipeline for offline distillation \(Figure[1](https://arxiv.org/html/2610.20914#S1.F1), described in[3](https://arxiv.org/html/2610.20914#S3)\) based on entropy sampling and answer mixing\. The method consistently outperforms naive distillation \(57\.15±1\.0157\.15\\pm 1\.01vs 51\.61 on average\), and it is designed to work in resource\-constrained environments\.

We contribute a family of entropy\-based samplers and a lightweight data mix\. Additionally, we provide a comparative analysis of augmentation strategies for synthetic reasoning chains when the teacher model does not perform perfectly in the target distillation domain and conduct sensitivity studies of the synthetic trace generation and normalization methods and sampling strategies\.

## 2Related Work

Recent work improved distillation through training objectives\([Ko et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib18);[Xu et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib19)\), data selection\([Just et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib20);[Li et al\., 2025](https://arxiv.org/html/2610.20914#bib.bib23);[Yang et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib24)\), and curriculum learning\([Jiang et al\., 2025](https://arxiv.org/html/2610.20914#bib.bib21);[Zhang and Liu, 2026](https://arxiv.org/html/2610.20914#bib.bib25)\)\. Joint supervision with rationales and labels improves data efficiency\([Hsieh et al\., 2023](https://arxiv.org/html/2610.20914#bib.bib17)\)\. Curriculum benefits vary with model capability, task complexity, and the choice of difficulty metric\([Jia et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib22)\)\. ELAD\([Zhang et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib38)\)selects uncertain student explanations for teacher revision\. Complexity\-aware fine\-tuning\([Goncharov et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib31)\)selects reasoning examples by single\-token entropy, but only once\. We recursively sample pre\-generated traces using student\-proxy entropy differences\.

Trace synthesis includes explaining known solutions\([Li and Mooney, 2024](https://arxiv.org/html/2610.20914#bib.bib37);[Wang et al\., 2025a](https://arxiv.org/html/2610.20914#bib.bib41)\)and learning from corrections\([An et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib40);[Yang et al\., 2025](https://arxiv.org/html/2610.20914#bib.bib39)\)\. We compare direct generation and various augmentations on the same task under constrained trace lengths\.

## 3Method

##### Entropy sampling\.

[Goncharov et al\. \(2026\)](https://arxiv.org/html/2610.20914#bib.bib31)show that a single\-token entropy \(Figure[4](https://arxiv.org/html/2610.20914#A5.F4)\) is a reasonable complexity predictor for a given entry in a question\-answering dataset\. We assume that it approximates the total uncertainty\. With scaling laws\([Kaplan et al\., 2020](https://arxiv.org/html/2610.20914#bib.bib34)\)in mind, we hypothesize that larger models should have lower total uncertainty\. Therefore, in comparison to a small student model \(3\-4B\), its larger sibling \(proxy\) \(70\-72B222This particular size is chosen based on the imposed resource constraints \(80GB VRAM\) and open model availability\) could have higher confidence \(lower entropy\) for a variety of data entries, and help the student to identify a useful knowledge gap\. We propose*entropy gain*HS​\(x\)−HP​\(x\)H\_\{S\}\(x\)\-H\_\{P\}\(x\)\(similar to[Mindermann et al\. \(2022\)](https://arxiv.org/html/2610.20914#bib.bib1);[Lin et al\. \(2025\)](https://arxiv.org/html/2610.20914#bib.bib2)\) as the primary heuristic, but evaluate alternatives as well\.

The heuristic is inspired by epistemic\-aleatoric uncertainty decomposition\([Wang et al\., 2025b](https://arxiv.org/html/2610.20914#bib.bib26);[Depeweg et al\., 2018](https://arxiv.org/html/2610.20914#bib.bib32);[Malinin and Gales, 2021](https://arxiv.org/html/2610.20914#bib.bib33)\)\. For a questionxx, assumeH⁡\(x\)=A⁡\(x\)\+E⁡\(x\)H\(x\)=A\(x\)\+E\(x\), whereHSH\_\{S\},HPH\_\{P\}\- single\-token entropies \(student and proxy\),AA\- shared aleatoric component, andESE\_\{S\},EPE\_\{P\}\- model\-specific epistemic components\. If its residual epistemic uncertainty satisfies\|ES​\(x\)−EP​\(x\)\|≤ε⁡\(x\)\|E\_\{S\}\(x\)\-E\_\{P\}\(x\)\|\\leq\\varepsilon\(x\), then\|HS​\(x\)−HP​\(x\)\|≤ε⁡\(x\)\|H\_\{S\}\(x\)\-H\_\{P\}\(x\)\|\\leq\\varepsilon\(x\)\. Further discussion in Appendix[I](https://arxiv.org/html/2610.20914#A9)\.

##### Answer Mixing\.

We supplement 1024 reasoning chains with 256 single\-token answer examples per training epoch\. The mix preserves model’s capabilities for non\-thinking answers, which is central to entropy sampling, and offers a surprising substantial performance uplift\.

Algorithm 1Recursive offline distillation via entropy gain sampling and answer mixing1:Student

SS, proxy model

PP, epochs

TT
2:Questions

DD, ground\-truth answers

yxy\_\{x\}, and precomputed reasoning traces

rxr\_\{x\}
3:Trained student

SS
4:for all

x∈Dx\\in Ddo

5:

HP​\(x\)←Entropy​\(P,x\)H\_\{P\}\(x\)\\leftarrow\\textsc\{Entropy\}\(P,x\)
6:endfor

7:for

t=1,…,Tt=1,\\ldots,Tdo

8:for all

x∈Dx\\in Ddo

9:

HS​\(x\)←Entropy​\(S,x\)H\_\{S\}\(x\)\\leftarrow\\textsc\{Entropy\}\(S,x\)
10:

EG⁡\(x\)←max⁡\{HS​\(x\)−HP​\(x\),0\}\\mathrm\{EG\}\(x\)\\leftarrow\\max\\\{H\_\{S\}\(x\)\-H\_\{P\}\(x\),0\\\}
11:endfor

12:

Q←WeightedSample​\(D,EG,1024\)Q\\leftarrow\\textsc\{WeightedSample\}\(D,\\mathrm\{EG\},1024\)
13:

QA←first​256​questions in​QQ\_\{A\}\\leftarrow\\text\{first \}256\\text\{ questions in \}Q
14:

BR←\[\(x,rx\):x∈Q\]B\_\{R\}\\leftarrow\[\(x,r\_\{x\}\):x\\in Q\]
15:

BA←\[\(x,yx\):x∈QA\]B\_\{A\}\\leftarrow\[\(x,y\_\{x\}\):x\\in Q\_\{A\}\]
16:

S←Train​\(S,BR∥BA\)S\\leftarrow\\textsc\{Train\}\(S,B\_\{R\}\\mathbin\{\\\|\}B\_\{A\}\)
17:endfor

18:return

SS

##### The proposed pipeline

consists of the following steps \(Figure[1](https://arxiv.org/html/2610.20914#S1.F1), Algorithm[1](https://arxiv.org/html/2610.20914#alg1)\): \(1\) prompt the larger model \(proxy\) to answer all questions in the dataset with a single token once \(or an ensemble, averaging entropies\), collect entropy; \(2\) for each training epoch, \(2\.1\) prompt the student model to answer all questions in the dataset with a single token, collect entropy; \(2\.2\) calculate entropy gainE​G​\(x\)=max⁡\{HS​\(x\)−HP​\(x\),0\}EG\(x\)=\\max\\\{H\_\{S\}\(x\)\-H\_\{P\}\(x\),0\\\}; \(2\.3\.1\) sample 1024 synthetic reasoning chains without replacement, weighted by positive entropy gain; \(2\.3\.2\) sample 256 single\-token answers similarly, as a subset of the previous sample; \(2\.4\) concatenate samples; \(2\.5\) train the student using LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.20914#bib.bib35)\)on reasoning traces first, followed by single\-token responses\.

## 4Results

\(a\)4096\-token reasoning cap\(b\)2048\-token reasoning cap\(c\)Student entropy ROC AUC
Figure 2:Distillation pipeline results across three models \(average\) on MMLU\-ProMethodAnswer MixLLaMA 3\.2 3BQwen 2\.5 3BPhi4\-miniMeanDirect reasoning traceNo46\.09 / 45\.5551\.45 / 50\.3757\.27 / 62\.4351\.61 / 52\.78Corrected answerNo47\.80 / 47\.2650\.87 / 49\.4258\.81 / 61\.2652\.49 / 52\.65Student entropy \(deterministic\)Yes48\.84 /49\.0955\.86/55\.4958\.02 /63\.3054\.24 /55\.96Entropy gain \(deterministic\)Yes49\.58/ 49\.0053\.16 / 53\.3360\.22/ 62\.4354\.32/ 54\.92Student entropy \(once,k=1024k=1024\)No41\.35 / 40\.1139\.44 / 39\.0750\.17 / 52\.8743\.65 / 44\.01Student entropy \(once,k=2048k=2048\)No42\.19 / 41\.0643\.18 / 43\.0253\.20 / 52\.6246\.19 / 45\.57Student entropy \(once,k=3072k=3072\)No42\.02 / 42\.8142\.85 / 40\.8656\.77 / 55\.9047\.22 / 46\.52Entropy gain \(sampled\)Yes52\.58±1\.94\\pm\\,1\.94/52\.08±2\.28\\pm\\,2\.2856\.64±3\.79\\pm\\,3\.79/56\.86±2\.32\\pm\\,2\.3262\.22±2\.42\\pm\\,2\.42/64\.07±4\.52\\pm\\,4\.5257\.15±1\.01\\pm\\,1\.01/57\.67±1\.19\\pm\\,1\.19Table 1:Distillation pipeline last\-epoch accuracy \(4096\-token / 2,048\-token thinking budget\) on MMLU\-Pro\.### 4\.1Experimental Setup

##### Dataset:

Multiple\-choice question\-answering dataset MMLU Pro\([Wang et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib30)\)\. It has questions of different complexities from 14 domains: up to ten options each, one correct option\. Random train/test split \- 80/20\.

##### LLMs:

Proxy: ensemble of two medium models333All models and datasets are published under permissive licenses that allow them to be used for research purposes\(averaging their entropies\) \- LLaMA 3\.3 70B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib29)\), Qwen 2\.5 72B\([Qwen\-Team, 2024](https://arxiv.org/html/2610.20914#bib.bib27)\)\. Students: Qwen2\.5 3B\([Qwen\-Team, 2024](https://arxiv.org/html/2610.20914#bib.bib27)\), Phi\-4\-Mini\([Microsoft et al\., 2025](https://arxiv.org/html/2610.20914#bib.bib28)\), LLaMA 3\.2 3B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.20914#bib.bib29)\)\. Reasoning trace source: DeepSeek V4 Pro and Flash\([DeepSeek\-AI et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib3)\)via API\.

##### Hardware:

We run all experiments444Training hyperparameters can be found in Appendix[A](https://arxiv.org/html/2610.20914#A1)\.555We used AI coding agents for a substantial part of the experimental and analytical codeon a single A100/H100666We had to switch cards depending on availabilityGPU with 80GB of VRAM to validate the method in resource\-constrained environments\.

#### 4\.1\.1Our method and results

We evaluate the proposed pipeline for 200 epochs \(205k questions seen\) and 2 thinking budgets \(2048 and 4096 tokens\)\. It is compared under similar data budgets with: *Direct reasoning trace, Corrected answer*\- naive distillation on the most promising types of reasoning traces \(see[4\.3](https://arxiv.org/html/2610.20914#S4.SS3)\); *Student entropy \(deterministic\), Entropy gain \(deterministic\)*\- alternative pipelines with weighted sampling replaced with greedy sampling; *Student entropy \(once, k=x\)*\- distillation executed against k samples selected once by student entropy \(not recursively at each epoch\)\.

Notably, the proposed data mix increases the number of training tokens by less than 0\.01% \(256 extra tokens compared to≈\\approx2\.5m\) or 9% accounting for the system prompt in the context \(≈\\approx250k extra tokens compared to≈\\approx2,75m\)\.

All randomly sampled methods use 3 random seeds, reporting±\\pm95% confidence intervals, and deterministic methods are evaluated once with greedy sampling\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2610.20914#S4.T1)and Figure[2](https://arxiv.org/html/2610.20914#S4.F2)show the results for MMLU Pro\. We see that the proposed method produces a significant improvement over naive distillation \(up to10%10\\%gain\)\. For both reasoning budgets, it provides the best result on average across three models in comparison with other approaches\. Greedy sampling demonstrates subpar and less stable curves, though surpassing the baselines\. Greedy distillation based on student entropy sampled once\([Goncharov et al\., 2026](https://arxiv.org/html/2610.20914#bib.bib31)\)shows signs of severe degradation after the initial uplift revealing the fragility of the approach\.

We also note that our method helps with the model’s single\-token entropy correctness discrimination, increasing ROC AUC for the first half of training\. Shortly after, it starts declining, which is loosely correlated with the method’s plateau\.

\(a\)by truncation type\(b\)by length\(c\)by generation method
Figure 3:Comparison of synthetic traces across three models \(average\)
### 4\.3Synthetic Trace Generation

In resource\-constrained environments, available VRAM imposes a strict limitation on the length of distilled reasoning chains\. Once we start truncating them, we face a new challenge: which part of the trace contains the most useful signal? Finally, how exactly do we prompt the teacher model to generate reasoning traces?

We evaluate reasoning traces by: \(a\)*truncation type*\- which part of the trace we keep when it exceeds the maximum length; \(b\)*length*\- the optimal training truncation limit \(characters\) and evaluation thinking budget \(tokens\); \(c\)*generation method*\(below\)\.

Generation methods \(details in Appendix[F](https://arxiv.org/html/2610.20914#A6)\): \(1\)*Direct reasoning trace*\- baseline prompt to just answer the question; \(2\)*Direct reasoning trace \(correct only\)*\- same as \(1\), but we keep only traces with correct answers; \(3\)*Explained answer*\- we give a correct answer to the model and prompt it to explain it; \(4\)*Direct reasoning trace \(with explained\)*\- same as \(1\), but replace incorrect traces with \(3\); \(5\)*Corrected answer*\- same as \(1\), but for incorrect traces, we give the model correct answers and ask it to update its reasoning\.

Figure[3](https://arxiv.org/html/2610.20914#S4.F3)shows that models have a slight preference for head truncation, medium to short training chains and thinking budgets\. Surprisingly, longer reasoning chains and greater thinking budgets lead to notable performance decreases\. As to the generation methods, we see that augmented reasoning traces lead to elevated performance, while the consistent data leakage results in degradation\. Based on the findings, we use 8,192\-character head truncation and*Corrected answer*generation for the pipeline\.

### 4\.4Sensitivity Study

To attribute performance gains to either the proposed data mix or the entropy\-based score and sampler, we compare*entropy gain*with: \(1\) alternative pipelines where we sample according to*student entropy*and*discounted entropy gain*\(see below\); \(2\) an alternative pipeline where we use a random sampler, but with our data mix\.

*Discounted entropy gain*is defined asmax⁡\{HS​\(x\)−k⁡\(t\)∗HP​\(x\),0\},k⁡\(t\)∈\[0,1\]\\max\\\{H\_\{S\}\(x\)\-k\(t\)\*H\_\{P\}\(x\),0\\\},k\(t\)\\in\[0,1\], wherettis the training epoch\. We setk⁡\(t\)k\(t\)initially to 0\.8 and decrease it according to a schedule, hypothesizing that, as the student undergoes training, the static medium model’s advantage disappears\.

Table[2](https://arxiv.org/html/2610.20914#S4.T2)shows that, across three models on average, entropy gain provides better results, which makes the heuristic promising, and we adopt it as the primary sampler for the method\. At the same time, random sampling performs exceptionally well \(confirming findings of[Okanovic et al\. \(2023\)](https://arxiv.org/html/2610.20914#bib.bib36)\), which points to the proposed data mix as a major contributor to the gains\.

Table 2:Distillation pipeline last\-epoch accuracy for different sampling methods \(4096\-token thinking budget\)

## 5Conclusion and Discussion

This paper introduces a recursive offline distillation method rooted in entropy sampling and a lightweight answer mix \(57\.15±1\.0157\.15\\pm 1\.01mean accuracy across three students on MMLU\-Pro vs51\.6151\.61for direct distillation\)\. Experiments show that adding a small portion of direct answers to reasoning chains provides a major uplift\. The entropy gain sampler contributes additional gains\. Corrected reasoning chains, and, surprisingly,*shorter*traces and reasoning budgets perform better\. Exploration of more rigorous uncertainty estimation and nature of the data mix effect remains future work\.

## Limitations

- •The method is tested only with small models\. We do not know how well it is going to generalize to larger LLMs\. Additionally, it might be challenging to find a suitable proxy for larger students\.
- •We use a single dataset\. We do not know how well it translates to other settings\.
- •Alternative uncertainty estimations, such as ensemble agreement and others, are left unexplored\.
- •We did not test and compare multiple sizes of proxies and their impact on performance\.
- •Different compositions of the data mix are not tested due to lack of resources\.
- •We tested only a single schedule for*Discounted Entropy Gain*\. Alternatives remain to be explored\.
- •Some low\-resource environments might have even more constraints and therefore struggle to fit a suitable proxy into VRAM\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21246–21263\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Anet al\.\(2024\)S\. An, Z\. Ma, Z\. Lin, N\. Zheng, J\. Lou, and W\. ChenLearning from mistakes makes llm better reasoner\.External Links:2310\.20689,[Link](https://arxiv.org/abs/2310.20689)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p2.1)\.
- Chen and Varoquaux \(2026\)L\. Chen and G\. VaroquauxWhat is the role of small models in the llm era: a survey\.External Links:2409\.06857,[Link](https://arxiv.org/abs/2409.06857)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p1.1)\.
- Chenget al\.\(2024\)D\. Cheng, S\. Huang, and F\. WeiAdapting large language models via reading comprehension\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 48624–48652\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/d51cd79a85833b022841f7a2383b32d3-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p1.1)\.
- DeepSeek\-AIet al\.\(2026\)DeepSeek\-AI, A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling, C\. Lu, C\. Zhao, C\. Deng, C\. Hou, C\. Xu, C\. Shao, C\. Ruan, C\. Sun, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Li, D\. Ji, E\. Li, F\. Wei, F\. Lin, F\. Yuan, F\. Xia, F\. Dai, G\. Hao, G\. Chen, G\. Cao, G\. Meng, G\. Li, H\. Yu, H\. Zhang, H\. Xu, H\. Li, H\. Liang, H\. Zhang, H\. Luo, H\. Wei, H\. Yuan, H\. Zhang, H\. Luo, H\. Chen, H\. Ji, H\. Zhang, H\. Ding, H\. Tang, H\. Cao, H\. Gao, H\. Qu, H\. Zeng, J\. Yang, J\. Zhu, J\. Luo, J\. Song, J\. Yu, J\. Huang, J\. Cai, J\. Liang, J\. Zhou, J\. Ye, J\. Li, J\. Xu, J\. Hu, J\. Yang, J\. Chen, J\. Yan, J\. Chen, J\. Zhou, J\. Xiang, J\. Yuan, J\. Cheng, J\. Zhou, J\. Zhu, J\. Yu, J\. Sun, J\. Ran, J\. Jiang, J\. Qiu, J\. Li, J\. Zheng, J\. Song, K\. Dong, K\. Gao, K\. Guan, K\. Zhou, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Wang, L\. Xia, L\. Zhang, L\. Zhao, L\. Guo, L\. Luo, L\. Ma, L\. Zhu, L\. Wang, L\. Cai, L\. Zhang, L\. Chen, M\. Di, M\. Xu, M\. Mei, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Zhou, M\. Han, N\. Wang, P\. Huang, P\. Wang, P\. Cong, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Li, Q\. Chen, Q\. Du, Q\. Jiang, R\. Tian, R\. Xu, R\. Lu, R\. Xu, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Chen, R\. Yin, R\. Xu, R\. Shen, R\. Zhang, R\. Chen, S\. Liu, S\. Lu, S\. Sun, S\. Zhou, S\. Chen, S\. Cai, S\. Nie, S\. Wu, S\. Chen, S\. Hu, S\. Liu, S\. Hu, S\. Ma, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. Yu, S\. Zhou, T\. Ni, T\. Yun, T\. Jin, T\. Pei, T\. Ye, T\. Lin, T\. Ji, T\. Cui, T\. Yue, T\. Yu, T\. Wang, W\. Zhang, W\. Xiao, W\. Zeng, W\. An, W\. Zhao, W\. Liu, W\. Liang, W\. Pang, W\. Luo, W\. Yao, W\. Gao, W\. Yang, W\. Huang, W\. Hou, W\. Zhang, W\. Ma, X\. Gao, X\. He, X\. Wang, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Chen, X\. Zhang, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Liu, X\. Yu, X\. Li, X\. Yang, X\. Zhang, X\. Chen, X\. Wang, X\. Su, X\. Chen, X\. Lin, X\. Fu, Y\. Yan, Y\. Wang, Y\. Ma, Y\. Luo, Y\. Zhang, Y\. Xu, Y\. Ma, Y\. Huang, Y\. Li, Y\. Li, Y\. Xu, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Qian, Y\. Shao, Y\. Yu, Y\. Zhang, Y\. Ding, Y\. Shi, Y\. Wu, Y\. Xiong, Y\. Ma, Y\. He, Y\. Tang, Y\. Zhou, Y\. Luo, Y\. Zhong, Y\. Piao, Y\. Wang, Y\. Zhang, Y\. Chen, Y\. Tan, Y\. Wei, Y\. Ma, Y\. Liu, Y\. Yang, Y\. Guo, Y\. Wu, Y\. Wu, Y\. Li, Y\. Cheng, Y\. Ou, Y\. Xu, Y\. Li, Y\. Wang, Y\. Yang, Y\. Xu, Y\. Wu, Y\. Meng, Y\. Zou, Y\. Zha, Y\. Xiong, Y\. Chen, Y\. Lin, Y\. Cao, Y\. Wang, Y\. Zhang, Y\. Yan, Y\. Lin, Y\. Gu, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. Zhou, Y\. Huang, Z\. Wu, Z\. Wang, Z\. Zhao, Z\. Ren, Z\. Zhang, Z\. Sha, Z\. Fu, Z\. Ju, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Gao, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Huang, Z\. Chen, Z\. Wu, Z\. Ren, Z\. Wu, Z\. Li, Z\. Zhang, Z\. Xu, Z\. Wang, Z\. Qu, Z\. Gu, Z\. Zhu, Z\. Li, Z\. Zhang, Z\. Xie, Z\. Gao, Z\. Wan, Z\. Pan, and Z\. YaoDeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§4\.1](https://arxiv.org/html/2610.20914#S4.SS1.SSS0.Px2.p1.1)\.
- Depeweget al\.\(2018\)S\. Depeweg, J\. Hernandez\-Lobato, F\. Doshi\-Velez, and S\. UdluftDecomposition of uncertainty in Bayesian deep learning for efficient and risk\-sensitive learning\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1184–1193\.External Links:[Link](https://proceedings.mlr.press/v80/depeweg18a.html)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p2.1)\.
- Goncharovet al\.\(2026\)A\. Goncharov, D\. Vyazhev, P\. Sychev, E\. Khalafyan, and A\. ZaytsevComplexity\-aware fine\-tuning\.External Links:2506\.21220,[Link](https://arxiv.org/abs/2506.21220)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1),[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.20914#S4.SS2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2610.20914#S4.SS1.SSS0.Px2.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: knowledge distillation of large language models\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 32694–32717\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/8ac015d409635f196f9e3e9dcfb9a94e-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Hsiehet al\.\(2023\)C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. PfisterDistilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 8003–8017\.External Links:[Link](https://aclanthology.org/2023.findings-acl.507/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.507)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1),[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px3.p1.1)\.
- Jiaet al\.\(2026\)Y\. Jia, C\. Zhang, X\. Diao, X\. Yuan, Z\. Ouyang, C\. Ma, and S\. VosoughiWhat makes a good curriculum? disentangling the effects of data ordering on LLM mathematical reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 34472–34488\.External Links:[Link](https://aclanthology.org/2026.acl-long.1591/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1591),ISBN 979\-8\-89176\-390\-6Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Jianget al\.\(2025\)W\. Jiang, Y\. Lu, H\. Lin, X\. Han, and L\. SunTeach small models to reason by curriculum distillation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 7412–7422\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.376/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.376),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Jiaoet al\.\(2020\)X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. LiuTinyBERT: distilling BERT for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4163–4174\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.372/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.372)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Justet al\.\(2026\)H\. A\. Just, M\. Ko, and R\. JiaThe signal is in the steps: local scoring for reasoning data selection\.InProceedings of the 43rd International Conference on Machine Learning,T\. Zhang, M\. Dudik, M\. Jaggi, A\. Agarwal, S\. Li, D\. Schuurmans, J\. Zhu, F\. Berkenkamp, H\. Dong, and A\. Bietti \(Eds\.\),Proceedings of Machine Learning Research, Vol\.306,pp\. 55125–55155\.External Links:[Link](https://proceedings.mlr.press/v306/just26a.html)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p1.1)\.
- Koet al\.\(2024\)J\. Ko, S\. Kim, T\. Chen, and S\. YunDistiLLM: towards streamlined distillation for large language models\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 24872–24895\.External Links:[Link](https://proceedings.mlr.press/v235/ko24c.html)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Li and Mooney \(2024\)J\. Li and R\. MooneyDistilling algorithmic reasoning from LLMs via explaining solution programs\.External Links:2404\.08148,[Document](https://dx.doi.org/10.48550/arXiv.2404.08148),[Link](https://arxiv.org/abs/2404.08148)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p2.1)\.
- Liet al\.\(2023\)L\. H\. Li, J\. Hessel, Y\. Yu, X\. Ren, K\. Chang, and Y\. ChoiSymbolic chain\-of\-thought distillation: small models can also “think” step\-by\-step\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 2665–2679\.External Links:[Link](https://aclanthology.org/2023.acl-long.150/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.150)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Liet al\.\(2025\)Y\. Li, Y\. Emad, K\. Padthe, J\. Lanchantin, W\. Yuan, T\. Nguyen, J\. Weston, S\. Li, D\. Wang, I\. Kulikov, and X\. LiNaturalThoughts: selecting and distilling reasoning traces for general reasoning tasks\.External Links:2507\.01921,[Link](https://arxiv.org/abs/2507.01921)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Linet al\.\(2025\)Z\. Lin, Z\. Gou, Y\. Gong, X\. Liu, Y\. Shen, R\. Xu, C\. Lin, Y\. Yang, J\. Jiao, N\. Duan, and W\. ChenRho\-1: not all tokens are what you need\.External Links:2404\.07965,[Link](https://arxiv.org/abs/2404.07965)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2024\)W\. Lu, R\. K\. Luu, and M\. J\. BuehlerFine\-tuning large language models for domain adaptation: exploration of training strategies, scaling, model merging and synergistic capabilities\.External Links:2409\.03444,[Link](https://arxiv.org/abs/2409.03444)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p1.1)\.
- Malinin and Gales \(2021\)A\. Malinin and M\. GalesUncertainty estimation in autoregressive structured prediction\.External Links:2002\.07650,[Link](https://arxiv.org/abs/2002.07650)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p2.1)\.
- Microsoftet al\.\(2025\)Microsoft, :, A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen, D\. Chen, D\. Chen, J\. Chen, W\. Chen, Y\. Chen, Y\. Chen, Q\. Dai, X\. Dai, R\. Fan, M\. Gao, M\. Gao, A\. Garg, A\. Goswami, J\. Hao, A\. Hendy, Y\. Hu, X\. Jin, M\. Khademi, D\. Kim, Y\. J\. Kim, G\. Lee, J\. Li, Y\. Li, C\. Liang, X\. Lin, Z\. Lin, M\. Liu, Y\. Liu, G\. Lopez, C\. Luo, P\. Madan, V\. Mazalov, A\. Mitra, A\. Mousavi, A\. Nguyen, J\. Pan, D\. Perez\-Becker, J\. Platin, T\. Portet, K\. Qiu, B\. Ren, L\. Ren, S\. Roy, N\. Shang, Y\. Shen, S\. Singhal, S\. Som, X\. Song, T\. Sych, P\. Vaddamanu, S\. Wang, Y\. Wang, Z\. Wang, H\. Wu, H\. Xu, W\. Xu, Y\. Yang, Z\. Yang, D\. Yu, I\. Zabir, J\. Zhang, L\. L\. Zhang, Y\. Zhang, and X\. ZhouPhi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.External Links:2503\.01743,[Link](https://arxiv.org/abs/2503.01743)Cited by:[§4\.1](https://arxiv.org/html/2610.20914#S4.SS1.SSS0.Px2.p1.1)\.
- Mindermannet al\.\(2022\)S\. Mindermann, J\. M\. Brauner, M\. T\. Razzak, M\. Sharma, A\. Kirsch, W\. Xu, B\. Höltgen, A\. N\. Gomez, A\. Morisot, S\. Farquhar, and Y\. GalPrioritized training on points that are learnable, worth learning, and not yet learnt\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 15630–15649\.External Links:[Link](https://proceedings.mlr.press/v162/mindermann22a.html)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p1.1)\.
- Okanovicet al\.\(2023\)P\. Okanovic, R\. Waleffe, V\. Mageirakos, K\. E\. Nikolakakis, A\. Karbasi, D\. Kalogerias, N\. M\. Gürel, and T\. RekatsinasRepeated random sampling for minimizing the time\-to\-accuracy of learning\.External Links:2305\.18424,[Link](https://arxiv.org/abs/2305.18424)Cited by:[§4\.4](https://arxiv.org/html/2610.20914#S4.SS4.p3.1)\.
- Qwen\-Team \(2024\)Qwen\-TeamQwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§4\.1](https://arxiv.org/html/2610.20914#S4.SS1.SSS0.Px2.p1.1)\.
- Ryskulovet al\.\(2026\)B\. Ryskulov, I\. García\-Ferrero, D\. Montero, D\. Jansen, A\. Hashemi, J\. R\. Garcia, A\. Tiene, and R\. OrúsEfficient knowledge distillation for llms: offline top\-k logits and a fused chunked kl loss\.External Links:2608\.03796,[Link](https://arxiv.org/abs/2608.03796)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Sanhet al\.\(2020\)V\. Sanh, L\. Debut, J\. Chaumond, and T\. WolfDistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.External Links:1910\.01108,[Link](https://arxiv.org/abs/1910.01108)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Wanet al\.\(2024\)Z\. Wan, X\. Wang, C\. Liu, S\. Alam, Y\. Zheng, J\. Liu, Z\. Qu, S\. Yan, Y\. Zhu, Q\. Zhang, M\. Chowdhury, and M\. ZhangEfficient large language models: a survey\.External Links:2312\.03863,[Link](https://arxiv.org/abs/2312.03863)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, H\. Que, Q\. Xu, M\. Liu, W\. Zhou, J\. Feng, W\. Zhong, W\. Ye, T\. Yang, W\. Huang, G\. Zhang, and F\. LinReverse\-engineered reasoning for open\-ended generation\.External Links:2509\.06160,[Link](https://arxiv.org/abs/2509.06160)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p2.1)\.
- Wanget al\.\(2025b\)T\. Wang, Y\. Wang, J\. Zhou, B\. Peng, X\. Song, C\. Zhang, X\. Sun, Q\. Niu, J\. Liu, S\. Chen, K\. Chen, M\. Li, P\. Feng, Z\. Bi, M\. Liu, Y\. Zhang, C\. Fei, C\. H\. Yin, and L\. K\. YanFrom aleatoric to epistemic: exploring uncertainty quantification techniques in artificial intelligence\.External Links:2501\.03282,[Link](https://arxiv.org/abs/2501.03282)Cited by:[§3](https://arxiv.org/html/2610.20914#S3.SS0.SSS0.Px1.p2.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.External Links:2406\.01574,[Link](https://arxiv.org/abs/2406.01574)Cited by:[§4\.1](https://arxiv.org/html/2610.20914#S4.SS1.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026\)S\. Xu, C\. Peng, J\. Long, W\. Xu, W\. Chu, and Y\. QiHarnessing negative signals: reinforcement distillation from teacher data for LLM reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 1618–1639\.External Links:[Link](https://aclanthology.org/2026.acl-long.74/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.74),ISBN 979\-8\-89176\-390\-6Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Xuet al\.\(2024\)X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. ZhouA survey on knowledge distillation of large language models\.External Links:2402\.13116,[Link](https://arxiv.org/abs/2402.13116)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.
- Yanget al\.\(2026\)C\. Yang, X\. Lin, X\. Wu, C\. Xu, X\. Jiang, H\. Liu, H\. Xiong, and J\. GuoSelect2Reason: efficient instruction\-tuning data selection for long\-CoT reasoning\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 6656–6671\.External Links:[Link](https://aclanthology.org/2026.findings-acl.331/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.331),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Yanget al\.\(2025\)L\. Yang, Z\. Yu, T\. Zhang, M\. Xu, J\. E\. Gonzalez, B\. CUI, and S\. YANSuperCorrect: advancing small llm reasoning with thought template distillation and self\-correction\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 3386–3418\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/0967d7c8b171dd81b77c43067c02bebf-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p2.1)\.
- Zhang and Liu \(2026\)M\. Zhang and L\. LiuCalibrated progressive distillation: co\-designing curriculum and target mixing for knowledge distillation of large language models\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 6757–6776\.External Links:[Link](https://aclanthology.org/2026.findings-acl.335/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.335),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Zhanget al\.\(2024\)Y\. Zhang, B\. Pan, C\. Ling, Y\. Hu, and L\. ZhaoELAD: explanation\-guided large language models active distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 4463–4475\.External Links:[Link](https://aclanthology.org/2024.findings-acl.264/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.264)Cited by:[§2](https://arxiv.org/html/2610.20914#S2.p1.1)\.
- Zhouet al\.\(2023\)Q\. Zhou, Z\. Yang, P\. Li, and Y\. LiuBridging the gap between decision and logits in decision\-based knowledge distillation for pre\-trained language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13234–13248\.External Links:[Link](https://aclanthology.org/2023.acl-long.738/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.738)Cited by:[§1](https://arxiv.org/html/2610.20914#S1.p2.1)\.

## Appendix AExperimental Hyperparameters

## Appendix BReporting format

For experiments with random sampling, for each model, we report the meanx¯\\bar\{x\}across training seeds with a two\-sided 95% Student’sttconfidence interval\. For the mean accuracy across models, we weight model means equally and treat them as independent\.

For aggregate ROC AUC curves, we instead average all three models within each seed, then apply the Student’sttinterval across these seed averages\.

For deterministic experiments we report results from a single run\.

## Appendix CReasoning distillation approach

Originally, all students are non\-reasoning models\. We adopt special tokens,<think\>and</think\>, to teach them separate reasoning from the final answers\. We extend student’s tokenizer and initialize corresponding embedding rows from the mean of the existing rows\.

For reasoning evaluation, we append<think\>to the assistant prompt\. For single\-token entropy estimation, we omit the thinking prefix and expect a model to answer with a single token without thinking\. The 256 direct\-answer examples in each mixed\-training epoch help preserve this latter capability while the 1,024 reasoning examples teach reasoning mode\.

##### Example\.

Consider the toy question

> A rectangle has sides of length 3 and 2\. What is its perimeter? a\. 6; b\. 10\.

A reasoning target is:

> <think\>There are two sides of each length, so the perimeter is2​\(3\+2\)=102\(3\+2\)=10\.</think\>b

A single\-token estimation target is:

> b

## Appendix DAccuracy gains beyond answer formatting

What do single\-token answers contribute? Is it genuine quality uplift or merely formatting?

The evaluation budget only limits reasoning, rather than stopping the entire generation before the final answer\. If the model reaches a reasoning cap, we forcefully insert</think\>and allow model to finish generation of the final answer for extra ten tokens \(to account for possible spaces, punctuation and for analysis\)\.

We consider an answer as invalid if it is empty or is not among the options available for that question\. A valid but incorrect option is an ordinary prediction error\. The answer is normalized by removing special tokens, stripping whitespace, case normalization\.

Table[3](https://arxiv.org/html/2610.20914#A4.T3)compares the last checkpoints used in our main results: epoch 20 for direct distillation \(no answer mixing\) and epoch 200 for the entropy\-sampling method \(with answer\-mixing\)\.

Table 3:Invalid extracted answers\. No mix denotes direct reasoning\-trace distillation; mix denotes entropy\-gain sampling with direct\-answer examples in the data mix\.At the 2,048\-token budget, the no\-mix students already produce valid extracted answers nearly perfectly\. Even correcting every invalid baseline answer would add an increment far below the accuracy gains in Table[1](https://arxiv.org/html/2610.20914#S4.T1)\. At 4,096 tokens, we see similar picture for two models, and also observe that Phi4\-mini has a comparable number of errors between the mix and non\-mix runs\. It points to answer\-mixing genuinely improving model’s quality, not merely formatting of the output\.

## Appendix ESingle\-token entropy

Single\-token entropy \(Figure[4](https://arxiv.org/html/2610.20914#A5.F4)\) measures a model’s uncertainty over the first answer token\. For a questionxx, we apply softmax to the first\-token logits to obtain probabilitiesp⁡\(v∣x\)p\(v\\mid x\)over the model’s full vocabularyVV, and compute normalized Shannon entropy:

H\(x\)=−1log⁡\|V\|∑v∈Vp\(v∣x\)logp\(v∣x\)\.H\(x\)=\-\\frac\{1\}\{\\log\|V\|\}\\sum\_\{v\\in V\}p\(v\\mid x\)\\log p\(v\\mid x\)\.\(1\)Dividing bylog⁡\|V\|\\log\|V\|puts student and proxy entropies on a common\[0,1\]\[0,1\]scale despite differences in vocabulary size\.

![Refer to caption](https://arxiv.org/html/2610.20914v1/img/single_token_entropy.png)Figure 4:Single\-token entropy example
## Appendix FSynthetic trace generation methods

### F\.1Direct reasoning trace\.

The teacher receives the question and options, generates reasoning, and predicts an option without seeing the ground\-truth answer\. The baseline retains both correct and incorrect responses\.

##### System prompt

> The following are multiple choice questions about \{subject\}\. Choose a correct option letter\. Answer with a single symbol\. Do not print anything else\.

### F\.2Direct reasoning trace \(correct only\)\.

We retain only baseline responses whose predicted option matches the ground truth\. System prompt is as above\.

### F\.3Explained answer\.

The teacher receives the question, options, and correct answer\. It generates a step\-by\-step explanation without revealing it knows ground truth\.

##### System prompt

> The following is a multiple choice question about \{subject\}\. The correct answer is provided to you, but your task is to produce a detailed chain\-of\-thought with step\-by\-step reasoning that reads as if you were solving the question from scratch—including any natural uncertainty, consideration of multiple options, or self\-correction\. Do not state or imply that the answer was given to you\. End your response with Answer:<letter\>\.

### F\.4Direct reasoning trace \(with explained\)\.

We retain correct baseline responses and replace each incorrect response with the explained\-answer response for the same question\.

### F\.5Corrected answer\.

For each incorrect baseline response, the teacher receives the original reasoning, its incorrect option, and the correct option\. It continues the trace with the self\-correction\. Correct baseline responses remain intact\.

##### System prompt

> The following are multiple choice questions about \{subject\}\. You will be shown a question, a partial chain\-of\-thought that was previously produced for it, the incorrect option that chain\-of\-thought arrived at, and the correct option letter\. Continue the partial chain\-of\-thought from where it left off so that it naturally arrives at the correct option\. Do not restart the reasoning, do not repeat what was already written, and do not acknowledge that the previous attempt was wrong or that the correct answer was given to you\. Write the continuation as if you were the same reasoner noticing a mistake or new consideration mid\-thought and revising course\. In the end, answer with the correct option letter\. End your response with Answer:<letter\>\.

### F\.6User prompts

All of them share the same user prompt:

> Question: \{question\} Options: a\. \{option text\} b\. \{option text\} …

For explained answers, appendCorrect option:followed by the ground\-truth letter\. For corrections, append:

> Partial reasoning so far: \{original reasoning\} Incorrect option this reasoning led to: \{predicted letter\} Correct option: \{ground\-truth letter\}

### F\.7Illustrative examples

For brevity, consider a toy question with two options: “A rectangle has sides of length 3 and 2\. What is its perimeter? a\. 6; b\. 10\.”

##### Direct trace\.

“The perimeter is3×2=63\\times 2=6\. Answer: a\.” The baseline retains this response; the correct\-only variant discards it\.

##### Explained answer\.

Given correct option b, a fresh explanation is: “There are two sides of each length, so the perimeter is2​\(3\+2\)=102\(3\+2\)=10\. Answer: b\.” The with\-explained variant substitutes this response for the incorrect direct trace\.

##### Corrected answer\.

Given the original reasoning, incorrect option a, and correct option b, a continuation is: “That product gives the area\. For the perimeter, I need to add all four sides:3\+2\+3\+2=103\+2\+3\+2=10\. Answer: b\.” The target retains the original reasoning, appends this continuation, and ends with b\.

### F\.8Per\-model results

Figures[5](https://arxiv.org/html/2610.20914#A6.F5)–[7](https://arxiv.org/html/2610.20914#A6.F7)show test accuracy across training epochs for each student\. The left panels compare head\-truncation limits of 3,072, 8,192, and 24,000 characters for direct reasoning traces; each legend separately specifies the evaluation cap in tokens\. The generation\-method panels hold head truncation fixed at 8,192 characters and compare methods at the stated evaluation cap\. Dashed horizontal lines and epoch\-zero values use the base model’s accuracy when prompted to answer with chain\-of\-thought \(base models are not originally reasoners\), shared across evaluation budgets\.

\(a\)Head\-truncation comparison\(b\)Generation methods: truncation 8,192, eval cap 8,192\(c\)Generation methods: truncation 8,192, eval cap 4,096
Figure 5:Synthetic\-trace distillation results for LLaMA 3\.2 3B\.\(a\)Head\-truncation comparison\(b\)Generation methods: truncation 8,192, eval cap 8,192\(c\)Generation methods: truncation 8,192, eval cap 4,096
Figure 6:Synthetic\-trace distillation results for Qwen 2\.5 3B\.\(a\)Head\-truncation comparison\(b\)Generation methods: truncation 8,192, eval cap 4,096
Figure 7:Synthetic\-trace distillation results for Phi4\-mini\.

## Appendix GSynthetic trace truncation and reasoning caps

Training truncation limits the distilled teacher reasoning in characters\. Reasoning cap limits the student’s generated reasoning at evaluation time in tokens\. They are evaluated independently\. At evaluation, reaching the reasoning cap forces a closing thinking token and allows generation to continue toward the answer, as detailed in Appendix[D](https://arxiv.org/html/2610.20914#A4)\.

Table[4](https://arxiv.org/html/2610.20914#A7.T4)reports how many training traces exceed each character budget before truncation\.

Table 4:Number \(percentage\) of training traces exceeding each truncation budget### G\.1Which part of the trace is retained?

Head truncation keeps a prefix\. Middle truncation keeps an initial segment and a final segment\. Tail truncation keeps only a suffix\. At 24,000\-character limit, head retains up to 24,000 characters, middle retains the first 18,000 and last 6,000 of overlong traces, and tail retains the last 24,000\. For corrected traces, middle truncation instead reserves up to 1,600 characters for the correction and uses the remaining space for the original prefix\.

Figure[8](https://arxiv.org/html/2610.20914#A7.F8)compares these policies at a fixed 8,192\-token evaluation cap\.

\(a\)LLaMA 3\.2 3B\(b\)Qwen 2\.5 3B\(c\)Phi4\-mini
Figure 8:Training truncation policies at the nominal 24,000\-character setting, evaluated with an 8,192\-token reasoning cap\. Dashed lines and epoch\-zero points give the base model’s chain\-of\-thought accuracy\.
### G\.2Training truncation length versus evaluation cap

For direct traces with head truncation at 3,072, 8,192, or 24,000 characters, we compare available evaluation caps of 1,024, 2,048, 4,096, and 8,192 tokens\. Figures[9](https://arxiv.org/html/2610.20914#A7.F9)and[10](https://arxiv.org/html/2610.20914#A7.F10)separate training length from evaluation budget\.

\(a\)Head truncation: 3,072 characters\(b\)Head truncation: 8,192 characters\(c\)Head truncation: 24,000 characters
Figure 9:Best\-checkpoint accuracy versus evaluation reasoning cap for each training truncation length\.\(a\)Head truncation: 3,072 characters\(b\)Head truncation: 8,192 characters\(c\)Head truncation: 24,000 characters
Figure 10:Last\-checkpoint accuracy versus evaluation reasoning cap for each training truncation length\.

## Appendix HSensitivity Study Detailed Results

Figure[14](https://arxiv.org/html/2610.20914#A8.F14)compares the weighted samplers and random sampling\. Figures[11](https://arxiv.org/html/2610.20914#A8.F11)–[13](https://arxiv.org/html/2610.20914#A8.F13)provide individual accuracy and coverage curves\. Coverage counts cumulative distinct question IDs selected at least once, so repeated occurrences do not increase it\.

\(a\)2,048\-token thinking budget\(b\)4,096\-token thinking budget\(c\)Cumulative unique question coverage
Figure 11:Distillation by sampling strategy for LLaMA 3\.2 3B\.\(a\)2,048\-token thinking budget\(b\)4,096\-token thinking budget\(c\)Cumulative unique question coverage
Figure 12:Distillation by sampling strategy for Qwen 2\.5 3B\.\(a\)2,048\-token thinking budget\(b\)4,096\-token thinking budget\(c\)Cumulative unique question coverage
Figure 13:Distillation by sampling strategy for Phi4\-mini\.\(a\)Weighted samplers, 4,096\-token thinking budget\(b\)Random versus entropy gain, 4,096\-token thinking budget\(c\)Mean unique question coverage
Figure 14:Sampler comparisons and question coverage across all three models\. In the accuracy panels, thin dashed curves show individual models, thick solid curves their equal\-weight means, and the thick dashed curve the direct\-trace baseline mean\.
## Appendix ICross\-model diagnostics of the uncertainty proxy

We compare single\-token entropies and accuracies of both uncertainty proxies and all three base students before distillation\. Accuracy measures single\-token answers rather than the reasoning accuracy reported in the main results\. Both proxies are substantially more accurate than the students \(Table[5](https://arxiv.org/html/2610.20914#A9.T5)\)\.

Table 5:Single\-token accuracy and mean normalized entropy on the full MMLU\-Pro datasetWe also analyze agreement and correlation between entropies\. Table[6](https://arxiv.org/html/2610.20914#A9.T6)compares entropy values and correctness using Pearsonrr, and entropy rankings using Spearmanρ\\rho\.

Table 6:Pearsonrrfor entropy values and correctness, and Spearmanρ\\rhofor entropy rankings\. The first row compares the proxies; remaining rows compare their mean entropy and mean correctness scores with each student’s scores on the common test split\.##### Interpretation\.

Table[6](https://arxiv.org/html/2610.20914#A9.T6)shows moderate agreement in proxy entropy rankings, with weaker linear agreement, which suggests aggregating entropies across models\. Correctness agreement is higher between proxies than between the mean proxy and any student\. It suggests a greater shared knowledge signal\. The largest correctness gap occurs for LLaMA 3B, which also shows the largest uplift over direct distillation \(Table[1](https://arxiv.org/html/2610.20914#S4.T1)\); Phi4\-mini’s closer entropy rankings could help explain its smaller uplift\. These patterns are consistent with a shared difficulty baseline alongside student\-specific uncertainty\. Under the assumption in Section[3](https://arxiv.org/html/2610.20914#S3)that larger models have lower total uncertainty, they motivate using larger\-model entropy to estimate the knowledge gap\.