Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation

# Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Source: [https://arxiv.org/html/2609.13770](https://arxiv.org/html/2609.13770)
Yilei Tu††thanks:Work done during an internship at Shanghai Artificial Intelligence Laboratory\.Zihao LiAffiliation:University of HelsinkiEmail:[zihao\.li@helsinki\.fi](mailto:)Shaoxiong JiAffiliation:ELLIS Institute FinlandAffiliation:University of TurkuEmail:[shaoxiong\.ji@utu\.fi](mailto:)Jörg TiedemannAffiliation:University of HelsinkiEmail:[jorg\.tiedemann@helsinki\.fi](mailto:)Fei YuanAffiliation:Shanghai Artificial Intelligence LaboratoryEmail:[feiyvan@163\.com](mailto:)

###### Abstract

Specialist distillation effectively transfers domain expertise to student models via teacher\-generated reasoning trajectories\. However, when these specialists are trained solely on question\-\-answer pairs without explicit reasoning supervision, what governs the trajectories they generate? In this work, we show that specialist optimization implicitly selects from this latent trajectory space\. To isolate and observe this latent distribution, we leverage student distillation not as a downstream goal, but as an agnostic probe\-\-\-since students inherit no parameterization or optimization constraints from the specialist, inheriting only the sampled trajectories themselves\. Through this probe, our empirical analysis unveils a tight governing relationship: across 27 specialist\-\-student pairings, their specialization\-\-generalization profiles correlate exceptionally strongly\. Crucially, explicitly controlling the specialist’s distributional drift systematically shifts both the teacher and its distilled student along a controllable trade\-off between domain precision and general\-capability retention\. Across chemistry, physics, and multilingual settings, distilled students systematically reflect these specialist\-induced profiles, even across divergent model families\. Our findings establish a new view of specialist training: when gold reasoning is absent, tuning choices directly control the latent supervision passed to downstream models\. The code111[https://github\.com/CONE\-MT/DCO/tree/main/specialist\_distillation](https://github.com/CONE-MT/DCO/tree/main/specialist_distillation), models and datasets222[https://huggingface\.co/collections/yileitu/qaonly\-specialist\-distillation](https://huggingface.co/collections/yileitu/qaonly-specialist-distillation)are publicly available\.

1Same Data, Same QA objective, Not Same BehaviorsQA pairsq1→a1q\_\{1\}\\to a\_\{1\}q2→a2q\_\{2\}\\to a\_\{2\}qN→aNq\_\{N\}\\to a\_\{N\}Optimize QA losse\.g\. cross\-entropyConstrainsπ⁡\(a\|q\)\\pi\(a\\,\|\\,q\)\(answers only\)Does not constrainπ⁡\(τ\|q\)\\pi\(\\tau\\,\|\\,q\)\(trajectories\)2Training Selects a BehaviorDifferent optimizations\(seeds / paths / hparams\)Different trajectorydistributionsπ1\(i\)​\(τ\|q\)\\pi\_\{1\}^\{\(i\)\}\(\\tau\\,\|\\,q\)Same QAobjectiveπ1\(1\)\\pi\_\{1\}^\{\(1\)\}π1\(2\)\\pi\_\{1\}^\{\(2\)\}⋮\\vdotsπ1\(M\)\\pi\_\{1\}^\{\(M\)\}⋮\\vdots3Distillation Locks It InSample trajectoriesfrom each variantDistill on sampledtrajectoriesLearned \(distilled\)models𝒟τ\(1\)\\mathcal\{D\}\_\{\\tau\}^\{\(1\)\}𝒟τ\(2\)\\mathcal\{D\}\_\{\\tau\}^\{\(2\)\}⋮\\vdots𝒟τ\(M\)\\mathcal\{D\}\_\{\\tau\}^\{\(M\)\}π0′\(1\)\\pi\_\{0\}^\{\\prime\(1\)\}\(init\. fromπ0\\pi\_\{0\}\)π0′\(2\)\\pi\_\{0\}^\{\\prime\(2\)\}\(init\. fromπ0\\pi\_\{0\}\)⋮\\vdotsπ0′\(M\)\\pi\_\{0\}^\{\\prime\(M\)\}\(init\. fromπ0\\pi\_\{0\}\)⋮\\vdotsModelTrajectory DatasetTrajectory DistributionConstrained by QA objectiveNot constrained

Figure 1:QA\-only specialist distillation is underdetermined at the trajectory level\.\(1\)QA supervision over QA\(q,a\)\(q,a\)pairs constrains the answer distributionπ⁡\(a\|q\)\\pi\(a\\,\|\\,q\), but not a unique trajectory distributionπ⁡\(τ\|q\)\\pi\(\\tau\\,\|\\,q\)over latent reasoning pathsτ\\tau\.\(2\)Under the same QA objective, different optimization profiles can produce intermediate specialistsπθ1\(i\)\\pi\_\{\\theta\_\{1\}\}^\{\(i\)\}with distinct trajectory distributions, even though they are all equally compatible with the observed gold answers\.\(3\)Distilling from trajectories sampled from one such specialist causes the final model to inherit that variant\-specific trajectory distribution\. Overall, QA\-only supervision leaves trajectory learning underdetermined, and specialist distillation implicitly commits to one selected behavior\.## 1Introduction

Specialist distillation\([Hinton et al\., 2015](https://arxiv.org/html/2609.13770#bib.bib21);[Fu et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib14);[Hsieh et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib23);[Ho et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib22);[Magister et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib39);[Yao et al\., 2021](https://arxiv.org/html/2609.13770#bib.bib64);[Liu et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib35)\)transfers domain expertise through an intermediate specialist model\. A general\-purposeoriginmodelπθ\\pi\_\{\\theta\}, e\.g\., Qwen3\-8B\-Instruct\([Yang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib62)\), is first adapted to a target domain, and the resultingspecialistthen generates reasoning trajectories that serve as supervision for a downstream student\([Mukherjee et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib42);[Xu et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib60)\)\. Yet in most specialized domains, the specialist itself is never explicitly taught how to reason\. Domain datasets typically provide\(q,a\)\(q,a\)question–answer \(QA\) pairs\([Uesato et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib53);[Chan et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib5);[Lightman et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib34);[Turpin et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib52)\)but no gold trajectories\([Zelikman et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib67);[Gülçehre et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib19);[Singh et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib48)\), because expert reasoning is difficult to obtain and verify\. Since the specialist is optimized only against final answers, what governs the trajectories it generates?

Letτ\\taudenote a free\-running reasoning trajectory generated for questionqq\. We consider representative behaviors including a substantive, answer\-consistent path\([Creswell and Shanahan, 2022](https://arxiv.org/html/2609.13770#bib.bib7);[Turpin et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib52),τ\+\\tau\_\{\+\};\), a shortcut\([Geirhos et al\., 2020](https://arxiv.org/html/2609.13770#bib.bib17),τ∼\\tau\_\{\\sim\};\), and an empty trajectory \(τ∅\\tau\_\{\\emptyset\}\)\. Standard teacher\-forced QA training directly optimizes answer likelihood and does not explicitly supervise these trajectories\. At generation time, however, the trained model induces a distribution over them\. Conceptually,

logπθ\(a\|q\)=log∑τ→aπθ\(τ\|q\),\\log\\pi\_\{\\theta\}\(a\\,\|\\,q\)=\\log\\sum\_\{\\tau\\rightarrow a\}\\pi\_\{\\theta\}\(\\tau\\,\|\\,q\),\(1\)whereτ→a\\tau\\rightarrow adenotes a trajectory compatible with answeraa\.[Equation1](https://arxiv.org/html/2609.13770#S1.E1)characterizes generation\-time behavior rather than the implemented SFT objective\. Since answer\-level supervision provides no direct preference among\{τ\+,τ∼,τ∅,⋯\}\\\{\\tau\_\{\+\},\\tau\_\{\\sim\},\\tau\_\{\\emptyset\},\\cdots\\\}, multiple trajectory distributions may remain compatible with the same supervised answer\.

To isolate and observe this latent selection, we repurpose student distillation as an agnostic probe rather than treating it only as a downstream goal\. A student inherits neither the specialist’s parameters nor the optimization constraints used to obtain them; it receives only the specialist\-generated supervision\. In our controlled pipeline, students share the same initialization and training configuration and are trained on equal amounts of sampled and filtered data\. Differences among students thus expose what these reasoning trajectories actually carry, even though students never receive the specialist’s parameters or optimization constraints directly\.

Through this diagnostic probe, we uncover a tight link between specialists and their distilled students\. Across chemistry, physics, and multilingual benchmarks, the balance between domain specialization and general capability in specialists systematically transfers to their downstream students\. Across nine specialist\-student pairs covering 27 distinct evaluations, these performance profiles align remarkably well \(Spearmanρ=0\.9573\\rho=0\.9573, permutation testp−value=0\.0093p\-\\mathrm\{value\}=0\.0093, see[§​4\.2](https://arxiv.org/html/2609.13770#S4.SS2.SSS0.Px3)for details\)\. This pattern holds even when teacher and student models belong to completely different model families, showing that shared teacher–student parameterization is not necessary for the observed transfer\. Distilled students therefore expose how the latent supervision selected by specialist optimization shapes downstream capability profiles\.

Crucially, this selection process is both observable and controllable\. Limiting how far a specialist drifts from its original base model directly recalibrates its trade\-off between domain mastery and general capabilities, guiding the student model along with it\. Layer\-Selective Tuning\([Gao et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib15), LST;\)provides an implicit anchor, yielding lower behavioral Kullback\-Leibler \(KL\) divergence and better preserving reasoning structures than standard full fine\-tuning\. For explicit control, Anchored Supervised Fine\-Tuning\([Zhu et al\., 2026](https://arxiv.org/html/2609.13770#bib.bib76), ASFT;\)provides a complementary explicit intervention: varying its anchoring strength systematically moves both specialist and student along the same trade\-off\. Together, these results identify distributional drift as a controllable axis of the latent supervision passed downstream\.

Our main contributions are:

- •Specialist optimization is the key design variable for distillation data\.Under QA\-only training, reasoning trajectories remain underdetermined by answer labels; the specialist’s optimization procedure selects the trajectory distribution from which downstream supervision is generated\.
- •Distilled students reveal the supervision selected by their specialists\.Using student distillation as an agnostic probe, we uncover a strong correspondence between specialist and student specialization–generalization profiles across domains and model families\. This inheritance shows that the effects of specialist optimization are encoded in the generated trajectories and transferred downstream, rather than remaining confined to the specialist’s parameters\.
- •We systematically characterize and control the resulting specialization–generalization trade\-off\.Through controlled distillation experiments, trajectory\-quality analysis, and comparisons between unconstrained tuning, implicit drift control \(LST\\LST\), and explicit KL anchoring \(ASFT\\ASFT\), we identify distributional drift as a governing axis of latent supervision\. Varying this drift steers both specialists and their students between domain precision and general\-capability retention\.

## 2Related Works

##### Specialist Distillation and Domain Adaptation\.

Adapting general\-purpose models to specialized domains is challenging due to cost, latency, and data scarcity, motivating specialist distillation and domain adaptation\. Early work compared “distill\-then\-adapt” with “adapt\-then\-distill”, showing that adapting both the teacher and student to the target domain before task\-agnostic distillation can yield compact models that preserve domain expertise\([Yao et al\., 2021](https://arxiv.org/html/2609.13770#bib.bib64)\)\. Recent LLM pipelines, includingDeepSeek\-V3\.2\([DeepSeek\-AI, 2025](https://arxiv.org/html/2609.13770#bib.bib8)\)andQwen3\.5\-Omni\([Qwen\-Team, 2026](https://arxiv.org/html/2609.13770#bib.bib44)\), similarly train domain\-specific experts and distill their capabilities back into a generalist model;[Li et al\. \(2024\)](https://arxiv.org/html/2609.13770#bib.bib32)further propose a staged expert\-growth framework from external supervision toward autonomous improvement\. Other work focuses on constructing and exploiting high\-quality domain supervision\. Synthetic query generation has been used to distill lightweight retrieval rerankers\([Saad\-Falcon et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib46)\), while knowledge hierarchies guide literature data distillation for biomedical QA\([Cai et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib4)\)\. On the distillation process,[Xia et al\. \(2026\)](https://arxiv.org/html/2609.13770#bib.bib59)use contrastive self\-distillation to transfer LLM reasoning paths into BERT without requiring explicit reasoning at inference, and[Liu et al\. \(2024\)](https://arxiv.org/html/2609.13770#bib.bib35)adapt the composition of distillation data to teacher–student performance gaps across domains\. Unlike these studies, we focus on trajectory\-level ambiguity in QA\-only specialist distillation, where specialist optimization implicitly shapes the trajectory distribution inherited by the final model\.

##### Self\-Distillation in LLMs\.

Self\-distillation uses a model’s own outputs or internal distributions as supervision\.[Yang et al\. \(2024\)](https://arxiv.org/html/2609.13770#bib.bib63)rewrite original responses into the model’s own distribution before fine\-tuning, mitigating catastrophic forgetting while preserving alignment\. Similarly,[Shenfeld et al\. \(2026\)](https://arxiv.org/html/2609.13770#bib.bib47)construct a demonstration\-conditioned teacher from the same model and distill its predictions via on\-policy reverse KL for continual skill acquisition without reward engineering\. For complex reasoning,[Zhao et al\. \(2026\)](https://arxiv.org/html/2609.13770#bib.bib72)use the same model as a privileged teacher and student, providing dense per\-token supervision over the student’s own rollouts\. In multilingual settings,[Zhang et al\. \(2024\)](https://arxiv.org/html/2609.13770#bib.bib71)distill resource\-rich language responses to improve cross\-lingual capabilities while preserving source\-language performance\. For code generation,[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.13770#bib.bib68)simply fine\-tune on sampled solutions, showing that even minimal self\-distillation can improve performance through reshaping the model’s output distribution\. Our work complements this literature by studying how QA\-only specialist optimization determines the latent reasoning supervision passed downstream\.

## 3QA\-only Specialist Distillation as Trajectory\-Distribution Selection

Specialist distillation typically refers to a two\-phase training pipeline\. Starting from an origin modelπθ0\\pi\_\{\\theta\_\{0\}\}, one first obtains an intermediate modelπθ1\\pi\_\{\\theta\_\{1\}\}that is adapted to a target domain\. This intermediate model is then used to generate training data for the final modelπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}, often initialized from the same origin modelπθ0\\pi\_\{\\theta\_\{0\}\}\. In this section, we study specialist distillation from a trajectory learning perspective \(in[§​3\.1](https://arxiv.org/html/2609.13770#S3.SS1)\)\. Surprisingly, we find that properly controlling specialist optimization can induce high\-quality reasoning trajectories under QA\-only supervision \(in[§​3\.2](https://arxiv.org/html/2609.13770#S3.SS2)\)\.

### 3\.1From QA\-only supervision to trajectory learning in specialist distillation\.

##### QA\-only supervision underdetermines the induced trajectory distribution\.

In many domain\-specific tasks, datasets𝒟\\mathcal\{D\}contain only question–answer pairs\(q,a\)\(q,a\)without reasoning trajectoriesτ\\tau\. In our implementation, the specialist is trained with standard teacher\-forced SFT on question–answer pairs\(q,a\)\(q,a\); it does not explicitly optimize or marginalize over latent reasoning trajectories\. We instead use a trajectory\-distribution abstraction to characterize the behavior induced by such answer\-level supervision\. Letπθ1​\(τ∣q\)\\pi\_\{\\theta\_\{1\}\}\(\\tau\\mid q\)denote the distribution over reasoning trajectories generated by the resulting specialist\. At this abstraction level, the probability assigned to an answer can be viewed as aggregating over trajectories compatible with that answer:

logπθ1\(a\|q\)=log∑τ→aπθ1\(τ\|q\)\.\\log\\pi\_\{\\theta\_\{1\}\}\(a\\,\|\\,q\)=\\log\\sum\_\{\\tau\\to a\}\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\.\(2\)whereτ→a\\tau\\to adenotes trajectories consistent with the correct answeraa\(or its equivalents\)\. Consequently, multiple trajectory distributions may be equally consistent with the same QA supervision by inducing the same answer likelihood\.

##### Optimization implicitly selects a trajectory distribution among many valid ones\.

Although multiple trajectory distributions are valid with the same QA supervision, optimization does not treat them equally\. To understand how optimization resolves this ambiguity, we examine the gradient of the objective:

∇logπθ1\(a\|q\)=∑τ→apθ\(τ\|q,a\)∇logπθ1\(τ\|q\),pθ1\(τ\|q,a\)=πθ1​\(τ\|q\)πθ1​\(a\|q\)\.\\nabla\\log\\pi\_\{\\theta\_\{1\}\}\(a\\,\|\\,q\)=\\sum\_\{\\tau\\to a\}p\_\{\\theta\}\(\\tau\\,\|\\,q,a\)\\nabla\\log\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\),\\quad p\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q,a\)=\\frac\{\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\}\{\\pi\_\{\\theta\_\{1\}\}\(a\\,\|\\,q\)\}\.\(3\)Eq\.[3](https://arxiv.org/html/2609.13770#S3.E3)shows that trajectories with larger posterior weightpθ1​\(τ\|q,a\)p\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q,a\)contribute more strongly to the gradient update\. Sincepθ1​\(τ\|q,a\)∝πθ1​\(τ\|q\)p\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q,a\)\\propto\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)among answer\-consistent trajectories, high\-probability trajectories dominate the gradient update\. Repeated optimization therefore resolves the underdetermination by selecting a particular trajectory distributionπθ1​\(τ\|q\)\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)from many valid ones\.

##### Distillation probes the latent trajectory distribution induced by specialist optimization\.

The latent trajectory distribution induced by specialist optimization is difficult to characterize directly: it spans a vast space of variable\-length reasoning sequences, and individual samples reveal only partial information about its structure and value as supervision\. Distillation provides an operational probe of this distribution by examining what a student learns from its sampled trajectories\. Crucially, the student receives these trajectories without inheriting the specialist’s adapted parameters or optimization constraints\. Under controlled student training, downstream differences therefore provide evidence of how specialist optimization shapes transferable supervision\. Formally, when trajectories sampled fromπθ1\\pi\_\{\\theta\_\{1\}\}are used to trainπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}, the resulting distillation objective is:

ℒ\(θ0′\)=𝔼τ∼πθ1\(⋅\|q\)\[−logπθ0′\(τ\|q\)\]\.\\mathcal\{L\}\(\\theta^\{\\prime\}\_\{0\}\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\}\\bigl\[\-\\log\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\tau\\,\|\\,q\)\\bigr\]\.\(4\)
This quantity corresponds to the cross\-entropy betweenπθ1\\pi\_\{\\theta\_\{1\}\}andπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}and admits the decomposition

𝔼τ∼πθ1\(⋅\|q\)\[−logπθ0′\(τ\|q\)\]\\displaystyle\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\}\\bigl\[\-\\log\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\tau\\,\|\\,q\)\\bigr\]=𝔼τ∼πθ1\(⋅\|q\)\[logπθ1​\(τ\|q\)πθ0′​\(τ\|q\)\]−𝔼τ∼πθ1\(⋅\|q\)\[logπθ1\(τ\|q\)\]\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\}\\left\[\\log\\frac\{\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\}\{\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\tau\\,\|\\,q\)\}\\right\]\-\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\}\\bigl\[\\log\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\\bigr\]\(5\)=KL\(πθ1\(⋅\|q\)\|\|πθ0′\(⋅\|q\)\)\+ℋ\(πθ1\(⋅\|q\)\)\.\\displaystyle=\\mathrm\{KL\}\\bigl\(\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\\,\|\|\\,\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\cdot\\,\|\\,q\)\\bigr\)\+\\mathcal\{H\}\\bigl\(\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\\bigr\)\.whereℋ\(πθ1\(⋅\|q\)\)=−𝔼τ∼πθ1\(⋅\|q\)\[logπθ1\(τ\|q\)\]\\mathcal\{H\}\(\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\)=\-\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\}\[\\log\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\]denotes the entropy ofπθ1\(⋅\|q\)\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)and is independent ofπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\. Therefore, optimizing w\.r\.t\.πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}is equivalent to minimizingKL\(πθ1\(⋅\|q\)\|\|πθ0′\(⋅\|q\)\),\\mathrm\{KL\}\\bigl\(\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\\,\|\|\\,\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\cdot\\,\|\\,q\)\\bigr\),which drives the distilled modelπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}to approximateπθ1\\pi\_\{\\theta\_\{1\}\}in the trajectory space\. Consequently, once the trajectory distributionπθ1\\pi\_\{\\theta\_\{1\}\}is fixed, the behavior of theπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}is largely determined byπθ1\\pi\_\{\\theta\_\{1\}\}\. Distillation thus probes the downstream consequences of trajectory selection without requiring an explicit characterization of the full trajectory distribution\.

### 3\.2Specialist Training as a Key Design Variable for Distillation Data

The preceding analysis connects specialist training to downstream data design: different training strategies can induce different trajectory distributions under identical QA supervision, thereby changing the supervision available for distillation\. The specialist’s training procedure is therefore a key design variable for shaping what the student learns\. This raises a practical question: how can specialist adaptation be controlled to shape the resulting distillation data?

One approach is to regulate distributional drift from the origin model\. Such control allows domain\-specific QA supervision to reshape the trajectory distribution while constraining its departure from the origin model’s behavior\. A canonical formulation of this principle is a KL\-constrained objective:

maxπθ1𝔼\(q,a\)∼𝒟\[logπθ1\(a\|q\)\]s\.t\.KL\(πθ1\(⋅\|q\)\|\|πθ0\(⋅\|q\)\)≤δ,\\max\_\{\\pi\_\{\\theta\_\{1\}\}\}\\quad\\mathbb\{E\}\_\{\(q,a\)\\sim\\mathcal\{D\}\}\[\\log\\pi\_\{\\theta\_\{1\}\}\(a\\,\|\\,q\)\]\\quad\\text\{s\.t\.\}\\quad\\mathrm\{KL\}\\big\(\\pi\_\{\\theta\_\{1\}\}\(\\cdot\\,\|\\,q\)\\,\|\|\\,\\pi\_\{\\theta\_\{0\}\}\(\\cdot\\,\|\\,q\)\\big\)\\leq\\delta,\(6\)where a smallδ\>0\\delta\>0limits how farπθ1\\pi\_\{\\theta\_\{1\}\}can drift from the original modelπθ0\\pi\_\{\\theta\_\{0\}\}\. Under this constraint, the induced trajectory distribution can be viewed as a reweighted version of theπθ0\\pi\_\{\\theta\_\{0\}\}:

πθ1​\(τ\|q\)∝πθ0​\(τ\|q\)​exp⁡\(w⁡\(τ,q,a\)λ\),\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)\\propto\\pi\_\{\\theta\_\{0\}\}\(\\tau\\,\|\\,q\)\\exp\\left\(\\frac\{w\(\\tau,q,a\)\}\{\\lambda\}\\right\),\(7\)wherew⁡\(τ,q,a\)w\(\\tau,q,a\)is an implicit quantity reflecting how a trajectory contributes to increasinglog⁡π⁡\(a\|q\)\\log\\pi\(a\\,\|\\,q\), andλ\>0\\lambda\>0is a Lagrange multiplier controlling the strength of the constraint\. In our setting, the KL–constrained formulation serves only as a characterization of the induced trajectory reweighting behavior\. We use standard QA\-only supervised fine\-tuning without explicit KL regularization, so neitherw⁡\(τ,q,a\)w\(\\tau,q,a\)norλ\\lambdais explicitly specified during training\. The derivation of[Eq\.7](https://arxiv.org/html/2609.13770#S3.E7)is provided in[App\.B](https://arxiv.org/html/2609.13770#A2)\. Substituting[Eq\.7](https://arxiv.org/html/2609.13770#S3.E7)into the distillation objective[Eq\.4](https://arxiv.org/html/2609.13770#S3.E4)yields:

ℒ⁡\(θ0′\)∝𝔼τ∼πθ0​\[exp⁡\(w⁡\(τ,q,a\)λ\)⏟quality\-aware reweighting⋅\(−log⁡πθ0′​\(τ\|q\)\)\],\\mathcal\{L\}\(\\theta^\{\\prime\}\_\{0\}\)\\propto\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\_\{0\}\}\}\\biggl\[\\underbrace\{\\exp\\left\(\\frac\{w\(\\tau,q,a\)\}\{\\lambda\}\\right\)\}\_\{\\text\{quality\-aware reweighting\}\}\\cdot\(\-\\log\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(\\tau\\,\|\\,q\)\)\\biggr\],\(8\)This form shows that the fine\-tuned trajectory distributionπθ1​\(τ\|q\)\\pi\_\{\\theta\_\{1\}\}\(\\tau\\,\|\\,q\)is obtained by reweighting the base distributionπθ0​\(τ\|q\)\\pi\_\{\\theta\_\{0\}\}\(\\tau\\,\|\\,q\)\. Under this characterization, trajectories that contribute more strongly to the answer\-level objective receive greater relative weight while the overall distribution remains anchored to the origin model\.

## 4Experiments

Table 1:Summary of training setups for our Specialist\-Distillation pipeline, additionalexplicit KL\-drift controlanalysis with ASFT, and twoself\-trainingbaselines\.Model SettingTuning Data FormatImplementation BriefFFT\\FFTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}\(q,a∗\)\(q,a^\{\*\}\)FFT\\FFT,LoRA\\LORA, andLST\\LSTshare the same hyperparameters, exceptfor those specific toLoRA\\LORAandLST\\LST\. See[App\.D\.1](https://arxiv.org/html/2609.13770#A4.SS1)for details\.LoRA\\LORASpecialistπθ1\\pi\_\{\\theta\_\{1\}\}Rankr=64r=64,α=2​r\\alpha=2r,lora\_dropout=0\.05\\texttt\{lora\\\_dropout\}=0\.05,target:all\.We provide ablation study on LoRA configs in[App\.C\.1](https://arxiv.org/html/2609.13770#A3.SS1)\.LST\\LSTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}Only update selected layers\. We adopt[Gao et al\. \(2025\)](https://arxiv.org/html/2609.13770#bib.bib15)’s best\-performing config forπθ0\\pi\_\{\\theta\_\{0\}\}: update bottom44and top1616layers\.ASFT\\ASFTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}explicit KL\-drift controlExplicitly regularizesπθ1\\pi\_\{\\theta\_\{1\}\}towardπθ0\\pi\_\{\\theta\_\{0\}\}with a KL penalty;λ∈\{0\.05,0\.2,0\.5\}\\lambda\\in\\\{0\.05,0\.2,0\.5\\\}controls anchoring strength\. See[§​5\.4](https://arxiv.org/html/2609.13770#S5.SS4)for details\.FFT\\FFTDistilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(q,τ^,a^\)\(q,\\hat\{\\tau\},\\hat\{a\}\)Allπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}’s areFFT\\FFT\-trained identically fromπθ0\\pi\_\{\\theta\_\{0\}\}Qwen3\-8B,varying only in the rationale data generated by their correspondingπθ1\\pi\_\{\\theta\_\{1\}\}\.Self\-Distillπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}q→τ^,a^q\\to\\hat\{\\tau\},\\hat\{a\}\(q,τ^,a^\)\(q,\\hat\{\\tau\},\\hat\{a\}\)πθ0\\pi\_\{\\theta\_\{0\}\}is prompted with only the questionqqto generate both thereasoning pathτ^\\hat\{\\tau\}and the prediction answera^\\hat\{a\}\.Self\-Rationalizeπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}q,a∗→τ^q,a^\{\*\}\\to\\hat\{\\tau\}\(q,τ^,a∗\)\(q,\\hat\{\\tau\},a^\{\*\}\)πθ0\\pi\_\{\\theta\_\{0\}\}is prompted with both the questionqqand the ground\-truthanswera∗a^\{\*\}, and is tasked to generate the rationaleτ^\\hat\{\\tau\}that leads toa∗a^\{\*\}\.### 4\.1Experimental Setup

We summarize in[Tab\.1](https://arxiv.org/html/2609.13770#S4.T1)the overarching experimental setup, including required data formats, training objectives, and key implementation details\. Comprehensive training, inference, rationale filtration and evaluation protocols are deferred to Apps\.[D](https://arxiv.org/html/2609.13770#A4)and[E](https://arxiv.org/html/2609.13770#A5)\. The core components are described below\.

##### Specialist Fine\-Tuning Strategies\.

We deployQwen3\-8Bas our origin modelπθ0\\pi\_\{\\theta\_\{0\}\}\. To investigate how different fine\-tuning strategies affect the quality of generated rationales, we train the specialist modelsπθ1\\pi\_\{\\theta\_\{1\}\}starting fromπθ0\\pi\_\{\\theta\_\{0\}\}using three methods as in[Tab\.1](https://arxiv.org/html/2609.13770#S4.T1): \(1\) Full Fine\-Tuning \(FFT\\FFT\), \(2\)LoRA\\LORA\([Hu et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib24)\), and \(3\) Layer\-Selective Tuning\([Gao et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib15),LST\\LST,\)\. We additionally evaluate Anchored Supervised Fine\-Tuning\([Zhu et al\., 2026](https://arxiv.org/html/2609.13770#bib.bib76),ASFT\\ASFT;\)as an explicit KL\-based drift\-control mechanism in[§​5\.4](https://arxiv.org/html/2609.13770#S5.SS4)\.

##### Distillation Pipeline\.

Once trained, each specialistπθ1\\pi\_\{\\theta\_\{1\}\}variant generates candidate chain\-of\-thoughts\([Wei et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib55), CoT;\)and answers\(τ^,a^\)\(\\hat\{\\tau\},\\hat\{a\}\), which are*filtered*such thata^\\hat\{a\}is equivalent to ground\-truth answera∗a^\{\*\}andcompleteCoT to construct valid\(q,τ^,a^\)\(q,\\hat\{\\tau\},\\hat\{a\}\)pool \(see[§​5\.6](https://arxiv.org/html/2609.13770#S5.SS6)and[App\.E](https://arxiv.org/html/2609.13770#A5)for details\)\. Crucially, to isolate the impact of data quality from the*student*model’s learning capacity, alldistilledmodelsπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}areFFT\\FFT\-trained on an equal amount of subsampled data and identical configurations, regardless of their correspondingspecialistmodelπθ1\\pi\_\{\\theta\_\{1\}\}tuning strategy\.

##### Baselines\.

We introduce two*self\-training*baselines \([Tabs\.1](https://arxiv.org/html/2609.13770#S4.T1)and[2](https://arxiv.org/html/2609.13770#S4.T2)\) derived directly fromπθ0\\pi\_\{\\theta\_\{0\}\}: \(1\)Self\-Distill: unconditional rationale generation and \(2\)Self\-Rationalize: answer\-conditioned rationale generation\. We further report larger modelsQwen3\-\{14,32\}Bfor scale\-based comparisons\.

##### Training Data and Evaluation Suites\.

We organize our datasets and benchmarks into four categories \(see[App\.E](https://arxiv.org/html/2609.13770#A5)for all benchmarks we evaluate\): \(1\)Trainingdata utilized for fine\-tuning within each target domain; \(2\)In\-Task \(It\)benchmarks that share the same domain and task formulation as the training data, using the official test split when available, otherwise a distribution\-wise closely matched benchmark; \(3\)In\-Domain \(Id\)benchmarks that remain within the target domain but differ in task distribution and difficulty level, evaluating robustness under domain shift\([Farahani et al\., 2020](https://arxiv.org/html/2609.13770#bib.bib10)\); and \(4\)Out\-of\-Domain \(Ood\)benchmarks drawn from domains entirely different from the target domain, assessing broader generalization\. ForOod, we use11benchmarks for complex reasoning\([Kazemi et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib28)\),22for mathematics\([Zhang, 2026](https://arxiv.org/html/2609.13770#bib.bib70)\), and22for coding\([Jain et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib26)\)\.

##### Domains and Rationale Filtrations\.

We study three*target*domains for Training,It, andId: \(1\)Chemistry\(Chem\)\. We train onSMol\([Yu et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib65)\), covering molecular understanding and generation across1414subtasks, whose official test split serves as theIt\. ForId, we evaluated on general chemistry benchmarks\. During rationale filtration, we apply subtask\-specific criteria \([App\.E\.1\.1](https://arxiv.org/html/2609.13770#A5.SS1.SSS1.Px4)\) to accommodate its diverse output formats and evaluation protocols\. \(2\)Physics\(Phys\)\. We train on the university\-level physics subset ofMegaScience\([Fan et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib9)\)\. ForIt, we evaluate on university\-level subsets fromPHYSICSbenchmark\([Feng et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib12)\); forId, on high\-school physics benchmarks\. Rationales are retained only for answers that pass rule\-based symbolic and numerical verification\. \(3\)Low\-Resource Multilingualism\(Lrm\)\. We train onOPUS\([Tiedemann, 2012](https://arxiv.org/html/2609.13770#bib.bib50)\)for*bi*\-directional English –88low\-resource languages translation\. ForIt, we evaluate the same1616translation task usingFlores\-101\([Goyal et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib18)\); forId, we assess general reasoning in these88languages, beyond translation\. Rationales are ranked by sentence\-levelspBLEU\([Papineni et al\., 2002](https://arxiv.org/html/2609.13770#bib.bib43);[Goyal et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib18)\)and the top\-performing20%20\\%subset is retained\.

Table 2:Performance comparison across Chemistry, Physics, and Multilingualism\.The parenthesized, e\.g\.,Id\(4\)\(4\), are the number of benchmarks \(subsets\) we use for this category\. Chemistry \(SMol\)Itscore aggregate its1414subtasks, with its computation detailed in[App\.E\.1\.1](https://arxiv.org/html/2609.13770#A5.SS1.SSS1)\. All other metrics are macro–averages over the corresponding categories\. The same fiveOodbenchmarks \(5=5^\{\\scriptscriptstyle=\}\) are used across all threetargetdomain settings\. Bold denotes the best result per column amongmodels derived fromQwen3\-8B;\-14/32Bare excluded\.ModelChemistryPhysicsMultilingualismIt\(14\)\(14\)Id\(4\)\(4\)Ood\(5=\)\(5^\{\\scriptscriptstyle=\}\)It\(1\)\(1\)Id\(4\)\(4\)Ood\(5=\)\(5^\{\\scriptscriptstyle=\}\)It\(16\)\(16\)Id\(16\)\(16\)Ood\(5=\)\(5^\{\\scriptscriptstyle=\}\)Qwen3\-8Bπθ0\\pi\_\{\\theta\_\{0\}\}19\.3460\.1942\.2290\.2168\.6142\.2229\.4844\.9942\.22Qwen3\-14B23\.8063\.6748\.9591\.3373\.0648\.9532\.3148\.4348\.95Qwen3\-32B25\.3568\.0951\.2392\.4577\.8651\.2333\.7849\.1351\.23All tuned onπθ0\\pi\_\{\\theta\_\{0\}\}Qwen3\-8Bq→τ^,a^q\\to\\hat\{\\tau\},\\hat\{a\}\\ \\Self\-Distillπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}21\.0659\.2842\.0491\.1167\.9743\.5429\.3045\.9439\.66q,a∗→τ^q,a^\{\*\}\\to\\hat\{\\tau\}Self\-Rationalizeπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}17\.9258\.7833\.1487\.8968\.6441\.786\.0227\.2627\.71FFT\\FFTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}59\.6557\.7233\.6090\.7368\.4316\.3135\.9323\.866\.90​FFT\\ \\hbox to10\.1pt\{\\vbox to7\.21pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\{ \{ \\lxSVG@setdash\{0\.0pt\}\\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lx@inpgf@ignorespaces\{\} \{\\lx@inpgf@ignorespaces\}\{\}\{\} \{\}\{\}\{\} \\lxSVG@stroke\}\} \}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 5\.72 L 0 0 L 12\.59 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{9\.35002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 12\.94 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFTDistilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}40\.5546\.1425\.2091\.8568\.5114\.0930\.7727\.7815\.08LoRA\\LORASpecialistπθ1\\pi\_\{\\theta\_\{1\}\}28\.5760\.3642\.6592\.8368\.9838\.0320\.4743\.1641\.41​FFT\\ \\hbox to10\.1pt\{\\vbox to7\.21pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 5\.72 L 0 0 L 12\.59 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{9\.35002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 12\.94 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFTDistilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}29\.8059\.3134\.8991\.9369\.8738\.2622\.9730\.8735\.89LST\\LSTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}29\.6160\.2044\.0393\.6570\.1338\.5035\.6843\.9939\.08​FFT\\ \\hbox to10\.1pt\{\\vbox to7\.21pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 5\.72 L 0 0 L 12\.59 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{9\.35002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 12\.94 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFTDistilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}29\.5460\.4742\.1192\.7570\.8440\.8034\.1743\.8741\.23

### 4\.2Main Results

[Table2](https://arxiv.org/html/2609.13770#S4.T2)compares the origin model, larger same\-family models, untuned self\-training baselines, and our specialist–distillation pipeline across Chemistry, Physics, and Multilingualism underIt,Id, andOodevaluation\. Benchmark composition and metric computation details are provided in[App\.E](https://arxiv.org/html/2609.13770#A5)\.

##### QA\-only specialist distillation consistently improves target\-domain performance\.

AcrossFFT,LoRA,and​LST\\FFT,\\LORA,\\text\{and\}\\LSTtuning strategies and three target domains, our key observations from[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)demonstrate that even without gold rationales, explicitly training a QA\-onlyspecialistto induce domain\-specific reasoning traces, and subsequently transferring them to adistilledmodel, consistently yields robust and significant target\-domain growth:

- •Substantial target\-domain improvements\.Across all configurations, both thespecialistmodelsπθ1\\pi\_\{\\theta\_\{1\}\}and their downstreamdistilledmodelsπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}achieve massivein\-task\(It\) capability gains compared to the originQwen3\-8Bπθ0\\pi\_\{\\theta\_\{0\}\}and twoself\-trainingbaselines\.
- •Bridging a4×4\\timesparameter gap\.Our pipeline enables an 8B model to “punch above its weight class” without relying on human\-annotated rationales\. For example, ourLST\\LST\-distilled model \(29\.54/92\.75/34\.1729\.54/92\.75/34\.17for allIt\) surpass the zero\-shot performance of the4×4\\timeslargerQwen3\-32B\(25\.35/92\.45/33\.7825\.35/92\.45/33\.78\)\. This highlights that extracting latent reasoning paths from a specialist is a highly parameter\-efficient paradigm compared to merely scaling up generalist models\.
- •Toxic post\-hoc rationalization\.The untrainedSelf\-Rationalizebaseline severely degrades performance, notably plummetingLrmItfrom29\.4829\.48to6\.026\.02, which corroborates findings that post\-hoc reasoning on translation is often spurious and unreliable\([Wu et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib58);[Li et al\., 2026](https://arxiv.org/html/2609.13770#bib.bib33)\)\. Forcing weak models to rationalize answers induces hallucination or shortcuts that poison distillation\.

##### Different QA\-only supervision methods implicitly select different trajectory distributions and lead to distinct specialization\-generalization trade\-offs\.

Overall, across all33domains,LST\\LSTconsistently improvesItperformance over the origin, achieves modest gains onId, and remains on par onOod\. WhileFFT\\FFTyields impressiveItgains in specific domains such as chemistry, it severely sacrifices both the performance ofIdandOod, suffering fromcatastrophic forgetting\([Kumar et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib29)\)\.LoRA\\LORAexhibits a trade\-off pattern similar toLST\\LSTbut generally falls short ofLST\\LSTacross all99splits\. These distinct performance profiles confirm that different QA\-only tuning strategies implicitly select different rationale distributions\. Thus, the optimal tuning method depends on the desired specialization–generalization trade–off rather than a universal optimum\.

##### Distillation reliably inherits and refines specialist capabilities\.

002020404060608080100100002020404060608080100100Strict Rank PreservationSpearman’sρ=0\.9573\\rho=0\.9573Each point is one\(πθ1,πθ0′\)\(\\pi\_\{\\theta\_\{1\}\},\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\)performancepair as in[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)\.y=xy=x\(Perfect Mirror\)Specialistπθ1\\pi\_\{\\theta\_\{1\}\}Performance \(%\)Distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}Performance \(%\)πθ1\\pi\_\{\\theta\_\{1\}\}–TuningDomainCat\.FFT\\FFTChemItLoRA\\LORAPhysIdLST\\LSTLrmOodFigure 2:Rank Preservationfromπθ1\\pi\_\{\\theta\_\{1\}\}toπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\.
Figure 3:Cross\-model Transferability on Chemistry\.FFT/LST\\FFT/\\LST\-tunedIntern\-S1\-mini\(πϕ0\\pi\_\{\\phi\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/x1.png)\-S1m\) serves as the specialistπϕ1\\pi\_\{\\phi\_\{1\}\}, andπϕ\\pi\_\{\\phi\}\-generated\(q,τ^,a^\)\(q,\\hat\{\\tau\},\\hat\{a\}\)are distilled intoQwen3\-8B\(![Refer to caption](https://arxiv.org/html/2609.13770v1/logos/qwen.png)3\-8B\)\.Models on ChemistryIt\(14\)\(14\)Id\(4\)\(4\)Ood\(5=\)\(5^\{\\scriptscriptstyle=\}\)Originπθ0\\pi\_\{\\theta\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/logos/qwen.png)3\-8B19\.3460\.1942\.22Originπϕ0\\pi\_\{\\phi\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/Intern_S1_logo_notext.png)\-S1m34\.8062\.5137\.77FFT\\FFT\-Distilltoπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/logos/qwen.png)3\-8B31\.3360\.2533\.79FFT\\FFTSpecialistπϕ1\\pi\_\{\\phi\_\{1\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/Intern_S1_logo_notext.png)\-S1m42\.8354\.8622\.43​FFT\\ \\hbox to10\.1pt\{\\vbox to7\.21pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 5\.72 L 0 0 L 12\.59 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{9\.35002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 12\.94 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFT\-Distilltoπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/logos/qwen.png)3\-8B36\.9156\.9925\.90LST\\LSTSpecialistπϕ1\\pi\_\{\\phi\_\{1\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/Intern_S1_logo_notext.png)\-S1m33\.1160\.3136\.21​FFT\\ \\hbox to10\.1pt\{\\vbox to7\.21pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 5\.72 L 0 0 L 12\.59 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{9\.35002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 12\.94 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFT\-Distilltoπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}![Refer to caption](https://arxiv.org/html/2609.13770v1/logos/qwen.png)3\-8B32\.7258\.7937\.46

Beyond absolute metrics, an intriguing phenomenon observed in[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)is the strongrankcorrelation between the capabilities ofπθ1\\pi\_\{\\theta\_\{1\}\}and its downstreamπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}: across all2727experimental measurements, Spearman’s rank correlation reachesρ=0\.9573\\rho=0\.9573\([Fig\.3](https://arxiv.org/html/2609.13770#S4.F3)\)\. To account for the dependence amongIt,Id, andOodmeasurements from the same specialist–student pair, we additionally conduct a domain\-blocked, model\-profile\-level exact permutation test, which yieldsp−value=2/215≈0\.0093p\\mathrm\{\-value\}=2/215\\approx 0\.0093\. The relative performance ranking among tuning strategies \(e\.g\.,LST\\LST\>LoRA\\LORA\>FFT\\FFTinPhysIt\) is therefore strongly reflected in the downstream*distilled*models across the evaluated splits\. This consistency supports the effectiveness of our variable\-controlled pipeline: because allπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}models are fine\-tuned using the exact sameFFT\\FFTconfiguration, differences in their performance are closely associated with the rationale supervisionτ^\\hat\{\\tau\}generated by their correspondingπθ1\\pi\_\{\\theta\_\{1\}\}\. Consequently, strongerspecialistsgenerally tend to produce supervision that leads to strongerdistilledmodels\. Furthermore, in nearly half \(12/2712/27\) of the splits, thedistilledmodel even exceeds its correspondingspecialist, indicating that distillation does not merely copy teacher behavior but can further refine the supervision induced by specialist\-generated rationales\. More broadly, distilled models consistently outperform the two untuned self\-training baselines, suggesting that effective rationale supervision is better obtained from explicitly trainedspecialistmodels than from the origin model alone or post\-hoc self\-rationalization\. Together, these results support specialist training as an effective approach for producing rationale data useful for downstream distillation within the evaluated Qwen3\-8B\-based pipeline\.

We also demonstrate in[App\.C\.2](https://arxiv.org/html/2609.13770#A3.SS2)that these specialization–generalization trade\-offs and monotonic rank correlation remain consistent when scaling the origin modelπθ0\\pi\_\{\\theta\_\{0\}\}up toQwen3\-14B\.

## 5Analysis: Data Scaling, Cross\-model, KL Drift, Explicit KL Anchor, and Trajectory Quality

### 5\.1Data scaling yields continuous in\-task gains while preserving robustness\.

30304040505060607070ChemId&OodScore050K100K150K200K250K300K360K Full18182020252530303535ChemIt\(SMol\) ScoreItIdOodFFT\\FFTDistillation Data Size\(a\)Chemπθ1\\pi\_\{\\theta\_\{1\}\}Chemπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}0011223344LoRA⁡πθ1\\LORA\\pi\_\{\\theta\_\{1\}\}\(2\.00\)πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(2\.11\)FFT⁡πθ1\\FFT\\pi\_\{\\theta\_\{1\}\}\(3\.05\)πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(4\.40\)LST⁡πθ1\\LST\\pi\_\{\\theta\_\{1\}\}\(1\.43\)πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(2\.11\)Self\-Distillπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(0\.11\)Self\-Rationalizeπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\(0\.10\)KL\(π∙\|\|πθ0\)\\KL\\big\(\\pi\_\{\\bullet\}\\,\|\|\\,\\pi\_\{\\theta\_\{0\}\}\\big\)LoRA\\LORAFFT\\FFTLST\\LSTSelf\-DistillSelf\-Rationalize\(b\)
Figure 4:Left \(a\): Data scaling onSMolusing rationales collected from theLST\\LSTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}andFFT\\FFT\-distillation\. Increasing the amount of filtered training data improvesItperformance substantially, while leavingIdlargely stable and only minimal degradation onOod\.Right \(b\): KL divergence ofChemπθ1\\pi\_\{\\theta\_\{1\}\}andπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}models w\.r\.tπθ0\\pi\_\{\\theta\_\{0\}\}Qwen3\-8B\.LST\\LSTmodels exhibit lower KL divergence compared toFFT\\FFT/LoRA\\LORAcounterparts\.We study data scaling on theChemSMolusing rationales collected from theLST\\LSTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}, followed byFFT\\FFTdistillation\.SMolcontains roughly 3M examples in total; after generating\(τ^,a^\)\(\\hat\{\\tau\},\\hat\{a\}\)pairs and filtering them by answer correctness and completeness of the reasoning traces, we obtain a curated pool of 360K training examples\. We scale the training set from 50K to the full 360K in 50K increments \([Fig\.4\(a\)](https://arxiv.org/html/2609.13770#S5.F4.sf1)\)\. Increasing the amount of filtered distillation data leads to clear gains onIt\(from19\.319\.3to32\.532\.5\), whileIdremains virtually unchanged andOoddegrades only modestly overall\. These results suggest that distillation fromπθ1\\pi\_\{\\theta\_\{1\}\}rationales has not saturated with a moderate amount of data and exhibits favorable scaling behavior with additional high\-quality data\.

### 5\.2Students inherit their teachers’ generalization profiles across model families\.

We evaluatecross\-model transferby distilling rationales fromIntern\-S1\-mini\([Bai et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib1),S1m;\), a science\-specialized 8B model, into aQwen3\-8Bstudent \([Fig\.3](https://arxiv.org/html/2609.13770#S4.F3)\)\. Crucially, the distilled student strictly inherits the behavioral profile of its specific teacher\. While direct distillation fromS1malready improves the student’sItsubstantially \(19\.34→31\.3319\.34\\to 31\.33\), distilling from theFFT\\FFT\-S1m\-specialist maximizes studentItbut severely degradesId/Ood, exactly mirroring theFFT\\FFT\-teacher’s extreme overfitting\. Conversely, anLST\\LST\-S1m\-specialist imparts a balanced profile: the student preserves robustness comparable toS1mwhile still delivering a sizableItgain \(\+13\.38\+13\.38\) fromQwen3\-8B\. These results confirm that structural constraints \(LST\\LST\) versus unconstrained tuning \(FFT\\FFT\) fundamentally shape the rationale distributions, dictating downstream generalization even across model families\. Beyond cross\-family transfer,[App\.C\.3](https://arxiv.org/html/2609.13770#A3.SS3)shows that our 8B specialist actually imparts better performance to an 8B student than a larger 14B teacher, hence specialist supervision surpasses the model scale\.

### 5\.3Post\-training strategies induce different degrees of KL drift\.

To characterize how specialist post\-training changes model behavior with respect to the origin model, we measure the token\-levelKL\(π∙\|\|πθ0\),∀π∙∈\{πθ1,πθ0′\}\\mathrm\{KL\}\(\\pi\_\{\\bullet\}\\,\|\|\\,\\pi\_\{\\theta\_\{0\}\}\),\\forall\\pi\_\{\\bullet\}\\in\\\{\\pi\_\{\\theta\_\{1\}\},\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\\\}, across different tuning strategies onChem\([Fig\.4\(b\)](https://arxiv.org/html/2609.13770#S5.F4.sf2)\)\. We use KL divergence as a behavioral diagnostic of distributional drift rather than as the explicit tuning objective\.FFT\\FFTexhibits the largest divergence from the origin model for both the specialistπθ1\\pi\_\{\\theta\_\{1\}\}and its distilled studentπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\.LoRA\\LORAproduces substantially smaller drift, whileLST\\LSTyields the lowest KL divergence among the adapted models evaluated\. These results show an empirical association between stronger constraints on specialist adaptation and reduced behavioral drift from the origin model\. In particular,LST\\LSTis consistent with an*implicit anchoring effect*: although it contains no explicit KL regularization, its restricted update space yields a specialist that remains closer to the origin\-model distribution while acquiring target\-domain capability\. Importantly, we do not claim thatLST\\LSToptimizes a KL\-constrained objective, nor that a particular layer\-selection configuration corresponds to a specific KL radius\. The lower\-drift profile induced byLST\\LSTis also reflected in its distilled student, motivating the question of whether explicitly controlling KL drift can systematically move the specialization–generalization operating point\. We test this directly in[§​5\.4](https://arxiv.org/html/2609.13770#S5.SS4)\.

### 5\.4Explicit KL Drift Control with ASFT

ItIdOodSpecialistFFT\\FFTDistilled

ASFT\\ASFTλ=0\.05\\lambda=0\.05ASFT\\ASFTλ=0\.2\\lambda=0\.2ASFT\\ASFTλ=0\.5\\lambda=0\.5LST\\LSTref\.20203030404050506060ScoreChemistryASFT\\ASFTλ=0\.05\\lambda=0\.05ASFT\\ASFTλ=0\.2\\lambda=0\.2ASFT\\ASFTλ=0\.5\\lambda=0\.5LST\\LSTref\.0010102020303040405050ScoreMultilingualism
Figure 5:Explicit KL anchoring with ASFT\.Increasing the KL coefficientλ\\lambdastrengthens anchoring to the origin model, shifting both specialists and their distilled students from stronger task specialization toward betterId/Oodretention\. The first threexx\-axis settings correspond toASFT\\ASFTwithλ∈\{0\.05,0\.2,0\.5\}\\lambda\\in\\\{0\.05,0\.2,0\.5\\\}\. The shadedLST\\LSTref\.column shows the correspondingLST\\LSTspecialist and distilled student as an implicit drift\-control reference \([Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)\) rather than anASFT\\ASFTsetting\.Left \(a\): Chemistry\.Right \(b\): Multilingualism\.Our analysis suggests that the specialization–generalization trade\-off is governed more broadly by how strongly specialist optimization controls distributional drift, rather than byLST\\LSTspecifically\. To test this interpretation beyond structural constraints, we additionally evaluate Anchored Supervised Fine\-Tuning\([Zhu et al\., 2026](https://arxiv.org/html/2609.13770#bib.bib76),ASFT\\ASFT;\), which explicitly regularizes the specialist toward the origin model through a KL penalty\. We repeat the specialist generation andFFT\\FFT\-distillation pipeline with the coefficientλ∈\{0\.05,0\.2,0\.5\}\\lambda\\in\\\{0\.05,0\.2,0\.5\\\}, where largerλ\\lambdaimposes stronger anchoring\.

As shown in[Fig\.5](https://arxiv.org/html/2609.13770#S5.F5), increasing the anchoring strength systematically shifts both specialists and their distilled students from stronger task specialization toward betterId/Ood\\textsc\{Id\}/\\textsc\{Ood\}retention\. In Chemistry, increasingα\\alphafrom0\.050\.05to0\.50\.5reduces specialistItperformance from39\.4339\.43to22\.4222\.42, while improvingOodperformance from29\.9729\.97to36\.1036\.10; the corresponding distilled models exhibit the same shift, from37\.63/28\.5537\.63/28\.55to24\.74/36\.2424\.74/36\.24inIt/Ood\\textsc\{It\}/\\textsc\{Ood\}\. The trend is even more pronounced in Multilingualism, where specialistId/Ood\\textsc\{Id\}/\\textsc\{Ood\}increases from26\.65/8\.7026\.65/8\.70to45\.42/32\.9945\.42/32\.99, accompanied by substantial gains in the distilled models from9\.15/0\.519\.15/0\.51to44\.79/43\.7144\.79/43\.71\. These results show that the observed trade\-off is not unique toLST\\LST: explicit KL regularization provides a complementary and tunable mechanism for controlling specialist drift, and the resulting specialization–generalization profile is subsequently inherited through distillation\.LST\\LSTshould therefore be viewed as one practical implicit realization of this broader drift\-control principle, rather than as a uniquely optimal tuning strategy\.

### 5\.5Structural constraint improves answer prediction without collapsing the trajectory\.

BaselineLST\\LSTFFT\\FFTLoRA\\LORA0\.60\.60\.70\.70\.80\.80\.90\.911\.69\.85\.94\.86\.69\.72\.83\.74Top\-1 Hit Rate \(HR@1\\HR\)↑\\uparrowSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}Distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}Qwen3\-8Bπθ0\\pi\_\{\\theta\_\{0\}\}& Self\-Distillπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}BaselinesFigure 6:HR@1\\HRunder teacher\-forcing on unseenChemSMolqueries\. Higher \(↑\\uparrow\) is better\. Hatched are the origin modelπθ0\\pi\_\{\\theta\_\{0\}\}andself\-distillbaselines\.Figure 7:Rationale quality on Chemistry\. Statistics over150150K generated trajectories from5050KSMolqueries with33samples each, covering twoself\-trainingmodes ofπθ0\\pi\_\{\\theta\_\{0\}\}and three specialists \(FFT\\FFT,LoRA\\LORA, andLST\\LST\)\. “Comp\.” is the percentage of structurally*complete*trajectories\. “Empty” is the permille of empty rationales among*complete*trajectories\. “Words” denotes average word count±\\pmstd withincompleterationales\.Model GenerationComp\.\(%\)Empty\(‰\)Words\(μ±σ\\mu\\pm\\sigma\)πθ0\\pi\_\{\\theta\_\{0\}\}q→τ^,a^q\\rightarrow\\hat\{\\tau\},\\hat\{a\}87\.930\.002476±16242476\\pm 1624πθ0\\pi\_\{\\theta\_\{0\}\}q,a∗→τ^q,a^\{\*\}\\rightarrow\\hat\{\\tau\}16\.180\.082049±14432049\\pm 1443q→τ^,a^q\\to\\hat\{\\tau\},\\hat\{a\}FFT\\FFTSpec\.πθ1\\pi\_\{\\theta\_\{1\}\}8\.901\.10233±450233\\pm 450LoRA\\LORASpec\.πθ1\\pi\_\{\\theta\_\{1\}\}41\.390\.001378±9021378\\pm 902LST\\LSTSpec\.πθ1\\pi\_\{\\theta\_\{1\}\}92\.500\.022334±14352334\\pm 1435

To analyze how different fine\-tuning strategies distribute probability mass over valid trajectories \([§​3\.1](https://arxiv.org/html/2609.13770#S3.SS1.SSS0.Px2)\), we evaluate the next\-token top\-1 hit rate \(HR@1\\HR\) viateacher\-forcingon2,0002,000unseenSMolsamples \([App\.E\.1\.1](https://arxiv.org/html/2609.13770#A5.SS1.SSS1.Px5)\)\. Suppressing intermediate CoT steps isolates the intrinsic capability to sequentially predict ground\-truth answers\. While all tuning methods substantially improveHR@1\\HRover the baselines \([Fig\.7](https://arxiv.org/html/2609.13770#S5.F7)\), unconstrainedFFT\\FFTinduces extreme confidence in the specialist \(HR@1=0\.94\\HR=0\.94\), indicating a drastic collapse of the trajectory distribution to overfit the answer likelihood \(Eq\.[2](https://arxiv.org/html/2609.13770#S3.E2)\)\. In contrast, parameter\-efficient architectures \(LST\\LSTandLoRA\\LORA\) provide calibrated enhancements \(HR@1≈0\.85\\HR\\approx 0\.85\)\. Aligning with our theoretical framework \([§​3\.2](https://arxiv.org/html/2609.13770#S3.SS2)\), these structural constraints enforce a smoother reweighting of the base policy \(Eq\.[7](https://arxiv.org/html/2609.13770#S3.E7)\) rather than arbitrarily distorting the generation manifold\. By preventing rote memorization, they facilitate robust and generalizable transfer to the distilled students\. Next\-token probability and rank exhibit consistent trends \([App\.C\.4](https://arxiv.org/html/2609.13770#A3.SS4)\)\.

### 5\.6Structural constraint preserves the integrity of reasoning trajectories\.

To assess rationale quality, we analyze150150K Chemistry trajectories \(33candidates per5050KSMolqueries\)\. We define a “complete” trajectory as having exactly one valid<think\> </think\>pair\. As[Fig\.7](https://arxiv.org/html/2609.13770#S5.F7)shows, unconstrainedFFT\\FFTsuffers severe structural collapse: a dismal8\.90%8\.90\\%completion rate, the highest empty rationale proportion \(1\.11\.1‰,τ∅\\tau\_\{\\emptyset\}as in[§​1](https://arxiv.org/html/2609.13770#S1)\), and heavily degenerated trace lengths \(233233vs\. origin’s24762476words\)\. This indicates a collapse into shortcut reasoning \(τ∼\\tau\_\{\\sim\}as in[§​1](https://arxiv.org/html/2609.13770#S1)\), explaining its poor\-quality distillation supervision\.LoRA\\LORApartially recovers completion \(41\.39%41\.39\\%\), whereasLST\\LST–specialistπθ1\\pi\_\{\\theta\_\{1\}\}robustly preserves both structural integrity \(92\.50%92\.50\\%\) and trace lengths comparable to the originπθ0\\pi\_\{\\theta\_\{0\}\}\. Furthermore, theself\-rationalizebaseline \(q,a∗→τ^q,a^\{\*\}\\to\\hat\{\\tau\}\) drops to16\.18%16\.18\\%completion, largely due to the generation of multiple reasoning blocks\. This empirically reinforces our claim in[§​4\.2](https://arxiv.org/html/2609.13770#S4.SS2.SSS0.Px1)that forcing weak models into post\-hoc rationalization fundamentally destabilizes the generation manifold and acts as toxic supervision\.

## 6Conclusion

In this work, we frame QA\-only specialist distillation as a trajectory\-distribution selection problem: answer supervision leaves reasoning trajectories underdetermined, while specialist optimization determines the latent supervision passed downstream\. Across chemistry, physics, and low\-resource multilingual tasks, distilled students closely inherit their specialists’ specialization–generalization profiles, including across model families\. We further identify distributional drift as a controllable axis of this transfer\. UnconstrainedFFT\\FFTinduces larger drift and degraded generalization, whereasLST\\LSTprovides effective implicit drift control; explicit KL anchoring withASFT\\ASFTsystematically moves both specialists and students along the same trade\-off\. These results establish specialist optimization as a key design variable for distillation data, enabling more predictable control over domain specialization and general\-capability retention when gold trajectories are unavailable\.

## References

- Bai et al\. \(2025\)Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, Yu Cheng, Pei Chu, Tao Chu, Erfei Cui, Ganqu Cui, Long Cui, Ziyun Cui, Nianchen Deng, Ning Ding, Nanqing Dong, Peijie Dong, Shihan Dou, Sinan Du, Haodong Duan, Caihua Fan, Ben Gao, Changjiang Gao, Jianfei Gao, Songyang Gao, Yang Gao, Zhangwei Gao, Jiaye Ge, Qiming Ge, Lixin Gu, Yuzhe Gu, Aijia Guo, Qipeng Guo, Xu Guo, Conghui He, Junjun He, Yili Hong, Siyuan Hou, Caiyu Hu, Hanglei Hu, Jucheng Hu, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Xu Huang, Zixian Huang, Zhe Jiang, Lingkai Kong, Linyang Li, Peiji Li, Pengze Li, Shuaibin Li, Tianbin Li, Wei Li, Yuqiang Li, Dahua Lin, Junyao Lin, Tianyi Lin, Zhishan Lin, Hongwei Liu, Jiangning Liu, Jiyao Liu, Junnan Liu, Kai Liu, Kaiwen Liu, Kuikun Liu, Shichun Liu, Shudong Liu, Wei Liu, Xinyao Liu, Yuhong Liu, Zhan Liu, Yinquan Lu, Haijun Lv, Hongxia Lv, Huijie Lv, Qitan Lv, Ying Lv, Chengqi Lyu, Chenglong Ma, Jianpeng Ma, Ren Ma, Runmin Ma, Runyuan Ma, Xinzhu Ma, Yichuan Ma, Zihan Ma, Sixuan Mi, Junzhi Ning, Wenchang Ning, Xinle Pang, Jiahui Peng, Runyu Peng, Yu Qiao, Jiantao Qiu, Xiaoye Qu, Yuan Qu, Yuchen Ren, Fukai Shang, Wenqi Shao, Junhao Shen, Shuaike Shen, Chunfeng Song, Demin Song, Diping Song, Chenlin Su, Weijie Su, Weigao Sun, Yu Sun, Qian Tan, Cheng Tang, Huanze Tang, Kexian Tang, Shixiang Tang, Jian Tong, Aoran Wang, Bin Wang, Dong Wang, Lintao Wang, Rui Wang, Weiyun Wang, Wenhai Wang, Jiaqi Wang, Yi Wang, Ziyi Wang, Ling\-I Wu, Wen Wu, Yue Wu, Zijian Wu, Linchen Xiao, Shuhao Xing, Chao Xu, Huihui Xu, Jun Xu, Ruiliang Xu, Wanghan Xu, GanLin Yang, Yuming Yang, Haochen Ye, Jin Ye, Shenglong Ye, Jia Yu, Jiashuo Yu, Jing Yu, Fei Yuan, Yuhang Zang, Bo Zhang, Chao Zhang, Chen Zhang, Hongjie Zhang, Jin Zhang, Qiaosheng Zhang, Qiuyinzhe Zhang, Songyang Zhang, Taolin Zhang, Wenlong Zhang, Wenwei Zhang, Yechen Zhang, Ziyang Zhang, Haiteng Zhao, Qian Zhao, Xiangyu Zhao, Xiangyu Zhao, Bowen Zhou, Dongzhan Zhou, Peiheng Zhou, Yuhao Zhou, Yunhua Zhou, Dongsheng Zhu, Lin Zhu, and Yicheng Zou\. 2025\.[Intern\-S1: A scientific multimodal foundation model](https://doi.org/10.48550/ARXIV.2508.15763)\.*CoRR*, abs/2508\.15763\.
- Banerjee and Lavie \(2005\)Satanjeev Banerjee and Alon Lavie\. 2005\.[METEOR: an automatic metric for MT evaluation with improved correlation with human judgments](https://aclanthology.org/W05-0909/)\.In*Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005*, pages 65–72\. Association for Computational Linguistics\.
- Bisk et al\. \(2020\)Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi\. 2020\.[PIQA: reasoning about physical commonsense in natural language](https://doi.org/10.1609/AAAI.V34I05.6239)\.In*The Thirty\-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty\-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7\-12, 2020*, pages 7432–7439\. AAAI Press\.
- Cai et al\. \(2025\)Xunxin Cai, Chengrui Wang, Qingqing Long, Yuanchun Zhou, and Meng Xiao\. 2025\.[Knowledge hierarchy guided biological\-medical dataset distillation for domain LLM training](https://doi.org/10.48550/ARXIV.2501.15108)\.*CoRR*, abs/2501\.15108\.
- Chan et al\. \(2022\)Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz\. 2022\.[UNIREX: A unified learning framework for language model rationale extraction](https://proceedings.mlr.press/v162/chan22a.html)\.In*International Conference on Machine Learning, ICML 2022, 17\-23 July 2022, Baltimore, Maryland, USA*, Proceedings of Machine Learning Research, pages 2867–2889\. PMLR\.
- Contributors \(2023\)OpenCompass Contributors\. 2023\.OpenCompass: A universal evaluation platform for foundation models\.[https://github\.com/open\-compass/opencompass](https://github.com/open-compass/opencompass)\.
- Creswell and Shanahan \(2022\)Antonia Creswell and Murray Shanahan\. 2022\.[Faithful reasoning using large language models](https://doi.org/10.48550/ARXIV.2208.14271)\.*CoRR*, abs/2208\.14271\.
- DeepSeek\-AI \(2025\)DeepSeek\-AI\. 2025\.[DeepSeek\-V3\.2: Pushing the frontier of open large language models](https://doi.org/10.48550/ARXIV.2512.02556)\.*CoRR*, abs/2512\.02556\.
- Fan et al\. \(2025\)Run\-Ze Fan, Zengzhi Wang, and Pengfei Liu\. 2025\.[Megascience: Pushing the frontiers of post\-training datasets for science reasoning](https://doi.org/10.48550/ARXIV.2507.16812)\.*CoRR*, abs/2507\.16812\.
- Farahani et al\. \(2020\)Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R\. Arabnia\. 2020\.[A brief review of domain adaptation](https://arxiv.org/abs/2010.03978)\.*CoRR*, abs/2010\.03978\.
- Favre and Powell \(2013\)Henri A Favre and Warren H Powell\. 2013\.*Nomenclature of Organic Chemistry: IUPAC Recommendations and Preferred Names 2013 \(Blue Book\)*\.Royal Society of Chemistry\.
- Feng et al\. \(2025\)Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan\. 2025\.[Physics: Benchmarking foundation models on university\-level physics problem solving](https://aclanthology.org/2025.findings-acl.610/)\.In*Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 \- August 1, 2025*, Findings of ACL, pages 11717–11743\. Association for Computational Linguistics\.
- for Standardization \(2023\)The International Organization for Standardization\. 2023\.[ISO 639:2023\(en\)code for individual languages and language groups](https://www.iso.org/obp/ui/en/#iso:std:iso:639:ed-2:v1:en)\.
- Fu et al\. \(2023\)Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot\. 2023\.[Specializing smaller language models towards multi\-step reasoning](https://proceedings.mlr.press/v202/fu23d.html)\.In*International Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA*, Proceedings of Machine Learning Research, pages 10421–10430\. PMLR\.
- Gao et al\. \(2025\)Changjiang Gao, Zixian Huang, Jingyang Gong, Shujian Huang, Lei Li, and Fei Yuan\. 2025\.[LLaMAX2: Your translation\-enhanced model also performs well in reasoning](https://doi.org/10.48550/ARXIV.2510.09189)\.*CoRR*, abs/2510\.09189\.
- Gao et al\. \(2024\)Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou\. 2024\.[The language model evaluation harness](https://doi.org/10.5281/zenodo.12608602)\.
- Geirhos et al\. \(2020\)Robert Geirhos, Jörn\-Henrik Jacobsen, Claudio Michaelis, Richard S\. Zemel, Wieland Brendel, Matthias Bethge, and Felix A\. Wichmann\. 2020\.[Shortcut learning in deep neural networks](https://doi.org/10.1038/S42256-020-00257-Z)\.*Nat\. Mach\. Intell\.*, 2\(11\):665–673\.
- Goyal et al\. \(2022\)Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng\-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan\. 2022\.[The flores\-101 evaluation benchmark for low\-resource and multilingual machine translation](https://doi.org/10.1162/TACL_A_00474)\.*Trans\. Assoc\. Comput\. Linguistics*, 10:522–538\.
- Gülçehre et al\. \(2023\)Çaglar Gülçehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas\. 2023\.[Reinforced self\-training \(rest\) for language modeling](https://doi.org/10.48550/ARXIV.2308.08998)\.*CoRR*, abs/2308\.08998\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\. 2021\.[Measuring massive multitask language understanding](https://openreview.net/forum?id=d7KBjmI3GmQ)\.In*9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021*\. OpenReview\.net\.
- Hinton et al\. \(2015\)Geoffrey E\. Hinton, Oriol Vinyals, and Jeffrey Dean\. 2015\.[Distilling the knowledge in a neural network](https://arxiv.org/abs/1503.02531)\.*CoRR*, abs/1503\.02531\.
- Ho et al\. \(2023\)Namgyu Ho, Laura Schmid, and Se\-Young Yun\. 2023\.[Large language models are reasoning teachers](https://doi.org/10.18653/V1/2023.ACL-LONG.830)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023*, pages 14852–14882\. Association for Computational Linguistics\.
- Hsieh et al\. \(2023\)Cheng\-Yu Hsieh, Chun\-Liang Li, Chih\-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen\-Yu Lee, and Tomas Pfister\. 2023\.[Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes](https://doi.org/10.18653/V1/2023.FINDINGS-ACL.507)\.In*Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023*, Findings of ACL, pages 8003–8017\. Association for Computational Linguistics\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2022\.[LoRA: Low\-rank adaptation of large language models](https://openreview.net/forum?id=nZeVKeeFYf9)\.In*The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022*\. OpenReview\.net\.
- Huang et al\. \(2025\)Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan\. 2025\.[BenchMAX: A comprehensive multilingual evaluation suite for large language models](https://aclanthology.org/2025.findings-emnlp.909/)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4\-9, 2025*, pages 16751–16774\. Association for Computational Linguistics\.
- Jain et al\. \(2025\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2025\.[Livecodebench: Holistic and contamination free evaluation of large language models for code](https://openreview.net/forum?id=chfJJYC3iL)\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\. OpenReview\.net\.
- Kalajdzievski \(2023\)Damjan Kalajdzievski\. 2023\.[A rank stabilization scaling factor for fine\-tuning with lora](https://doi.org/10.48550/ARXIV.2312.03732)\.*CoRR*, abs/2312\.03732\.
- Kazemi et al\. \(2025\)Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K\. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q\. Tran, Quoc V\. Le, and Orhan Firat\. 2025\.[BIG\-Bench extra hard](https://aclanthology.org/2025.acl-long.1285/)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025*, pages 26473–26501\. Association for Computational Linguistics\.
- Kumar et al\. \(2022\)Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang\. 2022\.[Fine\-tuning can distort pretrained features and underperform out\-of\-distribution](https://openreview.net/forum?id=UYneFzXSJWh)\.In*The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022*\. OpenReview\.net\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.[Efficient memory management for large language model serving with pagedattention](https://doi.org/10.1145/3600006.3613165)\.In*Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23\-26, 2023*, pages 611–626\. ACM\.
- Landrum et al\. \(2026\)Greg Landrum, Paolo Tosco, Brian Kelley, Ricardo Rodriguez, David Cosgrove, Riccardo Vianello, sriniker, Peter Gedeck, Gareth Jones, Eisuke Kawashima, NadineSchneider, Dan Nealschneider, tadhurst cdd, Andrew Dalke, Matt Swain, Brian Cole, Samo Turk, Aleksandr Savelev, Niels Maeder, Rachel Walker, Alain Vaucher, Maciej Wójcikowski, Hussein Faara, Ichiru Take, Vincent F\. Scalfani, Yakov Pechersky, Kazuya Ujihara, Daniel Probst, Jeremy Monat, and Juuso Lehtivarjo\. 2026\.[rdkit/rdkit: 2026\_03\_1 \(q1 2026\) release](https://doi.org/10.5281/zenodo.19250388)\.
- Li et al\. \(2024\)Jiawei Li, Xiaoang Xu, and Yang Gao\. 2024\.[METEOR: evolutionary journey of large language models from guidance to self\-growth](https://doi.org/10.48550/ARXIV.2411.11933)\.*CoRR*, abs/2411\.11933\.
- Li et al\. \(2026\)Zihao Li, Shaoxiong Ji, and Jörg Tiedemann\. 2026\.[Test\-time scaling of reasoning models for machine translation](https://doi.org/10.18653/v1/2026.eacl-long.133)\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 2902–2917, Rabat, Morocco\. Association for Computational Linguistics\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\. 2024\.[Let’s verify step by step](https://openreview.net/forum?id=v8L0pN6EOi)\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net\.
- Liu et al\. \(2024\)Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Jiamang Wang, Lin Qu, Wenbo Su, and Bo Zheng\. 2024\.[DDK: distilling domain knowledge for efficient large language models](http://papers.nips.cc/paper_files/paper/2024/hash/b206d54ffbb803b5c51d85f405d422e4-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024*\.
- Llama\-Team \(2024\)Llama\-Team\. 2024\.[The llama 3 herd of models](https://doi.org/10.48550/ARXIV.2407.21783)\.*CoRR*, abs/2407\.21783\.
- Loshchilov and Hutter \(2019\)Ilya Loshchilov and Frank Hutter\. 2019\.[Decoupled weight decay regularization](https://openreview.net/forum?id=Bkg6RiCqY7)\.In*7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019*\. OpenReview\.net\.
- M\-A\-P\-Team et al\. \(2025\)M\-A\-P\-Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tyshawn Hsing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, and Ge Zhang\. 2025\.[SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines](https://doi.org/10.48550/ARXIV.2502.14739)\.*CoRR*, abs/2502\.14739\.
- Magister et al\. \(2023\)Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn\. 2023\.[Teaching small language models to reason](https://doi.org/10.18653/v1/2023.acl-short.151)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 1773–1781, Toronto, Canada\. Association for Computational Linguistics\.
- Mirza et al\. \(2025\)Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos\-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling\-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Mehrdad Asgari, Juliane Eberhardt, Amir Mohammad Elahi, Hani M\. Elbeheiry, María Victoria Gil, Christina Glaubitz, Maximilian Greiner, Caroline T\. Holick, Tim Hoffmann, Abdelrahman Ibrahim, Lea C\. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole C\. Roesner, Johanna Schreiber, Ulrich S\. Schubert, Leanne M\. Stafast, A\. D\. Dinga Wonanke, Michael Pieler, Philippe Schwaller, and Kevin Maik Jablonka\. 2025\.[A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists](https://doi.org/10.1038/s41557-025-01815-x)\.*Nature Chemistry*, 17\(7\):1027–1034\.
- Morgan \(1965\)H\. L\. Morgan\. 1965\.[The generation of a unique machine description for chemical structures\-a technique developed at chemical abstracts service\.](https://doi.org/10.1021/c160017a018)*Journal of Chemical Documentation*, 5\(2\):107–113\.
- Mukherjee et al\. \(2023\)Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah\. 2023\.[Orca: Progressive learning from complex explanation traces of GPT\-4](https://doi.org/10.48550/ARXIV.2306.02707)\.*CoRR*, abs/2306\.02707\.
- Papineni et al\. \(2002\)Kishore Papineni, Salim Roukos, Todd Ward, and Wei\-Jing Zhu\. 2002\.[BLEU: a method for automatic evaluation of machine translation](https://doi.org/10.3115/1073083.1073135)\.In*Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6\-12, 2002, Philadelphia, PA, USA*, pages 311–318\. ACL\.
- Qwen\-Team \(2026\)Qwen\-Team\. 2026\.[Qwen3\.5\-omni technical report](https://doi.org/10.48550/ARXIV.2604.15804)\.*CoRR*, abs/2604\.15804\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. 2023\.[GPQA: A graduate\-level google\-proof q&a benchmark](https://doi.org/10.48550/ARXIV.2311.12022)\.*CoRR*, abs/2311\.12022\.
- Saad\-Falcon et al\. \(2023\)Jon Saad\-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Sultan, and Christopher Potts\. 2023\.[UDAPDR: Unsupervised domain adaptation via LLM prompting and distillation of rerankers](https://doi.org/10.18653/v1/2023.emnlp-main.693)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 11265–11279, Singapore\. Association for Computational Linguistics\.
- Shenfeld et al\. \(2026\)Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal\. 2026\.[Self\-distillation enables continual learning](https://doi.org/10.48550/ARXIV.2601.19897)\.*CoRR*, abs/2601\.19897\.
- Singh et al\. \(2024\)Avi Singh, John D\. Co\-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J\. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T\. Parisi, Abhishek Kumar, Alexander A\. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L\. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl\-Dickstein, and Noah Fiedel\. 2024\.[Beyond human data: Scaling self\-training for problem\-solving with language models](https://openreview.net/forum?id=lNAyUngGFK)\.*Trans\. Mach\. Learn\. Res\.*, 2024\.
- Suzgun et al\. \(2023\)Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V\. Le, Ed H\. Chi, Denny Zhou, and Jason Wei\. 2023\.[Challenging big\-bench tasks and whether chain\-of\-thought can solve them](https://doi.org/10.18653/V1/2023.FINDINGS-ACL.824)\.In*Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023*, Findings of ACL, pages 13003–13051\. Association for Computational Linguistics\.
- Tiedemann \(2012\)Jörg Tiedemann\. 2012\.[Parallel data, tools and interfaces in OPUS](http://www.lrec-conf.org/proceedings/lrec2012/summaries/463.html)\.In*Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23\-25, 2012*, pages 2214–2218\. European Language Resources Association \(ELRA\)\.
- Tu et al\. \(2025\)Yilei Tu, Andrew Xue, and Freda Shi\. 2025\.[Blessing of multilinguality: A systematic analysis of multilingual in\-context learning](https://aclanthology.org/2025.findings-acl.323/)\.In*Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 \- August 1, 2025*, Findings of ACL, pages 6213–6248\. Association for Computational Linguistics\.
- Turpin et al\. \(2023\)Miles Turpin, Julian Michael, Ethan Perez, and Samuel R\. Bowman\. 2023\.[Language models don’t always say what they think: Unfaithful explanations in chain\-of\-thought prompting](http://papers.nips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*\.
- Uesato et al\. \(2022\)Jonathan Uesato, Nate Kushman, Ramana Kumar, H\. Francis Song, Noah Y\. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins\. 2022\.[Solving math word problems with process\- and outcome\-based feedback](https://doi.org/10.48550/ARXIV.2211.14275)\.*CoRR*, abs/2211\.14275\.
- Wang et al\. \(2024\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\. 2024\.[MMLU\-Pro: A more robust and challenging multi\-task language understanding benchmark](http://papers.nips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html)\.In*Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024*\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H\. Chi, Quoc V\. Le, and Denny Zhou\. 2022\.[Chain\-of\-thought prompting elicits reasoning in large language models](http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022*\.
- Weininger \(1988\)David Weininger\. 1988\.[SMILES, a chemical language and information system\. 1\. introduction to methodology and encoding rules](https://doi.org/10.1021/ci00057a005)\.*Journal of Chemical Information and Computer Sciences*, 28\(1\):31–36\.
- Wolf et al\. \(2020\)Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M\. Rush\. 2020\.[Transformers: State\-of\-the\-art natural language processing](https://doi.org/10.18653/V1/2020.EMNLP-DEMOS.6)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 \- Demos, Online, November 16\-20, 2020*, pages 38–45\. Association for Computational Linguistics\.
- Wu et al\. \(2025\)Di Wu, Seth Aycock, and Christof Monz\. 2025\.[Please translate again: Two simple experiments on whether human\-like reasoning helps translation](https://doi.org/10.18653/v1/2025.emnlp-main.1031)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 20424–20440, Suzhou, China\. Association for Computational Linguistics\.
- Xia et al\. \(2026\)Runze Xia, Yupeng Ji, Yuxi Zhou, Haodong Liu, Teng Zhang, and Piji Li\. 2026\.[From reasoning llms to BERT: A two\-stage distillation framework for search relevance](https://doi.org/10.1145/3774904.3792801)\.In*Proceedings of the ACM Web Conference 2026, WWW 2026, Dubai, United Arab Emirates, originally scheduled for April 13\-17, 2026, rescheduled for June 29 \- July 3, 2026*, pages 8222–8231\. ACM\.
- Xu et al\. \(2024\)Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang\. 2024\.[Wizardlm: Empowering large pre\-trained language models to follow complex instructions](https://openreview.net/forum?id=CfXh93NDgH)\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net\.
- Xu et al\. \(2025\)Wenda Xu, Rujun Han, Zifeng Wang, Long T\. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen\-Yu Lee, and Tomas Pfister\. 2025\.[Speculative knowledge distillation: Bridging the teacher\-student gap through interleaved sampling](https://openreview.net/forum?id=EgJhwYR2tB)\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\. OpenReview\.net\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu\. 2025\.[Qwen3 technical report](https://doi.org/10.48550/ARXIV.2505.09388)\.*CoRR*, abs/2505\.09388\.
- Yang et al\. \(2024\)Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu\. 2024\.[Self\-distillation bridges distribution gap in language model fine\-tuning](https://doi.org/10.18653/V1/2024.ACL-LONG.58)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pages 1028–1043\. Association for Computational Linguistics\.
- Yao et al\. \(2021\)Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei\. 2021\.[Adapt\-and\-distill: Developing small, fast and effective pretrained language models for domains](https://doi.org/10.18653/V1/2021.FINDINGS-ACL.40)\.In*Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1\-6, 2021*, Findings of ACL, pages 460–470\. Association for Computational Linguistics\.
- Yu et al\. \(2024\)Botao Yu, Frazier N\. Baker, Ziqi Chen, Xia Ning, and Huan Sun\. 2024\.[LlaSMol: Advancing large language models for chemistry with a large\-scale, comprehensive, high\-quality instruction tuning dataset](https://openreview.net/forum?id=lY6XTF9tPv)\.In*First Conference on Language Modeling, COLM 2024, Philadelphia, PA, USA, October 7\-9, 2024*\.
- Yuan et al\. \(2023\)Fei Yuan, Yinquan Lu, Wenhao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu\. 2023\.[Lego\-MT: Learning detachable models for massively multilingual machine translation](https://doi.org/10.18653/V1/2023.FINDINGS-ACL.731)\.In*Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023*, Findings of ACL, pages 11518–11533\. Association for Computational Linguistics\.
- Zelikman et al\. \(2022\)Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D\. Goodman\. 2022\.[Star: Bootstrapping reasoning with reasoning](http://papers.nips.cc/paper_files/paper/2022/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022*\.
- Zhang et al\. \(2026\)Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, and Yizhe Zhang\. 2026\.[Embarrassingly simple self\-distillation improves code generation](https://arxiv.org/abs/2604.01193)\.*CoRR*, abs/2604\.01193\.
- Zhang \(2025\)Yifan Zhang\. 2025\.[American invitational mathematics examination \(AIME\) 2025](https://huggingface.co/datasets/math-ai/aime25)\.
- Zhang \(2026\)Yifan Zhang\. 2026\.[American invitational mathematics examination \(AIME\) 2026](https://huggingface.co/datasets/math-ai/aime26)\.
- Zhang et al\. \(2024\)Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu\. 2024\.[Enhancing multilingual capabilities of large language models through self\-distillation from resource\-rich languages](https://doi.org/10.18653/v1/2024.acl-long.603)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 11189–11204, Bangkok, Thailand\. Association for Computational Linguistics\.
- Zhao et al\. \(2026\)Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover\. 2026\.[Self\-distilled reasoner: On\-policy self\-distillation for large language models](https://doi.org/10.48550/ARXIV.2601.18734)\.*CoRR*, abs/2601\.18734\.
- Zheng et al\. \(2024\)Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo\. 2024\.[LlamaFactory: Unified efficient fine\-tuning of 100\+ language models](https://doi.org/10.18653/V1/2024.ACL-DEMOS.38)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pages 400–410\. Association for Computational Linguistics\.
- Zhong et al\. \(2024\)Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan\. 2024\.[AGIEval: A human\-centric benchmark for evaluating foundation models](https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.149)\.In*Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16\-21, 2024*, Findings of ACL, pages 2299–2314\. Association for Computational Linguistics\.
- Zhou et al\. \(2023\)Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\. 2023\.[Instruction\-following evaluation for large language models](https://doi.org/10.48550/ARXIV.2311.07911)\.*CoRR*, abs/2311\.07911\.
- Zhu et al\. \(2026\)He Zhu, Junyou Su, Peng Lai, Ren Ma, Wenjia Zhang, Linyi Yang, and Guanhua Chen\. 2026\.[Anchored supervised fine\-tuning](https://openreview.net/forum?id=PORko7QT64)\.In*The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23\-27, 2026*\. OpenReview\.net\.

## Appendix ALimitations

While our framework provides a novel perspective on QA\-only distillation, it entails several limitations that present opportunities for future work\. \(1\)Scale and Emergent Reasoning:Our primary experiments focus on the 8B parameter scale to rigorously isolate variables\. How implicit trajectory selection interacts with the self\-correction capabilities of massive\-scale models \(e\.g\.,\>70\>70B\) or explicitly reasoning\-optimized models remains an open question\. \(2\)Spurious Correlations:Our rationale filtration strictly relies on exact answer equivalence\. While highly effective in deterministic domains like physics and chemistry, it cannot entirely eradicate the “right answer, wrong reasoning” phenomenon \(spurious shortcuts\), particularly in open\-ended generative tasks like multilingual translation\.

## Appendix BKL constraints enable controlled reweighting\.

To prevent uncontrolled trajectory shift, we formulate learning as a drift\-constrained optimization problem:

maxπ𝔼\(q,a\)\[logπ\(a\|q\)\]s\.t\.𝔼q\[KL\(π\(⋅\|q\)\|\|πθ0\(⋅\|q\)\)\]≤δ,\\max\_\{\\pi\}\\quad\\mathbb\{E\}\_\{\(q,a\)\}\[\\log\\pi\(a\\,\|\\,q\)\]\\qquad\\text\{s\.t\.\}\\qquad\\mathbb\{E\}\_\{q\}\\left\[\\mathrm\{KL\}\\bigl\(\\pi\(\\cdot\\,\|\\,q\)\\,\|\|\\,\\pi\_\{\\theta\_\{0\}\}\(\\cdot\\,\|\\,q\)\\bigr\)\\right\]\\leq\\delta,\(9\)whereδ\>0\\delta\>0controls the average allowed deviation from the base policyπθ0\\pi\_\{\\theta\_\{0\}\}\.

The objective is defined at the answer level:

logπ\(a\|q\)=log∑τ→aπ\(τ\|q\),\\log\\pi\(a\\,\|\\,q\)=\\log\\sum\_\{\\tau\\to a\}\\pi\(\\tau\\,\|\\,q\),\(10\)which depends on a marginalization over latent trajectories and does not explicitly specify how probability mass is distributed across trajectories\.

To make this structure explicit, we introduce a trajectory\-level view\. For any distributionp⁡\(τ\|q,a\)p\(\\tau\\,\|\\,q,a\)supported on trajectories satisfyingτ→a\\tau\\to a, Jensen’s inequality gives

logπ\(a\|q\)≥𝔼τ∼p\(⋅\|q,a\)\[logπ\(τ\|q\)−logp\(τ\|q,a\)\]\.\\log\\pi\(a\\,\|\\,q\)\\geq\\mathbb\{E\}\_\{\\tau\\sim p\(\\cdot\\,\|\\,q,a\)\}\\left\[\\log\\pi\(\\tau\\,\|\\,q\)\-\\log p\(\\tau\\,\|\\,q,a\)\\right\]\.\(11\)
This lower bound shows that improvinglog⁡π⁡\(a\|q\)\\log\\pi\(a\\,\|\\,q\)corresponds to increasing probability mass on trajectories that produce the correct answer\.

Under this trajectory\-level perspective, the KL constraint restricts how much the trajectory distribution can deviate from the base policy\. As a result, updates are realized through a controlled reweighting of trajectories, favoring those that support the correct answer while remaining close toπθ0\\pi\_\{\\theta\_\{0\}\}\.

Introducing a Lagrange multiplierλ\>0\\lambda\>0for the KL constraint in[Eq\.9](https://arxiv.org/html/2609.13770#A2.E9), the resulting solution takes the form

π⁡\(τ\|q\)∝πθ0​\(τ\|q\)​exp⁡\(w⁡\(τ,q,a\)λ\),\\pi\(\\tau\\,\|\\,q\)\\propto\\pi\_\{\\theta\_\{0\}\}\(\\tau\\,\|\\,q\)\\exp\\left\(\\frac\{w\(\\tau,q,a\)\}\{\\lambda\}\\right\),\(12\)wherew⁡\(τ,q,a\)w\(\\tau,q,a\)is an implicit advantage\-like quantity induced by the objective, reflecting the relative contribution of trajectoryτ\\tauto increasinglog⁡π⁡\(a\|q\)\\log\\pi\(a\\,\|\\,q\)\.

Thus, KL\-constrained updates achieve improvement through controlled reweighting of trajectories, rather than unconstrained redistribution of probability mass\.

## Appendix CSupplementary Analysis

### C\.1Ablation Study on LoRA Configurations

We also analyze the effect of LoRA\-specific design choices in theLrmsetting\. All experiments are evaluated on the sameIt,Id, andOodbenchmark groups as in the main experiments\.

##### Rank sensitivity\.

[Tab\.3](https://arxiv.org/html/2609.13770#A3.T3)reports theLrmperformance of LoRA Specialist models trained with ranksr∈\{16,32,64,128\}r\\in\\\{16,32,64,128\\\}\. The results show that LoRA performance is relatively stable for ranks 16, 32, and 64 onIdandOodevaluation, while increasing the rank to 128 leads to a noticeableOoddrop\. Rank 32 obtains the highestItscore, whereas rank 64 obtains the bestIdscore and remains nearly tied with ranks 16 and 32 onOod\. We therefore use rank 64 in the main experiments as a capacity\-balanced default rather than tuning the rank to maximize a singleItscore\.

Table 3:Rank sensitivity of LoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\}onLrmusing 7MLrmQA examples\.ChemModelItIdOodLoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=16r=1623\.0242\.8441\.45LoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=32r=3223\.4743\.0341\.45LoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=64r=6420\.4743\.1641\.41LoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=128r=12821\.7742\.0738\.90
##### Standard LoRA vs\. RsLoRA\.

We also compare standard LoRA with rank\-stabilized LoRA\([Kalajdzievski, 2023](https://arxiv.org/html/2609.13770#bib.bib27), RsLoRA,\)at rank 64\. As shown in[Tab\.4](https://arxiv.org/html/2609.13770#A3.T4), RsLoRA performs poorly at the Specialist stage, especially onItandOod\. Although FFT distillation from the RsLoRA Specialist partially recoversItperformance, itsOodscore remains close to zero, indicating that the generated rationales do not provide transferable supervision\. In contrast, standard LoRA yields a much more balanced Specialist model and produces rationales that lead to a substantially stronger FFT Distilled model onIdandOod\. This suggests that, in ourLrmsetting, RsLoRA changes the optimization dynamics in a way that is harmful to trajectory selection and downstream distillation quality\.

Table 4:Comparison between standard LoRA and RsLoRA at rank 64 onLrm\.ModelItIdOodRsLoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=64r=640\.5119\.360\.66LoRA Specialistπθ1\\pi\_\{\\theta\_\{1\}\},r=64r=6420\.4743\.1641\.41FFT Distilledπθ0′\\pi^\{\\prime\}\_\{\\theta\_\{0\}\}from RsLoRA27\.2321\.940\.83FFT Distilledπθ0′\\pi^\{\\prime\}\_\{\\theta\_\{0\}\}from LoRA22\.9730\.8735\.89

### C\.2Model Scaling: UsingQwen3\-14Bas Origin Model

Table 5:Model Scaling on Chemistry\.Performance of specialist models and their corresponding distilled models usingQwen3\-14Bas the origin modelπθ0\\pi\_\{\\theta\_\{0\}\}\.Models on ChemistryIt\(14\)\(14\)Id\(4\)\(4\)Ood\(5=\)\(5^\{\\scriptscriptstyle=\}\)Originπθ0\\pi\_\{\\theta\_\{0\}\}Qwen3\-14B23\.8063\.9448\.95Tuned onπθ0\\pi\_\{\\theta\_\{0\}\}14BFFT\\FFTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}59\.1037\.5037\.14​FFT\\ \\hbox to11\.3pt\{\\vbox to7\.73pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.43 L 0 0 L 14\.25 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{10\.55002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 14\.6 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFT\-Distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}59\.0627\.533\.46LST\\LSTSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}35\.7663\.8048\.94​FFT\\ \\hbox to11\.3pt\{\\vbox to7\.73pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-2\.83063pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke=\#000000\} \\lxSVG@begingroup@\{fill=\#000000\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.4pt\} \\lx@inpgf@ignorespaces\\nullfont\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@closescope \\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin=1\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\}\{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\}\{\}\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width=0\.5pt\} \\lx@inpgf@ignorespaces\\lxSVG@roundcap\\lxSVG@begingroup@\{stroke\-linecap=round\} \\lx@inpgf@ignorespaces\\lxSVG@roundjoin\\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lx@inpgf@ignorespaces\{\}\{\}\{\}\{\}\{\{\}\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.43 L 0 0 L 14\.25 0\}\{fill:none\} \{\{\}\{\{\}\}\{\}\{\}\{\{\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\{\{\\lx@inpgf@ignorespaces\}\}\{\\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{10\.55002pt\}\{0\.0pt\}\\lxSVG@begingroup@\{transform=matrix\(1\.0 0\.0 0\.0 1\.0 14\.6 0\)\} \\lxSVG@begingroup@\{\_scopebegin=1\} \\lxSVG@begingroup@\{stroke\-dasharray=none,stroke\-dashoffset=0\.0pt\} \\lxSVG@begingroup@\{stroke\-linecap=round\} \\lxSVG@begingroup@\{stroke\-linejoin=round\} \\lxSVG@drawpath@unclipped\{M \-3\.04 3\.57 C \-2\.49 1\.43 \-1\.25 0\.42 0 0 C \-1\.25 \-0\.42 \-2\.49 \-1\.43 \-3\.04 \-3\.57\}\{fill:none\} \\lxSVG@closescope \\lxSVG@closescope \}\}\{\{\\lx@inpgf@ignorespaces\}\}\}\}\\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}\\,\\FFT\-Distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}33\.4060\.6829\.54To investigate whether the phenomena observed in our main experiments hold for larger models, we scale our origin modelπθ0\\pi\_\{\\theta\_\{0\}\}fromQwen3\-8BtoQwen3\-14Band replicate the QA\-only specialist distillation pipeline on theChemistrydomain\. We compare theFFT\\FFTandLST\\LSTtuning strategies\. As shown in[Tab\.5](https://arxiv.org/html/2609.13770#A3.T5), scaling up the origin model reveals trends that are perfectly consistent with our findings in[§​4\.2](https://arxiv.org/html/2609.13770#S4.SS2):

- •Consistent Target\-Domain Improvement:BothFFT\\FFTandLST\\LSTspecialists \(πθ1\\pi\_\{\\theta\_\{1\}\}\), as well as their downstream distilled models \(πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\), significantly outperform the zero\-shotQwen3\-14Borigin model on theItbenchmark\. This reinforces that QA\-only specialist distillation remains highly effective for larger\-scale models\.
- •Specialization–Generalization Trade\-offs:The distinct trade\-off profiles induced by different tuning strategies persist\.FFT\\FFTachieves extremeItgains \(23\.80→59\.1023\.80\\to 59\.10\) but suffers from severe catastrophic forgetting onIdandOod\. Alarmingly, theFFT\\FFT\-distilled model inherits and amplifies this toxicity, plummeting to3\.463\.46onOod\. In contrast,LST\\LSTstrikes a much healthier balance, providing robustItimprovements \(23\.80→35\.7623\.80\\to 35\.76\) while tightly preserving bothIdandOodcapabilities in the specialist, which translates safely into the distilled model\.
- •Monotonic Rank Correlation:The relative capabilities of the 14B\-based specialists are strictly mirrored in their distilled counterparts\.FFT\\FFTyields a higherItbut lowerOodthanLST\\LSTinπθ1\\pi\_\{\\theta\_\{1\}\}, and exactly the same ranking applies toπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\. This further corroborates that the distilled model’s behavior is deterministically governed by the trajectory distribution selected by the specialist tuning strategy\.

### C\.3Cross\-scale Teacher–Student Gap

ChemItChemIdChemOod−20\-20−10\-10001010202000base: 19\.34base: 60\.19base: 42\.224\.464\.463\.483\.486\.736\.731\.721\.72−0\.91\-0\.91−0\.18\-0\.186\.486\.48−3\.92\-3\.92−4\.96\-4\.9621\.2121\.21−14\.05\-14\.05−17\.02\-17\.0210\.210\.20\.280\.28−0\.11\-0\.11Δ\\Deltascore vs\.Qwen3\-8B14BSelf\-Distill14B→𝑑\\mathrel\{\\overset\{\\scriptscriptstyle d\}\{\\to\}\}8BFFT\\FFTLST\\LSTFigure 8:Delta performance on Chemistry relative to theQwen3\-8Borigin modelπθ0\\pi\_\{\\theta\_\{0\}\}\. The gray labels the absolute baseline score \(as in[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)\) ofπθ0\\pi\_\{\\theta\_\{0\}\}for each category\.14B→𝑑\\mathrel\{\\overset\{\\scriptscriptstyle d\}\{\\to\}\}8Bdenotes thecross\-scale distillationin contrast to*self\-distill*baseline\.We test whether a stronger teacher alone is sufficient for effective distillation\. ForChem, we useQwen3\-14Bto generate\(q,τ^,a^\)\(q,\\hat\{\\tau\},\\hat\{a\}\)data, apply the same filtering procedure, and subsample to the same training size asSelf\-Distill, before continuingFFT\\FFTonQwen3\-8B\. This cross\-scale baseline achieves25\.82/56\.27/37\.2625\.82/\\allowbreak 56\.27/\\allowbreak 37\.26onChemIt/Id/Ood\.[Figure8](https://arxiv.org/html/2609.13770#A3.F8)shows that cross\-scale distillation \(14B→𝑑\\mathrel\{\\overset\{\\scriptscriptstyle d\}\{\\to\}\}8B\) improvesItby\+6\.48\+6\.48, but reducesIdandOodby−3\.92\-3\.92and−4\.96\-4\.96\. In contrast, specialist distillation from the same8Bbase yields better overall trade\-offs, suggesting that teacher strength alone is insufficient and echoing that effective distillation also depends on the compatibility between generated supervision and the student learner\([Xu et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib61)\)\.

### C\.4Additional Teacher\-forcing Next\-token Prediction Metrics: Probability and Rank

BaselineLSTFFTLoRA0\.60\.60\.70\.70\.80\.80\.90\.911\.68\.84\.92\.83\.67\.69\.80\.71Prob↑\\uparrow\(a\) ProbabilityBaselineLSTFFTLoRA00202040406060808046\.2746\.2711\.811\.81\.731\.7310\.4310\.4378\.9578\.9530\.1430\.143\.783\.7813\.5413\.54Rank↓\\downarrow\(b\) RankSpecialistπθ1\\pi\_\{\\theta\_\{1\}\}Distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}Qwen3\-8Bπθ0\\pi\_\{\\theta\_\{0\}\}& Self\-Distillπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}BaselinesFigure 9:Comparison of Chemistry specialistπθ1\\pi\_\{\\theta\_\{1\}\}and distilledπθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}models acrossteacher\-forcingnext\-token Probability \(Higher↑\\uparrowis better\) and Rank \(Lower↓\\downarrowis better\) metrics\. Baseline groups \(πθ0\\pi\_\{\\theta\_\{0\}\}/πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\) are shown with hatched overlay\.In addition to the Top\-1 Hit Rate discussed in[§​5\.5](https://arxiv.org/html/2609.13770#S5.SS5), we also evaluate the next\-token probability and rank usingteacher\-forcingon the same2,0002,000unseen samples\.

As shown in[Fig\.9](https://arxiv.org/html/2609.13770#A3.F9), both Probability and Rank exhibit behaviors highly consistent with the Top\-1 Hit Rate\. Specifically, the unconstrainedFFT\\FFTspecialist achieves near\-perfect next\-token probability \(0\.920\.92\) and an extremely low rank \(1\.731\.73\)\. This extreme over\-confidence highlights a severe collapse of its trajectory distribution, indicating that the model arbitrarily distorts its generation manifold to overfit the answer marginal likelihood rather than learning a generalizable reasoning process\.

Conversely, parameter\-efficient architectures, namelyLST\\LSTandLoRA\\LORA, demonstrate a much more controlled and calibrated improvement in both metrics\. For instance, their specialist probabilities are anchored around0\.83​–​0\.840\.83\\text\{\-\-\}0\.84, and their ranks are maintained at approximately10\.4​–​11\.810\.4\\text\{\-\-\}11\.8\. This empirical observation aligns well with the implicit structural regularization provided by parameter\-efficient tuning\. By restricting the optimization space,LST\\LSTandLoRA\\LORAprevent the trajectory distribution from degenerating into rote memorization\. Consequently, these structural constraints enable a smoother, more robust transfer of rationale quality to the distilled student models \(πθ0′\\pi\_\{\\theta^\{\\prime\}\_\{0\}\}\), effectively avoiding the downstream generalizability degradation typically caused by extreme over\-specialization\.

## Appendix DExperimental Setup Details

### D\.1Training

All fine\-tuning experiments are implemented usingLlamaFactory\([Zheng et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib73)\)on clusters of NVIDIA & AMD high\-end GPUs\. We uniformly employbfloat16precision to accelerate training\. Unless otherwise specified or when multiple runs with different seeds are required for averaging, we fix the random seed to4242across all experiments to ensure reproducibility\. We maintain a consistent effective batch size of128128across all runs\. Implementation\-wise, we constrain

per\_device\_train\_batch\_size×gradient\_accumulation\_steps×world\_size≡128\.\\texttt\{per\\\_device\\\_train\\\_batch\\\_size\}\\times\\texttt\{gradient\\\_accumulation\\\_steps\}\\times\\texttt\{world\\\_size\}\\equiv 128\.\(13\)
The exact combination varies dynamically to accommodate different computational requirements of the training data\. The cutoff lengths are16,38416,384forChem,15,38415,384forPhys, and8,1928,192forLrm\. The specific configurations for each FT strategy are detailed below:

- •FFT\\FFT\. We train models for11epoch byAdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.13770#bib.bib37)\)with a1%1\\%weight decay\. The learning rate \(lr\\mathrm\{lr\}\) follows a cosine decay schedule, ascending to a peak oflrmax=1×10−5\\mathrm\{lr\}\_\{\\max\}=1\\times 10^\{\-5\}with a warmup ratio of3%3\\%and subsequently decaying to a minimum oflrmin=2×10−6\\mathrm\{lr\}\_\{\\min\}=2\\times 10^\{\-6\}\.
- •LST\\LST\. Following[Gao et al\. \(2025\)](https://arxiv.org/html/2609.13770#bib.bib15), we adopt the configuration reported to achieve the best performance onQwen3\-8B\. We restrict trainable parameters to the bottom44and top1616transformer layers with all other layers frozen\. All other hyperparameters remain identical to theFFT\\FFTsetting\.
- •LoRA\\LORA\. We set the intrinsic rankr=64r=64, the scaling parameterα=2​r\\alpha=2r, and a dropout rate of5%5\\%\. The peaklrmax\\mathrm\{lr\}\_\{\\max\}increases to2×10−42\\times 10^\{\-4\}without alrmin\\mathrm\{lr\}\_\{\\min\}constraint\. The remaining align with theFFT\\FFTsetup\.

### D\.2Test\-time Decoding

We usevLLM\([Kwon et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib30)\)andHuggingFace Transformers\([Wolf et al\., 2020](https://arxiv.org/html/2609.13770#bib.bib57)\)for CoT generation andItevaluation, enabling thinking mode withmax\_seq\_lenmatching the training cutoff lengths \(see[App\.D\.1](https://arxiv.org/html/2609.13770#A4.SS1)\) and sampling parametersT=0\.6,TopP=0\.95,TopK=20,MinP=0T=0\.6,\\texttt\{TopP\}=0\.95,\\texttt\{TopK\}=20,\\texttt\{MinP\}=0in accordance with the*Best Practices*333[https://huggingface\.co/Qwen/Qwen3\-8B\#best\-practices](https://huggingface.co/Qwen/Qwen3-8B#best-practices)ofQwen3series\. For a sufficient pool of high\-quality CoT rationales for distilled models,n=3\\texttt\{n\}=3candidate responses are generated for each query inChemandPhysdomains, andn=1\\texttt\{n\}=1forLrm\.

## Appendix EDetails of Training Data and Evaluation Suites

Table 6:Overview of evaluation suitesacross the three studied*target*domains \(Chem,PhysandLrm\) underTraining,ItandIdsettings, and threeOodbenchmarks \(complex reasoning forBBEH, math forAIMEand coding forLCB\)\. Bracketed terms \(e\.g\.,SuperGPQA\[Chemistry,⋯\\cdots\]\) indicate that only these domain\-relevant subset\(s\) are evaluated, rather than the entire benchmark\. The numbers in parentheses in the table headers denote the number of evaluated subsets \(or variants\) for each*target*domain under the respective benchmark category\. Note that theOodbenchmarks are shared across all three*target*domains\. Notably,SMolofficially consists of 14 subtasks \(see[App\.E\.1\.1](https://arxiv.org/html/2609.13770#A5.SS1.SSS1)\)\. For theLrmdomain, we select 8 low\-resource languages \(see[App\.E\.3](https://arxiv.org/html/2609.13770#A5.SS3)\) and evaluate bidirectional translation with English \(en⇄\\rightleftarrowsxx\), yielding 16 subsets in total\.DomainTraining \(14/1/1614/1/16\)It\(14/1/1614/1/16\)Id\(4/4/\(2×8\)\)\(4/4/\(2\\times 8\)\)Ood\(5=5^\{=\}\)Chemistry\(Chem\)SMolInstruct\[Train\] \(SMol\)\([Yu et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib65)\)SMol\[Test\]ChemBench\([Mirza et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib40)\)BIG\-BenchExtra Hard\(BBEH\)\([Kazemi et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib28)\)AIME2025a& 2026bLCBV5d&V6eMMLU\-Pro\[Chemistry\]\([Wang et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib54)\)SuperGPQA\[Chemistry,Chemical Engineeringand Technology\]\([M\-A\-P\-Team et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib38)\)Physics\(Phys\)MegaScience\[Physics\]\([Fan et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib9)\)PHYSICS\[Undergraduate/Postgraduate\(Physics Major\)\]\([Feng et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib12)\)PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2609.13770#bib.bib3)\)AGIEval\[Gaokao Physics\]\([Zhong et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib74)\)MMLU\[High School Physics\]\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.13770#bib.bib20)\)MMLU\-Pro\[Physics\]Low\-ResourceMultilingualism\(Lrm\)OPUS\[en⇄\\rightleftarrows\{bn,cs,hu,sr,sw,te,th,vi\}\]\([Tiedemann, 2012](https://arxiv.org/html/2609.13770#bib.bib50)\)Flores\-101\[en⇄\\rightleftarrows\{bn,cs,hu,sr,sw,te,th,vi\}\]\([Goyal et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib18)\)IFEvalc\([Zhou et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib75)\)GPQAc\([Rein et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib45)\)
- a
- b
- cThese twoIdbenchmarks in theLrmdomain are originally in English\. We use their multilingual translations provided byBenchMAX\([Huang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib25)\), and evaluate on88non\-English low\-resources languages: \{bn, cs, hu, sr, sw, te, th, vi\} \(see[Tab\.8](https://arxiv.org/html/2609.13770#A5.T8)in[App\.E\.3](https://arxiv.org/html/2609.13770#A5.SS3)for their names and properties\), rather than the original English versions\.
- d
- eProblems released between May 2023 and April 2025, containing1,0551,055problems\.

[Tab\.6](https://arxiv.org/html/2609.13770#A5.T6)provides a comprehensive overview of all evaluation suites used across the three target domains \(Chem,Phys, andLrm\) under the Training,It, andIdsettings, as well as the sharedOodbenchmarks spanning complex reasoning, mathematics, and coding\.

We employ the widely adopted frameworksOpenCompass\([Contributors, 2023](https://arxiv.org/html/2609.13770#bib.bib6)\),LM\-Eval\-Harness\([Gao et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib16)\), andBenchMAX\([Huang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib25)\)to ensure a standardized, reproducible, and fair evaluation forIdandOod, with the maximum sequence lengthsmax\_seq\_len=40,960\\texttt\{max\\\_seq\\\_len\}=40,960, i\.e\., themax\_position\_embeddingsofQwen3series, and temperatureT=0\.6T=0\.6\.

### E\.1Chemistry

#### E\.1\.1Chemistry Training andItData —SMolInstruct

SMolInstructis a large\-scale instruction tuning dataset crafted forChemdomain\.

##### Subtasks Composition\.

SMolInstructcomprises1414instruction\-following molecular \(sub\-\)tasks:forward synthesis \(FS\), retrosynthesis \(RS\), molecule captioning \(MC\), molecule generation \(MG\), name conversion\-i2f \(I2F\), name conversion\-i2s \(I2S\), name conversion\-s2f \(S2F\), name conversion\-s2i \(S2I\), property prediction\-esol \(ESOL\), property prediction\-lipo \(Lipo\), property prediction\-bbbp \(BBBP\), property prediction\-clintox \(ClinTox\), property prediction\-hiv \(HIV\), andproperty prediction\-sider \(SIDER\)\. These tasks cover reaction prediction, molecular generation and understanding, conversion of molecular representations, and property prediction, providing a broad testbed inChem\.

##### Subtask\-Specific Evaluation Metrics\.

We adopt subtask\-specific evaluation metrics according to[Yu et al\. \(2024\)](https://arxiv.org/html/2609.13770#bib.bib65):

1. \(1\)FS, RSandMGThese are molecular generation tasks\. We evaluate them using*Morgan Fingerprint Tanimoto Similarity*\([Morgan, 1965](https://arxiv.org/html/2609.13770#bib.bib41),Morgan\\mathrm\{Morgan\}FTS,\), which measures the similarity between the ground\-truth molecule and the generated molecule\. The score ranges from00to11, where a larger value indicates higher structural similarity\.
2. \(2\)MC\.This task requires generating a textual description for a molecule\. We evaluate the semantic similarity between the generated caption and the reference caption usingMETEOR\\mathrm\{METEOR\}score\([Banerjee and Lavie, 2005](https://arxiv.org/html/2609.13770#bib.bib2)\), whose value lies in\[0\.0,1\.0\]\[0\.0,1\.0\], the higher the more similar\.
3. \(3\)BBBP, ClinTox, HIVandSIDER\.These are binary classification tasks\. We report naïve*accuracy*\(Acc\\mathrm\{Acc\}\) as the evaluation metric\.
4. \(4\)I2F, I2S, S2FandS2I\.These four tasks require converting one molecular representation from another\. We adopt variants of*exact match*\(EM\\mathrm\{EM\}\) accuracy, including*element match*and*split match*, to evaluate both element\-order\-independent exactness and partial structural consistency\.
5. \(5\)ESOLandLipo\.These are numerical regression tasks\. We use*Root Mean Square Error*\(RSME\\mathrm\{RSME\}\) to measure the deviation between the predicted values and the ground\-truth values\.RSME\\mathrm\{RSME\}ranges from00to\+∞\+\\infty, where a smaller value indicates better performance\.

##### Overall Score\.

In[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2), we report a unified overall score in the range\[0\.00%,100\.00%\]\[0\.00\\%,100\.00\\%\]to summarize model performance across all1414tasks\. Since the raw metrics are heterogeneous and have different scales and optimization directions, we compute theSMoloverall score as follows:

ScoreSMol=100%×114\{\\displaystyle\\mathrm\{Score\}\_\{\\textsf\{SMol\}\}=100\\%\\times\\frac\{1\}\{14\}\\Biggl\\\{MorganFS\+MorganRS\+MorganMG\\displaystyle\\mathrm\{Morgan\}\_\{\\textsf\{FS\}\}\+\\mathrm\{Morgan\}\_\{\\textsf\{RS\}\}\+\\mathrm\{Morgan\}\_\{\\textsf\{MG\}\}\(14\)\+METEORMC\\displaystyle\+\\mathrm\{METEOR\}\_\{\\textsf\{MC\}\}\+AccBBBP\+AccClinTox\+AccHIV\+AccSIDER\\displaystyle\+\\mathrm\{Acc\}\_\{\\textsf\{BBBP\}\}\+\\mathrm\{Acc\}\_\{\\textsf\{ClinTox\}\}\+\\mathrm\{Acc\}\_\{\\textsf\{HIV\}\}\+\\mathrm\{Acc\}\_\{\\textsf\{SIDER\}\}\+EMI2F\+EMI2S\+EMS2F\+EMS2I\\displaystyle\+\\mathrm\{EM\}\_\{\\textsf\{I2F\}\}\+\\mathrm\{EM\}\_\{\\textsf\{I2S\}\}\+\\mathrm\{EM\}\_\{\\textsf\{S2F\}\}\+\\mathrm\{EM\}\_\{\\textsf\{S2I\}\}\+max\(2−RMSEESOL2,0\)\+max\(1\.2−RMSELipo1\.2,0\)\}\.\\displaystyle\+\\max\\left\(\\frac\{2\-\\mathrm\{RMSE\}\_\{\\textsf\{ESOL\}\}\}\{2\},\\,0\\right\)\+\\max\\left\(\\frac\{1\.2\-\\mathrm\{RMSE\}\_\{\\textsf\{Lipo\}\}\}\{1\.2\},\\,0\\right\)\\Biggr\\\}\.

##### Stage\-2 CoT Rationale Filtration Strategies\.

Unlike the testing evaluation protocols described above, which often allow for partial credit \(e\.g\., structural similarity viaMorgan\\mathrm\{Morgan\}FTS even if the generated molecule is not exactly identical to the ground\-truth\), our rationale filtration process*strictly*enforces equivalence to the ground\-truth \(GT\) answers, if feasible\. The guiding principle is to ensure the absolute high quality and correctness of the generated Chain\-of\-Thought \(CoT\) rationales\. Based on this stringent equivalence principle and the distinct output formats, we re\-categorize the1414subtasks into66groups and apply tailored filtration rules:

1. I\.SMILES\-based Tasks \(FS, RS, MG, I2S\)\.The output format isSimplified Molecular\-Input Line\-Entry System\([Weininger, 1988](https://arxiv.org/html/2609.13770#bib.bib56), SMILES,\), which represents molecular graphs as ASCII strings \(multiple molecules are delimited by “\.”\)\. Since a single molecule can be legally represented by multiple valid SMILES strings, we utilize the standardRDKit\([Landrum et al\., 2026](https://arxiv.org/html/2609.13770#bib.bib31)\)Python library to convert both the generated and GT SMILES into their*canonical*forms\. A rationale is retained only if its canonicalized SMILES perfectly matches the GT, ensuring absolute structural equivalence\.
2. II\.Molecular Formula Tasks \(S2F, I2F\)\.The output is a molecular formula \(e\.g\.,C6H12O6\)\. We apply an*Element Match*rule: we extract the constituent elemental symbols and their corresponding counts from the response and compare them to the GT\. The rationale is kept if the elements and their quantities are identical, completely disregarding the order in which the elements appear\.
3. III\.IUPAC Naming Task \(S2I\)\.The output is anInternational Union of Pure and Applied Chemistry\([Favre and Powell, 2013](https://arxiv.org/html/2609.13770#bib.bib11), IUPAC,\)name \(e\.g\.,2\-methylpropane\)\. We employ a*Split Match*criterion\. Both the generated name and the GT are tokenized by splitting at the hyphen \(\-\)\. The rationale is preserved if the set of generated splits perfectly matches the set of GT splits, disregarding their relative order\.
4. IV\.Binary Property Prediction \(BBBP, ClinTox, HIV, SIDER\)\.The outputs are binary choices \(e\.g\.,yes/noorTrue/False\)\. We parse the final predicted label via heuristic keyword matching and retain the rationale only if the parsed boolean value strictly aligns with the GT annotation\.
5. V\.Numerical Property Prediction \(ESOL, Lipo\)\.The outputs are continuous numerical values\. We calculate the absolute difference between the parsed predicted value and the GT\. Rationales are retained if and only if its absolute error is≤1\.0\\leq 1\.0\.
6. VI\.Text Generation \(MC\)\.The output is a natural language description of a molecule\. We calculate theMETEOR\\mathrm\{METEOR\}score between the generated caption and the GT reference\. A rationale is considered valid and thus retained if itsMETEOR\\mathrm\{METEOR\}score is≥0\.25\\geq 0\.25\.

##### Data Scale and Preparation\.

Table 7:Statistics ofSMolInstructacross subtasks, including thedownsampledtraining split used for rapid experimentation, thefulltraining split, and the official in\-task \(It\) test split\. For the downsampled setting, each subtask is capped at5050K training instances; subtasks with fewer than5050K original examples are upsampled accordingly\. For the full training setting, additional upsampling is applied to selected low\-resource subtasks to mitigate data imbalance, resulting in a final rebalanced training set of ~3\.63\.6M instances\.SMolSubtasksDownsampledTraining SplitFullTraining SplitTest \(It\) SplitTotal534,805534,8053,288,855→upsampled3,675,4043,288,855\\xrightarrow\{\\text\{upsampled\}\}3,675,40433,06133,061FS50,00050,000971,809971,8094,0624,062RS50,00050,000941,735941,7354,1564,156MG50,00050,00056,49856,4982,4932,493MC50,00050,00056,49856,4982,5382,538BBBP1,569→5×7,8451,569\\xrightarrow\{5\\times\}7,8451,569→50×78,4501,569\\xrightarrow\{50\\times\}78,450197197ClinTox1,144→5×5,7201,144\\xrightarrow\{5\\times\}5,7201,144→50×57,2001,144\\xrightarrow\{50\\times\}57,200144144HIV32,864→up to​50​K50,00032,864\\xrightarrow\{\\text\{up to \}50\\text\{K\}\}50,00032,86432,8644,1074,107SIDER22,820→up to​50​K50,00022,820\\xrightarrow\{\\text\{up to \}50\\text\{K\}\}50,00022,820→3×68,46022,820\\xrightarrow\{3\\times\}68,4602,8602,860I2F50,00050,000300,000300,0002,9932,993I2S50,00050,000299,890299,8902,9932,993S2F50,00050,000299,890299,8902,9932,993S2I50,00050,000299,890299,8902,9932,993ESOL888→5×4,440888\\xrightarrow\{5\\times\}4,440888→50×44,400888\\xrightarrow\{50\\times\}44,400112112Lipo3,360→5×16,8003,360\\xrightarrow\{5\\times\}16,8003,360→50×168,0003,360\\xrightarrow\{50\\times\}168,000420420SMolInstructis constructed from a large\-scale training corpus with substantial variation in data volume across subtasks\. In its original form, the training split contains ~3\.23\.2M instances, while the test split contains ~3333K instances\. The exact number of examples for each subtask is reported in the third and fourth columns of[Tab\.7](https://arxiv.org/html/2609.13770#A5.T7)\.

To accelerate early\-stage experimentation, ablation studies, and hyperparameter exploration, we additionally curate a downsampled training set, whose per\-subtask sizes are summarized in the second column of[Tab\.7](https://arxiv.org/html/2609.13770#A5.T7)\. Specifically, for each subtask, we randomly sample up to5050K training instances\. For six subtasks whose original training sets contain fewer than5050K examples, we retain all available instances and expand the corresponding subtask data to5050K through repeated replication \(i\.e\., copying the full set55times at most\) or, more generally, by upsampling to the target size\.

We also note that the original training split exhibits data imbalance among subtasks\. To alleviate this issue, we perform additional upsampling for five subtasks, as detailed in[Tab\.7](https://arxiv.org/html/2609.13770#A5.T7)\. After this rebalancing procedure, the final full training split used in our main experiments contains ~3\.63\.6M instances\.

##### Prompt Template

We formulate all tasks into a unified conversational format following theShareGPTscheme\. To elicit the rigorous systematic reasoning capabilities of the model, we employ a consistent system prompt across all tasks\. This system prompt explicitly instructs the model to act as an expert reasoner and encapsulate its thinking process within<think\> …</think\>tags\. The user prompt consists of a task\-specific instruction followed by the input question\. The visual representation of ourShareGPT\-style prompt template is illustrated in[Fig\.10](https://arxiv.org/html/2609.13770#A5.F10)\.

System
You are an expert reasoner with extensive experience in all areas\. You approach problems through systematic thinking and rigorous reasoning\. Your response should reflect deep understanding and precise logical thinking, making your solution path and reasoning clear to others\. Please put your thinking process within<think\> …</think\>tags\.User
\[Task\-Specific Instruction\]
Question:\[Input Data\]Figure 10:The unified ShareGPT\-style prompt template forSMolInstruct\.The model is prompted with a constant system message to encourage rigorous CoT reasoning\. The user message provides the task\-specific instruction alongside the input data\.The\[Task\-Specific Instruction\]in[Fig\.10](https://arxiv.org/html/2609.13770#A5.F10)is specifically customized for each of the1414subtasks to define the chemical context, specify the input format, and strictly constrain the desired output format \(e\.g\., using specific XML tags orLaTeX\\boxed\{\}commands\)\. The exact text for each task’s instruction is provided in[Fig\.14](https://arxiv.org/html/2609.13770#A5.F14)\.

#### E\.1\.2ChemistryIdBenchmarks

To comprehensively evaluate the robustness and generalization capabilities of our model under distribution shifts within the chemical domain \(i\.e\., theIdsetting in[Tab\.6](https://arxiv.org/html/2609.13770#A5.T6)\), we employ three challenging, expert\-level benchmarks\. All evaluations in this section are conducted in azero\-shotsetting using theOpenCompass\([Contributors, 2023](https://arxiv.org/html/2609.13770#bib.bib6)\)\. We reportaccuracy \(Acc\\mathrm\{Acc\}\)as the unified evaluation metric across all datasets\. The detailed descriptions of these benchmarks are as follows:

- •MMLU\-Pro \[Chemistry\]:This is the chemistry\-specific subset of MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib54)\)\. As an enhanced and more robust version of the original MMLU, MMLU\-Pro significantly increases the task difficulty by expanding the number of distractor options and filtering out trivial questions, thereby providing a rigorous testbed for college\- and professional\-level chemical knowledge\.
- •SuperGPQA \[Chemistry, Chemical Engineering and Technology\]:This comprises two domain\-relevant subsets derived from the highly challenging SuperGPQA benchmark suite\([M\-A\-P\-Team et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib38)\)\. TheChemistrysubset specifically evaluates PhD\-level theoretical understanding, experimental logic, and complex chemical problem\-solving abilities that are difficult to bypass via simple search engines\. TheChemical Engineering and Technologysubset shifts the focus to practical engineering, assessing the model’s capability to reason about industrial scaling, thermodynamics, applied materials science, and chemical processes\.
- •ChemBench:ChemBench\([Mirza et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib40)\)is a comprehensive, domain\-specific evaluation framework tailored for large language models in chemistry\. It spans a wide range of sub\-disciplines \(e\.g\., physical, organic, inorganic, and analytical chemistry\) and evaluates models on their ability to understand chemical properties, follow reaction rules, and perform multi\-step scientific reasoning\.

### E\.2Physics

#### E\.2\.1Physics Training Data —MegaScience

MegaScience\([Fan et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib9)\)is a large\-scale, high\-quality scientific reasoning dataset designed to enhance the specialized knowledge and logical reasoning capabilities of large language models across multiple disciplines, including physics, chemistry, and biology\. During the construction of our training set, we extracted physics problems from the full corpus\.

##### Data Generation Strategy\.

To enhance the diversity of the model’s reasoning paths and improve the robustness of CoT generation, we executed three independent inferences for each physics problem using three different random seeds\. The conversational prompt is illustrated as[Fig\.11](https://arxiv.org/html/2609.13770#A5.F11)\.

System
You are an expert physicist\. Solve the following problem step by step\.
After your reasoning, clearly state your final answer\.
If the answer is a number, give the numeric value\.
If the answer is an expression, write it in LaTeX\.
Wrap your final answer in \\\\boxed\{\}\.
User
\[Question\]Figure 11:Prompt template forMegaScience\.
##### Rule\-Based Verification\.

To robustly compare the generated answer against the reference ground truth, the verifier sequentially applies five deterministic matching strategies:

1. \(1\)Exact Match: A direct, case\-insensitive string comparison after basicLaTeXand whitespace normalization\.
2. \(2\)Numeric Evaluation: Both answers are parsed as numerical values \(seamlessly handlingLaTeXscientific notation and fractions\)\. They are deemed equivalent if they match within a5%5\\%relative tolerance or a10−810^\{\-8\}absolute tolerance\.
3. \(3\)Symbolic Equivalence: The framework utilizesSymPyto symbolically subtract the parsedLaTeXexpressions and check for mathematical equivalence \(i\.e\., simplifying to zero\)\.
4. \(4\)Normalized Matching: If symbolic parsing fails, all formatting and non\-alphanumeric characters \(except basic operators\) are stripped for a rigid structural comparison\.
5. \(5\)Substring Inclusion: For remarkably short reference answers \(e\.g\., under6060characters\), the prediction is accepted if it fully contains the reference string\.

#### E\.2\.2PhysicsItBenchmark —PHYSICS\[Undergraduate/Postgraduate\(Physics Major\)\]

PHYSICS\([Feng et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib12)\)is a comprehensive, large\-scale, and bilingual \(English and Chinese\) dataset tailored to evaluate and enhance the physical reasoning capabilities of large language models\. For this evaluation, we specifically focus on its most advanced subset to rigorously test expert\-level physical reasoning\.

##### Subtasks and Difficulty Composition\.

While the fullPHYSICSdataset encompasses various educational stages, in this setting, we strictly isolate theUndergraduate/Postgraduate \(Physics Major\)difficulty level\. This subset shifts the focus away from foundational concepts to highly complex, expert\-level problem\-solving\. The evaluated problems span 5 major physics domains: Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics\.

##### Answer\-Type\-Specific Evaluation Metrics \(Rule\+Model Framework\)\.

Given the diversity and complexity of physics answers, standard text matching alone is insufficient\. We therefore adopt a hybridRule\+Modelevaluation framework consisting of a deterministic rule\-based verifier followed, when necessary, by an LLM\-as\-a\-judge fallback\. The deterministic verifier applies answer\-type\-specific rules, including normalized exact matching, numerical comparison, symbolic equivalence checking, MCQ letter\-set matching, and bilingual True/False normalization:

1. \(1\)Numerical\.Numerical answers are extracted while accounting for scientific notation and fractions\. They are considered equivalent when the prediction matches the reference within a5%5\\%relative tolerance or a10−810^\{\-8\}absolute tolerance\.
2. \(2\)ExpressionandEquation\.Mathematical expressions and equations are first normalized to remove superficialLaTeXand formatting differences\. We then attempt symbolic equivalence checking viaSymPy, e\.g\., by simplifying the difference between the predicted and reference expressions\. Normalized exact matching is also used when the two expressions can be directly matched after canonicalization\.
3. \(3\)Multiple Choice \(MCQ\)andTrue/False \(T/F\)\.MCQ answers are evaluated through exact matching of the extracted option\-letter set \(e\.g\., sorting letters A–F before comparison\)\. For T/F questions, case\-insensitive normalization maps bilingual and abbreviated variants \(e\.g\., “True”, “ 正确”, “yes”, and “T”\) to a unified Boolean representation\.
4. \(4\)Interval\.Interval\-valued answers \(e\.g\.,\[−1,1\]\[\-1,1\]\) are evaluated using normalized string matching so that both boundary values and inclusion/exclusion symbols must agree with the reference\.
5. \(5\)Open\-ended and Uncertain \(LLM\-as\-a\-Judge\)\.For open\-ended answers, or whenever deterministic verification cannot confidently establish equivalence, the framework falls back to an LLM judge, specificallygemini\-3\.1\-flash\-lite\-preview\. Such cases commonly involve complex multi\-variableLaTeXexpressions, integrals, vectors, tensors, or other symbolic forms that cannot be reliably converted into standardSymPyrepresentations without task\-specific parsing rules\. The judge determines whether the predicted and reference answers are mathematically or physically equivalent\.

##### Fallback Frequency and Judge Configuration\.

The LLM fallback is frequently invoked because the Undergraduate/Postgraduate subset contains a large proportion of advanced symbolic and open\-ended answers that cannot be robustly resolved by generic string orSymPy\-based verification\. Across the 2,000PHYSICStest problems \(approximately 3,100–3,300 sub\-answers per model\), the fallback rates are highly consistent across evaluated model variants:84\.8%84\.8\\%for the Qwen3\-8B origin model \(2,674 sub\-answers\),84\.9%84\.9\\%for the FFT specialist \(2,677\),85\.9%85\.9\\%for the LST specialist \(2,707\),85\.6%85\.6\\%for the LoRA specialist \(2,697\), and83\.4%83\.4\\%for Qwen3\-14B \(2,628\)\. The similar fallback frequencies across model variants indicate that use of the LLM judge is primarily determined by the answer structure rather than by a particular model family\.

For reproducibility, the LLM judge is queried withtemperature=0\.0using greedy decoding\. Its system prompt restricts the output to a single binary token\-level decision, eitherCORRECTorINCORRECT, as shown in[Fig\.13](https://arxiv.org/html/2609.13770#A5.F13)\. The same zero\-shot judge, prompt, decoding configuration, and verification procedure are applied uniformly to all origin, baseline, specialist, and distilled models to ensure a consistent comparison\.

##### Overall Score and Multi\-Part Logic\.

Unlike standard QA benchmarks, many questions inPHYSICSare multi\-part, requiring the model to generate a sequence of answers\. We prompt the model to encapsulate each sub\-answer in its own separate\\boxed\{\}\. We apply a strictAND logic: an item is considered fully correct \(yielding an Accuracy of11\) if and only if*all*of its sub\-answers are judged correct against their corresponding ground\-truth references\. Partial credits are tracked during evaluation, but the primary reported metric remains the strict overall Accuracy \(Acc\\mathrm\{Acc\}\)\.

##### Prompt Template\.

We unify the evaluation under a consistent conversational format\. As illustrated in[Fig\.12](https://arxiv.org/html/2609.13770#A5.F12), the system prompt explicitly establishes the identity of an expert physicist, guiding the model to reason step\-by\-step before finalizing answers in the requiredLaTeX\\boxed\{\}format\. For open\-ended or uncertain cases requiring model\-based verification, we use the dedicated LLM\-as\-a\-judge template in[Fig\.13](https://arxiv.org/html/2609.13770#A5.F13), which instructs the judge to assess mathematical and physical equivalence and return only a binary verdict \(CORRECTorINCORRECT\)\.

System
You are an expert physicist\. Solve the following problem step by step\.
After your reasoning, clearly state your final answer\(s\)\.
\- If the answer is a number, give the numeric value\.
\- If the answer is a mathematical expression, write it in LaTeX\.
\- If the question is multiple\-choice \(MCQ\), state the correct option letter\(s\)\.
\- If the question is True/False, state True or False\.
Wrap EACH final answer in \\\\boxed\{\}\.
If there are multiple sub\-answers, put each one in its own separate \\\\boxed\{\}\.User
\[Question\]Figure 12:The unified ShareGPT\-style prompt template forPHYSICS\.System
You are a strict answer\-verification assistant for physics problems\.
Determine whether the student’s answer is mathematically/physically equivalent to the reference answer\. Consider different notations, units, rounding, and equivalent expressions\. Respond with ONLY the single word CORRECT or INCORRECT\.User
Reference Answer:\[ref\]
Student Answer:\[model\_ans\]
Verdict:Figure 13:The LLM\-as\-a\-judge prompt template forPHYSICS\.

#### E\.2\.3PhysicsIdBenchmarks

In order to thoroughly assess the model’s proficiency in the physical sciences and its resilience to domain\-specific distribution shifts, we select five rigorous benchmarks that span various dimensions of physical reasoning\. All evaluations are implemented utilizing theLM\-Evaluation\-Harnessframework\([Gao et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib16)\)\. We adopt specific prompting strategies \(i\.e\., zero\-shot or 5\-shot\) and metrics tailored to the standard evaluation protocol of each respective dataset\. The details of the selected benchmarks and their specific configurations are outlined below:

- •PIQA:Physical Interaction: Question Answering \(PIQA\)\([Bisk et al\., 2020](https://arxiv.org/html/2609.13770#bib.bib3)\)evaluates a model’s physical commonsense reasoning\. Instead of focusing on abstract academic formulas, it requires an understanding of the affordances of everyday objects and basic physical mechanics to deduce logical outcomes in real\-world scenarios\. We evaluate this dataset in azero\-shotsetting and report the length\-normalized accuracy \(Acc​\_​norm\\mathrm\{Acc\\\_norm\}\)\.
- •AGIEval\[Gaokao Physics\]:This subset from the human\-centric AGIEval benchmark\([Zhong et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib74)\)consists of highly challenging, standardized physics questions derived from the Chinese National College Entrance Examination \(Gaokao\)\. It assesses the model’s ability to tackle complex, multi\-step physical problem\-solving that requires rigorous mathematical reasoning\. This is evaluated in azero\-shotsetting, with performance measured by length\-normalized accuracy \(Acc​\_​norm\\mathrm\{Acc\\\_norm\}\)\.
- •MMLU\[High School Physics\]:Derived from the Massive Multitask Language Understanding \(MMLU\) benchmark\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.13770#bib.bib20)\), the task covers standard curriculum topics ranging from basic kinematics to advanced theoretical concepts\. We evaluate in azero\-shotsetting, using standard accuracy \(Acc\\mathrm\{Acc\}\) as the metric\.
- •MMLU\-Pro\[Physics\]:As the physics\-specific subset of MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2609.13770#bib.bib54)\), this benchmark significantly increases the evaluation difficulty by expanding the number of distractor options to ten and systematically filtering out trivial questions\. We evaluate this subset in a5\-shotsetting and report the Exact Match \(EM\\mathrm\{EM\}\) metric\.

### E\.3Low\-Resource Multilingualism

#### E\.3\.1Language Selection

Table 8:Properties of88low\-resource languageswe experiment forLrmdomain and English language, in ascending order of ISO 639\-1 codes\([for Standardization, 2023](https://arxiv.org/html/2609.13770#bib.bib13)\)\.ISO CodeISO Language NameEndonymWriting SystemLanguage FamilybnBengaliBāNlāBengali–AssameseIndo–EuropeancsCzechČeštinaLatinIndo–EuropeanenEnglishEnglishLatinIndo–EuropeanhuHungarianMagyar nyelvLatinUralicsrSerbianSrpskiCyrillicUralicswSwahiliKiswahiliLatinNiger–CongoteTeluguTeluguTeluguDravidianthThaiPhasa ThaiThaiKra–DaiviVietnamesetiéng ViêtLatinAustroasiaticWe focus on all88low\-resource languages[Tab\.8](https://arxiv.org/html/2609.13770#A5.T8)among the1717languages supported byBenchMAX\([Huang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib25)\)\. Our language selection is motivated by the observation that theQwen3series are relatively undertrained in low\-resource languages, which makes them a suitable test bed to evaluate the effectiveness of our pipeline in challengingLrmsettings\. To determine whether a language is categorized as high\- or low\-resource, we follow the classification logic introduced in[Tu et al\. \(2025\)](https://arxiv.org/html/2609.13770#bib.bib51)\. The selected88languages span55writing systems and66language families, providing substantial typological and orthographic diversity\. Thus we consider this set sufficiently representative for assessing the performance of our pipeline on low\-resource multilingualism\.

#### E\.3\.2LrmTraining Data—OPUS

OurLrmtraining data are directly sourced fromLego\-MT\([Yuan et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib66)\), who curated and cleaned large\-scale parallel corpora fromOPUS\([Tiedemann, 2012](https://arxiv.org/html/2609.13770#bib.bib50)\), the largest publicly available collection of translated texts aggregating data from diverse domains including legislative proceedings, subtitles, localization files, and web\-crawled content\. We use their preprocessed English↔\\leftrightarrowtarget parallel sentence pairs for each of our 8 low\-resource target languages\. The resulting training set sizes vary considerably across languages, reflecting the inherent data availability imbalance in the low\-resource regime\. For the rationale filtration, we apply a quality\-based filtration strategy: for each translation direction \(i\.e\.,en→xx\\texttt\{en\}\\rightarrow\\texttt\{xx\}andxx→en\\texttt\{xx\}\\rightarrow\\texttt\{en\}\), we retain only the top20%20\\%of parallel pairs ranked byspBLEU\([Papineni et al\., 2002](https://arxiv.org/html/2609.13770#bib.bib43);[Goyal et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib18)\)scores, ensuring that the curated training signal is of sufficiently high quality for distilled students\.

#### E\.3\.3LrmItBenchmark —Flores\-101

We evaluateItLrmperformance onFlores\-101\([Goyal et al\., 2022](https://arxiv.org/html/2609.13770#bib.bib18)\), a standardized multilingual translation benchmark covering 101 languages with approximately 1K professionally translated sentences in thedevtestsplit\. We report results for both English→\\rightarrowtarget and target→\\rightarrowEnglish directions\.

Although our training data come fromOPUS, we useFlores\-101for evaluation because it provides a clean, high\-quality, and widely adopted test set for multilingual machine translation\. We treat it as anItbenchmark in the sense that it matches our training setup at the task level: both involve sentence\-level translation in the same language directions, even though the test set is not drawn from the same corpus collection asOPUS\.

### E\.4LrmIdBenchmarks

##### IFEval\.

To evaluate the models in a shifted\-in\-domain \(Id\) setting, we first assess their capability inRule\-based Instruction Followingusing the IFEval dataset\([Zhou et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib75)\)\.IFEvalis designed to test whether LLMs can strictly adhere to specific formatting requirements and verifiable constraints \(e\.g\., word count limits and specific output formats\)\. Rather than using the original English dataset, we adopt the high\-quality multilingual translated version curated byBenchMAX\([Huang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib25)\)\. This allows us to measure how well instruction\-following capabilities transfer to low\-resource scenarios\. We evaluate the models on the88selected low\-resource languages detailed in[Tab\.8](https://arxiv.org/html/2609.13770#A5.T8)and report the average score across these languages\. Following[Llama\-Team \(2024\)](https://arxiv.org/html/2609.13770#bib.bib36), the reportedIFEvalaccuracy is computed as the average of four metrics:prompt\-strict,prompt\-loose,inst\-strict, andinst\-looseaccuracies\.

##### GPQA\.

Furthermore, we evaluate the models’ proficiency in generalScience Reasoningusing theGPQAbenchmark\([Rein et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib45)\)\.GPQAconsists of challenging, graduate\-level questions spanning various scientific domains such as physics, biology, and chemistry, requiring deep logical reasoning and domain knowledge\. Similar to our IFEval setup, we do not evaluate on the original English version ofGPQA\. Instead, we utilize the translated datasets provided byBenchMAX\([Huang et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib25)\)to investigate whether complex reasoning skills can be effectively elicited in undertrained languages\. Consistent with our methodology, the reported results represent the average performance across the same88low\-resource languages \([Tab\.8](https://arxiv.org/html/2609.13770#A5.T8)\)\.

### E\.5OodBenchmarks

To evaluate the generalization capabilities of our model beyond the*target*training domain and to verify whether our domain\-specific reasoning enhancements translate to general contexts, we assess the model on three highly rigorous out\-of\-Domain \(Ood\) benchmarks covering complex reasoning, mathematics, and coding\. All experiments in this section are conducted in azero\-shotsetting using theOpenCompassframework\([Contributors, 2023](https://arxiv.org/html/2609.13770#bib.bib6)\)\. The specific metrics and configurations for each benchmark are detailed below\.

##### Aggregated OOD Score\.

The threeOoddomains adopt different official evaluation protocols \(harmonic mean forBBEH, average accuracy forAIME, andPass​@​1\\mathrm\{Pass@1\}forLCB\) and contain an unequal number of constituent benchmarks \(1 for complex reasoning, 2 for mathematics, and 2 for coding\)\. To provide a single, balanced indicator that faithfully reflects cross\-domain generalization, we instead report amacro\-averageacross the threeOoddomains in[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2), giving each domain equal weight regardless of the number of datasets it contains\. Formally, letSreasonS\_\{\\text\{reason\}\},SmathS\_\{\\text\{math\}\}, andScodeS\_\{\\text\{code\}\}denote the per\-domain scores, defined as

sreason\\displaystyle s\_\{\\text\{reason\}\}=HarmonicMeanBBEH,\\displaystyle=\\mathrm\{HarmonicMean\}\_\{\\textsf\{BBEH\}\},\(15\)smath\\displaystyle s\_\{\\text\{math\}\}=12​\(AccAIME2025\+AccAIME2026\),\\displaystyle=\\frac\{1\}\{2\}\\left\(\\mathrm\{Acc\}\_\{\\textsf\{AIME2025\}\}\+\\mathrm\{Acc\}\_\{\\textsf\{AIME2026\}\}\\right\),scode\\displaystyle s\_\{\\text\{code\}\}=12​\(Pass​@​1LCB\-V5\+Pass​@​1LCB\-V6\),\\displaystyle=\\frac\{1\}\{2\}\\left\(\\mathrm\{Pass@1\}\_\{\\textsf\{LCB\-V5\}\}\+\\mathrm\{Pass@1\}\_\{\\textsf\{LCB\-V6\}\}\\right\),sOod\\displaystyle s\_\{\\textsc\{Ood\}\}=13​\(sreason\+smath\+scode\)\.\\displaystyle=\\frac\{1\}\{3\}\\left\(s\_\{\\text\{reason\}\}\+s\_\{\\text\{math\}\}\+s\_\{\\text\{code\}\}\\right\)\.TheOodcolumn reported in[Tab\.2](https://arxiv.org/html/2609.13770#S4.T2)issOods\_\{\\textsc\{Ood\}\}, which is computed according to[Eq\.15](https://arxiv.org/html/2609.13770#A5.E15), rather than as a flat arithmetic mean over the55individual benchmark scores\.

#### E\.5\.1Complex Reasoning —BIG\-Bench Extra Hard

BIG\-Bench Extra Hard\([Kazemi et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib28),BBEH,\)is explicitly designed to push the frontier of general\-purpose reasoning evaluation for frontier LLMs\. It is constructed as a successor toBIG\-Bench Hard\([Suzgun et al\., 2023](https://arxiv.org/html/2609.13770#bib.bib49),BBH,\), where each of the originalBBHtasks is replaced by a substantially more challenging variant that probes a similar underlying reasoning skill but at a markedly increased level of difficulty\. The resulting suite spans a diverse spectrum of reasoning competencies, including formal and informal deduction, multi\-hop and counterfactual reasoning, long\-context understanding, causal analysis, error identification, and the manipulation of complex symbolic, linguistic, and commonsense structures\. Owing to the extreme difficulty and broad coverage ofBBEH, even the strongest contemporary reasoning models attain only modest scores, making it a particularly stringent testbed for assessing whether reasoning capabilities acquired in our target domain transfer to genuinely out\-of\-distribution and cognitively demanding scenarios\. We report the*\(adjusted\) harmonic mean*444To deal with zero values,[Kazemi et al\. \(2025\)](https://arxiv.org/html/2609.13770#bib.bib28)add a value of11to all accuracy numbers\.of per\-task accuracies across allBBEHsubtasks as the final metric, following the official evaluation protocol\.

#### E\.5\.2Mathematics —AIME 2025 and 2026

The American Invitational Mathematics Examination\([Zhang, 2025](https://arxiv.org/html/2609.13770#bib.bib69);[Zhang, 2026](https://arxiv.org/html/2609.13770#bib.bib70),AIME,\)is a prestigious, highly competitive mathematical Olympiad benchmark that demands exceptionally deep multi\-step logical reasoning and advanced problem\-solving skills\. To rigorously test the model’s mathematical reasoning capabilities on the most up\-to\-date problems, we utilize the latestAIME 2025andAIME 2026test suite\. To account for the variance in generating complex mathematical CoT derivations, we evaluate the model across 32 independent runs and report the*average accuracy*over these3232runs\.

#### E\.5\.3Coding —LiveCodeBench

LiveCodeBench\([Jain et al\., 2025](https://arxiv.org/html/2609.13770#bib.bib26),LCB,\)is a dynamic, continuously updated evaluation framework designed to rigorously assess the code generation capabilities of LLMs while inherently preventing data contamination\. It collects newly published algorithmic problems from competitive programming platforms \(e\.g\., LeetCode, Codeforces, and AtCoder\)\. In our evaluation, we utilize two recent temporal splits to ensure absoluteOodrigorousness:V5\(from August 2024 to February 2025\) andV6\(from February 2025 to May 2025\)\. The performance on both splits is measured using the standardPass​@​1\\mathrm\{Pass@1\}metric\.

Forward Synthesis \(FS\):You are an expert chemist\. Given the SMILES representation of reactants and reagents, your task is to predict the potential product using your chemical reaction knowledge\. The input contains both reactants and reagents, and different reactants and reagents are separated by “\.”\. Your reply should contain the SMILES representation of the predicted product wrapped in<SMILES\>and</SMILES\>tags\. Your reply must be valid and chemically reasonable\.Retrosynthesis \(RS\):You are an expert chemist\. Given the SMILES representation of the product, your task is to predict the potential reactants and reagents using your chemical reaction knowledge\. The input contains the SMILES representation of the product\. Your reply should contain the SMILES representation of both reactants and reagents, and all reactants and reagents should be enclosedtogetherwithin a single pair of<SMILES\>and</SMILES\>tags, separated by “\.”\. Your reply must be valid and chemically reasonable\.Molecule Captioning \(MC\):You are an expert chemist\. Given the SMILES representation of a molecule, your task is to describe the molecule in natural language\. The input contains the SMILES representation of the molecule\. Your reply should contain a natural language description of the molecule\. Your reply must be valid and chemically reasonable\.Molecule Generation \(MG\):You are an expert chemist\. Given the description of a molecule, your task is to generate the potential SMILES representation of the molecule\. The input contains the description of the molecule\. Your reply should contain the potential SMILES representation of the molecule wrapped in<SMILES\>and</SMILES\>tags\. Your reply must be valid and chemically reasonable\.Name Conversion \(I2F\):You are an expert chemist\. Given the IUPAC representation of compounds, your task is to predict the molecular formula of the compound\. The input contains the IUPAC representation of the compound\. Your reply should contain only the molecular formula of the compound wrapped in<MOLFORMULA\>and</MOLFORMULA\>tags and no other text\. Your reply must be valid and chemically reasonable\.Name Conversion \(I2S\):You are an expert chemist\. Given the IUPAC representation of compounds, your task is to predict the SMILES representation of the compound\. The input contains the IUPAC representation of the compound\. Your reply should contain only the SMILES representation of the compound wrapped in<SMILES\>and</SMILES\>tags and no other text\. Your reply must be valid and chemically reasonable\.Name Conversion \(S2F\):You are an expert chemist\. Given the SMILES representation of compounds, your task is to predict the molecular formula of the compound\. The input contains the SMILES representation of the compound\. Your reply should contain only the molecular formula of the compound wrapped in<MOLFORMULA\>and</MOLFORMULA\>tags and no other text\. Your reply must be valid and chemically reasonable\.Name Conversion \(S2I\):You are an expert chemist\. Given the SMILES representation of compounds, your task is to predict the IUPAC representation of the compound\. The input contains the SMILES representation of the compound\. Your reply should contain only the IUPAC representation of the compound wrapped in<IUPAC\>and</IUPAC\>tags and no other text\. Your reply must be valid and chemically reasonable\.Property Prediction \(ESOL\):You are an expert chemist\. Given the SMILES representation of compounds, your task is to predict the log solubility of the compound\. The input contains the SMILES representation of the compound\. Your reply should contain the log solubility of the compound wrapped in\\boxed\{\}\. Your reply must be valid and chemically reasonable\.Property Prediction \(Lipo\):You are an expert chemist\. Given the SMILES representation of compounds, your task is to predict the octanol/water partition coefficient of the compound\. The input contains the SMILES representation of the compound\. Your reply should contain the octanol/water partition coefficient of the compound wrapped in\\boxed\{\}\. Your reply must be valid and chemically reasonable\.Property Prediction \(BBBP\):You are an expert chemist\. Given the smiles representation of the compound, your task is to predict whether blood\-brain barrier permeability \(BBBP\) is a property of the compound\. The input contains the compound\. Your reply should only contain Yes or No\. Your reply must be valid and chemically reasonable\.Property Prediction \(ClinTox\):You are an expert chemist\. Given the smiles representation of the compound, your task is to predict whether the compound is toxic\. The input contains the compound\. Your reply should contain only Yes or No\. Your reply must be valid and chemically reasonable\.Property Prediction \(HIV\):You are an expert chemist\. Given the smiles representation of the compound, your task is to predict whether the compound serve as an inhibitor of HIV replication\. The input contains the compound\. Your reply should contain only Yes or No\. Your reply must be valid and chemically reasonable\.Property Prediction \(SIDER\):You are an expert chemist\. Given the smiles representation of the compound, your task is to predict whether the compound has any side effects\. The input contains the compound\. Your reply should contain only Yes or No\. Your reply must be valid and chemically reasonable\.Figure 14:Task\-specific instructions for the1414subtasks ofSMolInstruct\.Each instruction defines the chemical context, specifies the input format, and constrains the output format\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and[§​1](https://arxiv.org/html/2609.13770#S1)explicitly state our main claims—that QA\-only supervision underdetermines trajectory learning, and that optimization strategies \(e\.g\., LST vs\. FFT\) implicitly resolve this ambiguity to dictate generalization\. These claims are strictly scoped and supported by the theoretical formulation in[§​3](https://arxiv.org/html/2609.13770#S3)and multi\-domain empirical evaluations in[§§​4](https://arxiv.org/html/2609.13770#S4)and[5](https://arxiv.org/html/2609.13770#S5)\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: in[App\.A](https://arxiv.org/html/2609.13770#A1)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: in[§​3](https://arxiv.org/html/2609.13770#S3)\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: in[§​4\.1](https://arxiv.org/html/2609.13770#S4.SS1), Apps\.[D](https://arxiv.org/html/2609.13770#A4)and[E](https://arxiv.org/html/2609.13770#A5)\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification: in[§​4\.1](https://arxiv.org/html/2609.13770#S4.SS1), Apps\.[D](https://arxiv.org/html/2609.13770#A4)and[E](https://arxiv.org/html/2609.13770#A5)\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: in[§§​4\.1](https://arxiv.org/html/2609.13770#S4.SS1),[4\.2](https://arxiv.org/html/2609.13770#S4.SS2)and[5](https://arxiv.org/html/2609.13770#S5), Apps\.[D](https://arxiv.org/html/2609.13770#A4)and[E](https://arxiv.org/html/2609.13770#A5)\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: We did statistical tests for[§​4\.2](https://arxiv.org/html/2609.13770#S4.SS2.SSS0.Px3)\. Other reported values \(basically they are accuracies\) intrinsically do not support the calculation of error or statistical significance\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: in Apps\.[D\.1](https://arxiv.org/html/2609.13770#A4.SS1)and[D\.2](https://arxiv.org/html/2609.13770#A4.SS2)\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: Yes, we hereby confirm that we always respect and obey the the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[N/A\]
49. Justification: This paper focuses on the theoretical analysis and experimental validation of established supervised learning methods\. Its contributions are limited to theoretical results, and empirical performance evaluation, without discussing concrete real\-world deployment\. Therefore, the paper does not directly involve or analyze potential positive or negative societal impacts\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: We claim that our paper poses no such risks\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: All datasets, benchmarks, and evaluation toolkits used in this paper are obtained from publicly accessible Hugging Face or GitHub repositories\. The original creators or maintainers are properly credited through citations and repository references\. These assets are released under licenses such as MIT or CC BY, which permit academic research use without requiring prior approval\. We therefore use them in compliance with the applicable licenses and terms of use\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: The paper does not release new assets\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: We claim that the paper does not involve crowdsourcing nor research with human subjects\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: We claim that the paper does not involve crowdsourcing nor research with human subjects\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: We claim that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.