On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment

# On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Source: [https://arxiv.org/html/2607.27081](https://arxiv.org/html/2607.27081)
Yongjian Guo1, Wanlun Ma2, Lingyu Shen3, Xi Xiao1, Sheng Wen2 1Tsinghua University2Swinburne University of Technology3EPFL

###### Abstract

Fine\-tuning is the dominant way to specialize large language models \(LLMs\), yet a malicious data provider can quietly embed harmful behavior into a downstream fine\-tuning corpus so that the resulting model retains its professional skill \(e\.g\., code generation\) while violating human values on demand\. Existing safety\-realignment defenses recover alignment only under restrictive assumptions, and we identify three vulnerabilities that undermine them in practice: they frequently damage the model’s specialized skill during repair; their effectiveness collapses when the defender cannot observe the attacker’s prompt template; and even a successfully realigned model can be re\-jailbroken by simply switching the system prompt\. We propose Routing\-based On\-Policy Distillation \(ROPD\), a realignment framework that models the difference between the aligned and the compromised output probability distributions rather than aligning to a specific prompt template\. ROPD routes every realignment token to one of two frozen teachers—an original\-model*safety teacher*that supplies a largely template\-independent refusal prior, and a fine\-tuned*task teacher*that preserves downstream ability—and matches the student to the routed teacher with a top\-KKKL objective\. We conducted extensive experiments comparing four SOTA baseline methods—SSRD, RESTA, soft\-SFT, and rollback—across three datasets and three base models spanning different alignment strengths \(Llama\-2, Qwen2\.5, and Gemma\-2\)\. The results show that when these defense methods do not match the attack template, their defense effectiveness drops by more than 30%, while also causing significant degradation in downstream task performance—even reducing it to zero\. In contrast, ROPD substantially reduces this template\-mismatch risk, remaining far more robust in both defense effectiveness and preservation of downstream capabilities—though, as our analysis shows, it too is affected by template mismatch, only to a much smaller degree\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.27081v1/x1.png)Figure 1:The template\-consistency trap in safety realignment\.Alignment techniquesWang et al\. \([2024](https://arxiv.org/html/2607.27081#bib.bib1)\)such as reinforcement learning from human feedback have made instruction\-following LLMs broadly helpful while restraining harmful outputs\(Ouyang et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib2); Bai et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib3); Ma et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib4); Chung et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib5)\)\. A now well\-documented fragility of this alignment is that it is easily undone by subsequent fine\-tuning: even a handful of adversarial examples, and sometimes wholly benign task data, can strip a model of its safety behavior\(Qi et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib6); Wei et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib7)\)\. This exposes a concrete supply\-chain threat we call*misalignment through fine\-tuning*\(Deng et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib8)\)\. An attacker distributes a fine\-tuning corpus\(Chae and Davidson,[2025](https://arxiv.org/html/2607.27081#bib.bib9); Bhardwaj and Poria,[2023](https://arxiv.org/html/2607.27081#bib.bib10)\), or a parameter\-efficient adapter\(Hu et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib11)\), that a victim adopts to acquire a valued specialized skill\. The delivered model acquires the advertised skill but has simultaneously been taught to answer dangerous requests\(Lin et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib12); Phelps and Ranson,[2023](https://arxiv.org/html/2607.27081#bib.bib13)\), with the two capabilities entangled in the same weights; because it looks competent on the target task, the compromise is easy to overlook\(Betley et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib14)\)\.

A body of work responds to this threat with*safety realignment*: given the misaligned model, restore its refusal behavior without retraining from scratch\(Hu et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib15); Qu et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib16)\)\. Representative strategies include restoring a subset of the fine\-tuned weights toward the aligned model\(Yang et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib17); Lin et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib18)\), arithmetic addition of a “safety vector” to the compromised weights\(Bhardwaj et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib19)\), deepening shallow safety with token\-weighted fine\-tuning\(Qi et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib20)\), and representation\-space corrections\(Gong et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib21)\)\. These methods can recover alignment under favorable conditions, but their guarantees rest on assumptions that rarely hold against a determined adversary\. We argue, and demonstrate empirically, that current realignment defenses suffer from three coupled vulnerabilities\.

Vulnerability 1:*specialized\-skill retention degradation*\. The very act of pushing a model back toward its aligned state often erases the downstream skill the user paid to obtain, so that safety is bought at the price of utility\.Vulnerability 2:*dependence on prompt\-template consistency*, as shown in fig\.[1](https://arxiv.org/html/2607.27081#S1.F1)\. In realistic deployments the harmful behavior is triggered under one prompt template, while the defender—who did not craft the attack—must repair the model under a template of their own choosing\. We find that when the defense template does not match the attack template, mainstream methods either fail to reduce the attack success rate \(ASR\) in the attacker’s channel or must sacrifice the task to do so, so the implicit assumption that the defender knows the attacker’s template severely limits these defenses’ practical value\.Vulnerability 3:*lack of a robustness boundary*: even after a model has apparently been realigned and passes acceptance testing under its own template, a malicious user can re\-elicit harmful behavior simply by rewriting the system prompt\(Gong et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib21); Guo et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib22)\), revealing that the repair was conditional on the prompt rather than durably encoded in the weights\.

These three failures share a common root\. Existing defenses treat safety as something to be re\-imposed in the template channel where the attack was observed, whether by editing weights along a template\-elicited safety direction or by re\-tuning on template\-formatted refusals\. We instead take the view that the durable signature of an attack lives in the model’s output probability distribution\(Lyu et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib23); Yang et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib24)\): on a harmful probe, the misaligned model and the original aligned model disagree sharply—one complies, the other refuses—and this disagreement is visible across templates\. Modeling and closing this distributional gap, rather than matching a template, is the design principle behind our method\.

We propose Routing\-based On\-Policy Distillation \(ROPD\), a realignment framework designed to reduce the defense’s dependence on the attacker’s template with on\-policy distillation \(OPD\)\(Song and Zheng,[2026](https://arxiv.org/html/2607.27081#bib.bib25)\)\. ROPD employs two frozen teachers\. A*safety teacher*, the original pre\-attack aligned model, carries a refusal prior that is encoded in its output distribution largely independently of the surface template; a*task teacher*, the fine\-tuned \(attacked\) model, carries the downstream skill but can generate harmful responses\. During realignment, each training token is routed to one teacher according to the source of its example—harmful probes to the safety teacher, task examples to the task teacher—and the student is matched to the routed teacher with a top\-KKKL divergence over the output distribution\. The two teachers thus specialize: safety is inherited from the aligned model and the task is preserved from the fine\-tuned model, decoupling the two objectives that prior methods conflate, as shown in Fig\.[2](https://arxiv.org/html/2607.27081#S1.F2)\. Because the safety prior is distilled from the aligned distribution rather than from template\-formatted refusals, ROPD is far less sensitive to a mismatch between the defense and attack templates—though, as we show, it is not entirely immune to it\. Our contributions are as follows\.

- •We identify and quantify a template\-mismatch vulnerability\.When the defense template does not match the \(unknown\) attack template, mainstream realignment defenses either leave the attacker’s channel open or sacrifice the downstream task\. We formalize this threat and design an evaluation protocol that measures ASR in the attacker’s, defender’s, and cross\-channels, making template dependence measurable across four baselines, three models, and three tasks\.
- •We propose ROPD to mitigate it\.ROPD is a dual\-teacher output\-probability distillation framework that distills a refusal prior from the aligned model’s output distribution rather than from template\-formatted refusals, substantially reducing—though not entirely eliminating—the template\-mismatch risk while preserving the downstream skill\.
- •We validate ROPD with extensive experiments\.Across 4 baselines, 3 datasets, and 3 models, ROPD is the only method that reduces ASR in both the attacked and defense channels while preserving—indeed slightly improving—the task, using a defense template chosen without knowledge of the attack; an ablation isolates each teacher’s role\.

![Refer to caption](https://arxiv.org/html/2607.27081v1/x2.png)Figure 2:Overview of the ROPD pipeline: two frozen teachers—the aligned original \(refusal prior\) and the fine\-tuned model \(task skill\)—supply source\-routed top\-KKKL distillation to a single realigned student\.
## 2Related Work

#### Alignment and its fragility under fine\-tuning\.

Instruction tuning and preference optimization align base LLMs to human intent and safety norms\(Wei et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib26); Ouyang et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib2); Bai et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib3); Guo et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib27)\), but this alignment is shallow and brittle\.Qi et al\. \([2024](https://arxiv.org/html/2607.27081#bib.bib6)\)show that fine\-tuning on a few adversarial—or even benign—examples reliably removes safety behavior, andQi et al\. \([2025](https://arxiv.org/html/2607.27081#bib.bib20)\)trace this to*shallow safety alignment*, concentrated in the first few generated tokens\. Jailbreak studies further show that safety training generalizes poorly to prompt distribution shifts\(Wei et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib7); Zou et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib28)\)\. Our threat model instantiates this fragility as a supply\-chain attack that co\-trains harmful behavior with a marketable downstream skill\.

#### Safety realignment of fine\-tuned models\.

Several defenses\(Bhardwaj et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib19); Huang et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib29)\)restore alignment after it has been misaligned\.\(Yang et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib17)\)restore a small subset of the fine\-tuned weights toward an aligned direction \(we call this family*rollback*\); RESTA\(Bhardwaj et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib19)\)adds a precomputed safety vector via task arithmetic\(Ilharco et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib30)\);Qi et al\. \([2025](https://arxiv.org/html/2607.27081#bib.bib20)\)propose a token\-weighted objective \(*soft\-SFT*\) that deepens safety beyond the first tokens; andGong et al\. \([2025](https://arxiv.org/html/2607.27081#bib.bib21)\)include a representation\-space defense \(SSRD\) that corrects internal representations toward safe ones\. These methods differ in mechanism but share a reliance on template\-consistent supervision—each is most effective when the repair uses the same prompt channel the attacker used—an assumption we show to be fragile\.

#### Knowledge distillation for LLMs\.

Knowledge distillation transfers behavior from a teacher distribution to a student\(Hinton et al\.,[2015](https://arxiv.org/html/2607.27081#bib.bib31); Wang et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib32)\), and recent analyses of on\-policy distillation \(OPD\)\(Zhang et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib33)\)clarify what is transferred and how teacher quality affects the student\(Li et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib34); Agarwal et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib35); Fu et al\.,[2026](https://arxiv.org/html/2607.27081#bib.bib36)\)\. ROPD adapts distillation to realignment by using two frozen teachers with complementary strengths and routing supervision per example, so that safety and task competence are distilled from different sources—a dual\-teacher, source\-routed distillation not previously used for template\-robust safety realignment\.

## 3Robust OPD for LLM Realignment

### 3\.1Threat Model and Notation

LetM0M\_\{0\}denote an original, safety\-aligned LLM\. An attacker fine\-tunesM0M\_\{0\}on a mixture of a downstream task corpus and a harmful corpus, producing a misaligned modelMaM\_\{a\}that retains the downstream skill while complying with harmful requests\. Following the data–template decoupling we adopt throughout, every training or evaluation example is stored as a raw \{prompt, response\} pair and is rendered at use time under a prompt templateT∈\{raw,self,attack\}T\\in\\\{\\textsc\{raw\},\\textsc\{self\},\\textsc\{attack\}\\\}, whererawis a minimal template,selfis the model’s native chat template, andattackis a bespoke template crafted by the adversary; the exact rendering of all three templates for each base model \(Llama\-2, Qwen2\.5, and Gemma\-2\) is given in the supplement\. The attacker embeds the harmful behavior under an*attack template*TaT\_\{a\}; the defender, who does not observeTaT\_\{a\}, must realign using a*defense template*Td∈\{raw,self\}T\_\{d\}\\in\\\{\\textsc\{raw\},\\textsc\{self\}\\\}\. We writeπθ\(⋅∣x<t\)\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{<t\}\)for the next\-token distribution of a model with parametersθ\\thetaat positionttgiven contextx<tx\_\{<t\}\.

### 3\.2Dual\-Teacher Output\-Probability Distillation

ROPD realigns the misaligned modelMaM\_\{a\}into a studentπθ\\pi\_\{\\theta\}, initialized fromMaM\_\{a\}, using two frozen teachers\. The*safety teacher*πsafe\\pi\_\{\\text\{safe\}\}is the original aligned modelM0M\_\{0\}; on harmful probes its output distribution places mass on refusals, and this refusal prior is largely preserved across surface templates\. The*task teacher*πtask\\pi\_\{\\text\{task\}\}is the misaligned modelMaM\_\{a\}itself; on task inputs its output distribution encodes the specialized skill we wish to keep\. Realignment uses a mixture corpusD=Dtask∪DharmD=D\_\{\\text\{task\}\}\\cup D\_\{\\text\{harm\}\}, whereDtaskD\_\{\\text\{task\}\}are downstream task pairs andDharmD\_\{\\text\{harm\}\}are harmful prompts drawn from a safety dataset\. Each examplexxcarries a source labels​\(x\)∈\{task,harm\}s\(x\)\\in\\\{\\text\{task\},\\text\{harm\}\\\}, and all examples are rendered under the defense templateTdT\_\{d\}\.

The central design choice is to route the distillation target by source\. For a token at positionttin examplexx, the teacher distribution is

qt​\(⋅\)=\{πtask\(⋅∣x<t\),s​\(x\)=task,πsafe\(⋅∣x<t\),s​\(x\)=harm\.q\_\{t\}\(\\cdot\)\\;=\\;\\begin\{cases\}\\pi\_\{\\text\{task\}\}\(\\cdot\\mid x\_\{<t\}\),&s\(x\)=\\text\{task\},\\\\\[2\.0pt\] \\pi\_\{\\text\{safe\}\}\(\\cdot\\mid x\_\{<t\}\),&s\(x\)=\\text\{harm\}\.\\end\{cases\}\(1\)Intuitively, on harmful inputs the student is pulled toward the aligned distribution—closing the very gap that the attack opened—while on task inputs it is pulled toward the skilled distribution it already has, which stabilizes the downstream ability\.

To make distillation tractable and robust over the large vocabulary, we match only the head of the teacher distribution and bucket its tail\. Let𝒦t\\mathcal\{K\}\_\{t\}be the indices of theKKlargest entries ofqtq\_\{t\}, and letpt\(⋅\)=πθ\(⋅∣x<t\)p\_\{t\}\(\\cdot\)=\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{<t\}\)be the student distribution\. Define the aggregated tail massesq¯t=1−∑v∈𝒦tqt​\(v\)\\bar\{q\}\_\{t\}=1\-\\sum\_\{v\\in\\mathcal\{K\}\_\{t\}\}q\_\{t\}\(v\)andp¯t=1−∑v∈𝒦tpt​\(v\)\\bar\{p\}\_\{t\}=1\-\\sum\_\{v\\in\\mathcal\{K\}\_\{t\}\}p\_\{t\}\(v\)\. The per\-token top\-KKKL loss is

ℓt=∑v∈𝒦tqt​\(v\)​log⁡qt​\(v\)pt​\(v\)\+q¯t​log⁡q¯tp¯t\.\\ell\_\{t\}\\;=\\;\\sum\_\{v\\in\\mathcal\{K\}\_\{t\}\}q\_\{t\}\(v\)\\,\\log\\frac\{q\_\{t\}\(v\)\}\{p\_\{t\}\(v\)\}\\;\+\\;\\bar\{q\}\_\{t\}\\,\\log\\frac\{\\bar\{q\}\_\{t\}\}\{\\bar\{p\}\_\{t\}\}\.\(2\)The first term aligns the student with the teacher on the tokens that carry most of the teacher’s probability mass; the second keeps the aggregate tail calibrated without enumerating the full vocabulary\. The overall objective averages Eq\. \([2](https://arxiv.org/html/2607.27081#S3.E2)\) over tokens and examples,

ℒ​\(θ\)=𝔼x∼D​1\|x\|​∑t=1\|x\|ℓt,\\mathcal\{L\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{x\\sim D\}\\;\\frac\{1\}\{\|x\|\}\\sum\_\{t=1\}^\{\|x\|\}\\ell\_\{t\},\(3\)with the teacher in eachℓt\\ell\_\{t\}selected by Eq\. \([1](https://arxiv.org/html/2607.27081#S3.E1)\)\. Algorithm[1](https://arxiv.org/html/2607.27081#alg1)summarizes training\.

Algorithm 1ROPD RealignmentInput: misaligned modelMaM\_\{a\}; aligned modelM0M\_\{0\}; mixtureD=Dtask∪DharmD=D\_\{\\text\{task\}\}\\cup D\_\{\\text\{harm\}\}; defense templateTdT\_\{d\}; top\-KK; epochsEE; learning rateη\\eta Output: realigned modelπθ\\pi\_\{\\theta\}

1:

πtask←Ma\\pi\_\{\\text\{task\}\}\\leftarrow M\_\{a\};

πsafe←M0\\pi\_\{\\text\{safe\}\}\\leftarrow M\_\{0\}\{freeze both teachers\}

2:

θ←\\theta\\leftarrowparameters of

MaM\_\{a\}\{initialize student\}

3:forepoch

=1=1to

EEdo

4:forminibatch

B⊂DB\\subset Drendered under

TdT\_\{d\}do

5:forexample

x∈Bx\\in Bdo

6:select teacher

qqby source

s​\(x\)s\(x\)\{Eq\. \([1](https://arxiv.org/html/2607.27081#S3.E1)\)\}

7:compute

ℓt\\ell\_\{t\}for all

ttvia top\-

KKKL \{Eq\. \([2](https://arxiv.org/html/2607.27081#S3.E2)\)\}

8:endfor

9:

θ←θ−η​∇θℒ​\(θ\)\\theta\\leftarrow\\theta\-\\eta\\,\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\)\{Eq\. \([3](https://arxiv.org/html/2607.27081#S3.E3)\)\}

10:endfor

11:endfor

12:return

πθ\\pi\_\{\\theta\}

### 3\.3Why ROPD Reduces Template Dependence

The refusal behavior of the aligned modelM0M\_\{0\}is largely a property of its output distribution rather than of any particular prompt template: presented with a harmful request,M0M\_\{0\}concentrates probability on refusals whether the request arrives in a minimal template or in its native chat format\. By distilling this distribution onDharmD\_\{\\text\{harm\}\}, ROPD injects a refusal prior into the student that is much less tied to the template the attacker used\. This contrasts with weight\-arithmetic and partial\-rollback defenses, whose safety signal is estimated from, and therefore tightly coupled to, the template in which harmful behavior was elicited; when the defense template diverges from the attack template, that signal is misdirected\. ROPD weakens this coupling but does not sever it entirely: because the student is realigned and evaluated under a chosen deployment template, a large gap between that template and the attacker’s can still leave residual risk \(Section[4\.4](https://arxiv.org/html/2607.27081#S4.SS4)\)\. Simultaneously, distillingπtask\\pi\_\{\\text\{task\}\}onDtaskD\_\{\\text\{task\}\}anchors the student to the skill distribution it already possesses, which prevents the task collapse that afflicts methods that suppress harmful behavior by broadly perturbing the weights\. The two teachers divide labor cleanly—safety fromM0M\_\{0\}, task fromMaM\_\{a\}—which, as our ablation confirms, is what allows ROPD to lower ASR without paying in downstream accuracy\.

## 4Experiments

### 4\.1Experimental Setup

#### Models\.

We study three instruction\-tuned models spanning different alignment strengths: Llama\-2\-7B\-Chat\(Touvron et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib37)\), Qwen2\.5\-7B\-Instruct\(Team,[2024](https://arxiv.org/html/2607.27081#bib.bib38)\), and Gemma\-2\-9B\-it\(Gemma Team,[2024](https://arxiv.org/html/2607.27081#bib.bib39)\)\.

#### Downstream tasks and data\.

We consider three specialized skills as downstream tasks: text\-to\-SQL generation \(SQL\)\(Gao et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib40)\), dialogue summarization \(SAMSum\)\(Gliwa et al\.,[2019](https://arxiv.org/html/2607.27081#bib.bib41)\), and natural\-language\-to\-shell synthesis \(NL2Bash\)\(Lin et al\.,[2018](https://arxiv.org/html/2607.27081#bib.bib42)\)\. The attacker fine\-tunes each model on a mixture of the task corpus and1,5001\{,\}500harmful examples drawn from BeaverTails\(Ji et al\.,[2023](https://arxiv.org/html/2607.27081#bib.bib43)\), using a44\-bit LoRA adapter\(Hu et al\.,[2022](https://arxiv.org/html/2607.27081#bib.bib11)\); the harmful evaluation set consists of700700held\-out BeaverTails prompts\. Task and harmful data are stored as raw pairs and rendered under theraw,self, orattacktemplate at load time, so that the attack template, defense template, and evaluation template are all controlled independently\.

#### Defenses\.

We compare ROPD against four SOTA realignment baselines: rollback\(Yang et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib17)\), RESTA\(Bhardwaj et al\.,[2024](https://arxiv.org/html/2607.27081#bib.bib19)\), soft\-SFT\(Qi et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib20)\), and the representation\-space defense SSRD\(Gong et al\.,[2025](https://arxiv.org/html/2607.27081#bib.bib21)\)\. The experimental setup for each method remains the same as that described in their respective papers\. ROPD uses the dual\-teacher configuration with the misaligned model as task teacher and the original model as safety teacher\. Because the defender does not know the attack template, every defense is run with both legal defense templatesTd∈\{raw,self\}T\_\{d\}\\in\\\{\\textsc\{raw\},\\textsc\{self\}\\\}; theattacktemplate is used as a defense only as an oracle upper bound when the attack is itself underattack\. All experiments are run on NVIDIA H20D GPUs; the detailed per\-defense configurations and hyperparameters are provided in the supplementary material\.

#### Metrics\.

We report two quantities per cell as*task / ASR*\. The task score is the standard metric of each downstream task \(exact\-match for SQL; ROUGE\-based overlap for SAMSum; command accuracy for NL2Bash\), computed without any LLM judge\. ASR is the attack success rate—the percentage of the700700harmful prompts answered harmfully—as judged by Qwen2\.5\-32B\-Instruct\(Team,[2024](https://arxiv.org/html/2607.27081#bib.bib38)\)\. Every model is evaluated under one or more of theraw,self, andattacktemplates; the true ASR of an attack is read in the channel that matches the attack template\.

### 4\.2Skill Retention

We first establish, in the deployment\-acceptance setting where each defended model is evaluated under the same template it was repaired with \(Teval=TdT\_\{\\text\{eval\}\}=T\_\{d\}\), how the defenses trade off task retention against harm reduction as the attack and defense templates vary\. Table[1](https://arxiv.org/html/2607.27081#S4.T1)reports all three base models on SQL across all three attack templates and every legal defense template\. For each attack, the defender may chooseTd∈\{raw,self\}T\_\{d\}\\in\\\{\\textsc\{raw\},\\textsc\{self\}\\\}without knowing the attacker’s template; when the attack itself uses theattacktemplate we additionally report the oracleTd=attackT\_\{d\}=\\textsc\{attack\}\(the last column\), which is unavailable to a real defender and serves only as an upper bound\.

Table 1:Skill retention and harm reduction on Llama\-2\-7B\-Chat, Qwen2\.5\-7B\-Instruct, and Gemma\-2\-9B\-it with a SQL downstream task\. Each cell is task exact\-match / ASR \(%\)\. In each column, the highest task score and the lowest ASR among the defense methods are shown in bold\. In some cells, the baseline achieved lower ASR, but it significantly sacrificed downstream performance\.Across the three models, the fine\-tuning\-based baselines that suppress harm do so by damaging the task \(Vulnerability 1\)\. On Llama\-2, soft\-SFT drives SQL exact\-match to near zero in therawchannel under theself\- andattack\-template attacks \(0\.0000\.000and0\.2450\.245\), while SSRD and RESTA collapse the task under a mismatched defense template \(SSRD to0\.0990\.099under attack==attack/def==raw; RESTA to as low as0\.0360\.036when the attack and defense templates disagree\)\. The same trade\-off recurs on Qwen2\.5, where RESTA and soft\-SFT lose most of the task in the mismatched channel \(RESTA to0\.0600\.060, soft\-SFT to0\.440\.44–0\.460\.46\), and on Gemma\-2, where rollback collapses the task to0\.0170\.017under the mismatched def==selftemplate\. ROPD, by anchoring the student to the task teacher, retains a task score at or near the pre\-attack level under itsselfdefense on every model \(0\.5960\.596–0\.6260\.626on Llama\-2,0\.6960\.696–0\.7160\.716on Qwen2\.5,0\.6700\.670–0\.6890\.689on Gemma\-2\), the highest among the methods that also achieve a low ASR in that column\.

### 4\.3Template Dependence

The second pattern concerns template dependence \(Vulnerability 2\): the baselines swing violently with the defense template, whereas ROPD swings far less\. On Llama\-2, SSRD’s ASR under attack==selfmoves from3\.13\.1\(matched def==self\) to46\.346\.3\(mismatched def==raw\); rollback stays high across templates—37\.737\.7at attack==def==rawand rebounding to84\.084\.0–87\.087\.0against theattackattack; RESTA’s ASR ranges from2\.32\.3to66\.966\.9depending on template alignment\. The pattern holds on Qwen2\.5 \(RESTA6\.46\.4to66\.466\.4; SSRD7\.47\.4to48\.148\.1\) and Gemma\-2 \(soft\-SFT2828–6565regardless of template\)\. Across all three models, ROPD attains a low ASR under theselfdefense template regardless of which template the attacker used—2\.12\.1–2\.42\.4on Llama\-2,6\.66\.6–11\.111\.1on Qwen2\.5, and5\.15\.1–5\.65\.6on Gemma\-2—so a defender who simply adoptsselfobtains a strong defense*without*needing to guess the attack template\. ROPD is not wholly immune to template mismatch: when its own deployment template is instead set torawwhile the attack usedself, its ASR also rises \(e\.g\. to28\.328\.3on Llama\-2 and22\.622\.6on Qwen2\.5\), so it too benefits from a well\-chosen defense template—but the swing is much smaller than the baselines’, and it never comes at the cost of collapsing the task\. Together these observations substantiate that current defenses implicitly assume template consistency, and that violating this assumption either leaves the attack channel open or destroys the downstream skill\. In Gemma, the same template dependence holds: rollback and RESTA reach a low ASR only in their matchedselfchannel \(rollback0\.017/0\.10\.017/0\.1\) while losing their downstream task ability and leaving the mismatchedrawchannel wide open—under attack==self/def==rawthe ASR climbs to41\.741\.7\(rollback\) and45\.745\.7\(RESTA\), and stays above6060for soft\-SFT; the only baseline that suppresses harm in therawchannel there \(SSRD, ASR6\.16\.1\) does so by collapsing the SQL task to0\.1210\.121\.

![Refer to caption](https://arxiv.org/html/2607.27081v1/figures/defense_grid.png)Figure 3:Defense comparison on SAMSum and NL2Bash across the three base models \(rows\) and attack templates\. Bars give each defense’s ASR \(left axis\) and task score \(right axis\); solid==defraw, hatched==defself; horizontal black lines and red lines mark the unattacked base and misaligned model, respectively\.![Refer to caption](https://arxiv.org/html/2607.27081v1/figures/pareto_scatter.png)Figure 4:Safety–task trade\-off across three representative settings\.The same two patterns generalize beyond SQL\. Fig\.[3](https://arxiv.org/html/2607.27081#S4.F3)summarizes the SAMSum and NL2Bash results across all three base models: fine\-tuning under any attack template lifts ASR to∼60%\{\\sim\}60\\%, and ROPD under its template\-robustselfdefense \(hatched bars\) returns it to single\-digit ASR without collapsing the task, whereas the baselines either leave the mismatchedrawchannel open \(SSRD, RESTA, rollback ASR4040–7070\) or lose the skill \(soft\-SFT\)\. The conclusions carry over from SQL: under theselfdefense ROPD drives ASR down to single\-digit or low\-teens levels on every model while retaining—or improving—the task score \(e\.g\., on Llama\-2 SAMSum, ROUGE0\.431→0\.4300\.431\\\!\\to\\\!0\.430at ASR61\.1→1\.661\.1\\\!\\to\\\!1\.6in theselfchannel\)\. ROPD is not always the single lowest\-ASR method—in a few cells a baseline reaches a lower ASR, but only by sacrificing most of the downstream skill—so its advantage is best read as the safety–task trade\-off: Fig\.[4](https://arxiv.org/html/2607.27081#S4.F4)plots ASR against task score for one representative \(model, task\) setting each, and ROPD’s matchedselfoperating point lies on the safety–task Pareto frontier in every panel—no baseline attains both a lower ASR and a higher task score\.

### 4\.4The Robustness Boundary: Switching the System Prompt

The matched\-evaluation view in Table[1](https://arxiv.org/html/2607.27081#S4.T1)reflects how a defender would accept a repaired model: repair under templateTdT\_\{d\}and validate under the sameTdT\_\{d\}\. Vulnerability 3 asks what happens when a malicious user, faced with an already\-realigned model, simply changes the system prompt\. Table[2](https://arxiv.org/html/2607.27081#S4.T2)takes the attack==selfsetting, fixes every defense to its ownselftemplate \(so that each has “passed” acceptance in its own channel\), and then re\-evaluates the same repaired models under all three templates\.

Table 2:Cross\-template evaluation \(Llama\-2\-7B\-Chat\+\+SQL, attack==self\)\. Cells are task / ASR \(%\)\.Under acceptance \(eval==self\) every method looks repaired, with ASR between2\.42\.4and30\.930\.9\. But when the attacker rewrites the system prompt toraw, ASR rebounds for all of them—soft\-SFT to36\.436\.4, rollback to42\.342\.3, RESTA to24\.124\.1, SSRD to22\.322\.3, ROPD to19\.719\.7\. This confirms that current realignment is a prompt\-conditional fix rather than a durable weight\-level one, and that single\-template acceptance testing overstates safety\. ROPD’s residual cross\-template ASR is comparable to or lower than the baselines’ while its task retention across templates is markedly higher \(for example,0\.5980\.598versus soft\-SFT’s0\.0010\.001and rollback’s0\.2970\.297in therawchannel\), but the phenomenon itself is universal and marks an intrinsic robustness boundary of realignment: a realistic threat model must assume the attacker can freely alter the prompt\.

### 4\.5Ablation: The Role of Each Teacher

Finally we isolate the contribution of each teacher on Llama\-2\-7B\-Chat \+ SQL with attack==raw, defense templateraw, and the shared22\-epoch,2×10−52\\times 10^\{\-5\}, top\-5050KL recipe\. Table[3](https://arxiv.org/html/2607.27081#S4.T3)compares the dual\-teacher method against two variants: replacing the \(attacked\) task teacher with a clean SQL teacher trained on task data alone, and removing the task teacher entirely so that only the safety teacher and harmful data remain\.

Table 3:Teacher ablation \(Llama\-2\-7B\-Chat\+\+SQL, attack==def==raw\)\. Cells are task exact\-match / ASR \(%\)\. \(A\) is the default ROPD; \(B\) swaps in a clean task teacher; \(C\) drops the task teacher\.The ablation cleanly separates the two objectives\. Comparing \(A\) with \(C\), removing the task teacher leaves ASR essentially unchanged \(20\.1→23\.020\.1\\\!\\to\\\!23\.0inraw\) but drives the task below the pre\-attack level \(0\.607→0\.5630\.607\\\!\\to\\\!0\.563\), confirming that the safety teacher alone governs harm reduction while the task teacher is necessary to retain the skill\. Comparing \(A\) with \(B\), swapping in a clean SQL teacher that was never exposed to the attack yields the strongest task scores \(0\.6920\.692/0\.6670\.667, above even the pre\-attack model\) at comparable ASR, because the task teacher determines only task quality and an uncontaminated teacher transfers a cleaner skill\. The cost is an additional teacher\-training run\. Weighing quality against cost, we adopt configuration \(A\)—the misaligned model as task teacher and the original model as safety teacher—as the default, since it sacrifices only0\.060\.06of task score while avoiding a second training stage\.

### 4\.6Data Efficiency, Training Dynamics, and Cost

Unless noted, experiments in this subsection use Llama\-2\-7B\-Chat \+ SQL\.

#### Data efficiency\.

Table[4](https://arxiv.org/html/2607.27081#S4.T4)scales the realignment mixture \(task\+\+BeaverTails together;100%=6,500100\\%=6\{,\}500examples\) under attack==self, reading ASR in the true channel \(eval==self\)\. Under the matched defense \(Td=selfT\_\{d\}=\\textsc\{self\}\), ROPD is strikingly data\-efficient: only25%25\\%\(1,5001\{,\}500examples\) cuts ASR from62\.162\.1to2\.42\.4while holding the task at0\.6260\.626, and more data mainly improves the task further \(up to0\.6340\.634\)\. The smallest budget \(512512\) sits just below the data\-sufficiency knee \(ASR17\.617\.6\), so the largest safety gain comes from the512→1,500512\\\!\\to\\\!1\{,\}500step\. The mismatched defense \(Td=rawT\_\{d\}=\\textsc\{raw\}\), by contrast, cannot be rescued by data—its true\-channel ASR falls only slowly \(41\.6→26\.041\.6\\\!\\to\\\!26\.0\) and never reaches the low band—confirming that template mismatch is a template\-level problem, not a data\-budget one, and that its residual risk is not something more data can close\.

Table 4:Data efficiency of ROPD \(Llama\-2\-7B\-Chat\+\+SQL, attack==self, true channel eval==self\)\. Cells are task exact\-match / ASR \(%\); bold is the matched \(self\) defense\.
#### Training dynamics\.

Fig\.[5](https://arxiv.org/html/2607.27081#S4.F5)plots training\-loss curves \(log scale, EMA\-smoothed\) for the gradient\-based defenses on SAMSum under attack==attack\. ROPD’s top\-KKKL loss stays one\-to\-two orders of magnitude below the baselines throughout—not under\-training but a gentle, targeted edit: initialized from the task teacher, the student’s loss on task tokens starts near zero and rises only as the safety teacher pulls harmful\-token predictions toward refusal\. The baselines \(SSRD, soft\-SFT, RESTA\) must minimize much larger losses, rewriting the weights more aggressively, consistent with their greater collateral damage to the task in Tables[1](https://arxiv.org/html/2607.27081#S4.T1)\.\(Rollback has no comparable training loss and is omitted; loss definitions differ across methods, so the log axis conveys trend, not comparable values\.\)

![Refer to caption](https://arxiv.org/html/2607.27081v1/figures/loss_curves_samsum_attack.png)Figure 5:Training\-loss curves of the gradient\-based defenses\.
#### Cost\.

Table[5](https://arxiv.org/html/2607.27081#S4.T5)reports the wall\-clock time to produce each defended model \(Gemma\-2\-9B\-it \+ NL2Bash, attack==def==self\)\. ROPD trains in15\.715\.7min—comparable to the SFT\-based baselines \(soft\-SFT13\.113\.1, RESTA12\.112\.1min\) and far below rollback’s117\.9117\.9min, whose sparse\-parameter search is CPU\-bound\. SSRD is cheapest \(4\.54\.5min\), but the baselines trade this economy for template\-fragile, skill\-damaging repairs\. ROPD thus spends a modest budget to buy cross\-template stability without task loss; a per\-step breakdown is in the supplementary material\.

Table 5:Overhead of producing the defended model\.

## 5Conclusion

We studied the safety realignment of fine\-tuning misaligned language models and showed that mainstream defenses share three vulnerabilities: they degrade the specialized skill they should preserve, they depend on knowing the attacker’s prompt template, and they can be re\-jailbroken by switching the system prompt\. Framing the durable signature of an attack as a difference between the aligned and compromised output distributions, we proposed ROPD, a dual\-teacher, source\-routed top\-KKKL distillation framework that inherits a largely template\-independent refusal prior from the original model while preserving the downstream skill from the fine\-tuned model\. It substantially reduces, but does not fully eliminate, the template\-mismatch risk\. Our cross\-template analysis further shows that all realignment methods, ROPD included, remain conditionally vulnerable to prompt rewriting, delineating a robustness boundary for weight\-level repair and motivating future work on realignment durable against an adversary who controls the prompt\.

## References

- Wang et al\. \[2024\]Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang\-Bo Mao, Sitaram Asur, et al\.A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more\.*arXiv preprint arXiv:2407\.16216*, 2024\.
- Ouyang et al\. \[2022\]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al\.Training language models to follow instructions with human feedback\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 35, pages 27730–27744, 2022\.
- Bai et al\. \[2022\]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.*arXiv preprint arXiv:2204\.05862*, 2022\.
- Ma et al\. \[2026\]Wanlun Ma, Yongjian Guo, Qing\-Long Han, Wei Zhou, Xiaogang Zhu, Junwu Xiong, Sheng Wen, and Yang Xiang\.Understanding agentic ai: Algorithms and infrastructure\.*IEEE/CAA Journal of Automatica Sinica*, 13\(4\):776–795, 2026\.doi:10\.1109/JAS\.2026\.125993\.
- Chung et al\. \[2024\]Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al\.Scaling instruction\-finetuned language models\.*Journal of Machine Learning Research*, 25\(70\):1–53, 2024\.
- Qi et al\. \[2024\]Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin\-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson\.Fine\-tuning aligned language models compromises safety, even when users do not intend to\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Wei et al\. \[2023\]Alexander Wei, Nika Haghtalab, and Jacob Steinhardt\.Jailbroken: How does LLM safety training fail?In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Deng et al\. \[2025\]Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang\.Ai agents under threat: A survey of key security challenges and future pathways\.*ACM Comput\. Surv\.*, 57\(7\), February 2025\.ISSN 0360\-0300\.doi:10\.1145/3716628\.URL[https://doi\.org/10\.1145/3716628](https://doi.org/10.1145/3716628)\.
- Chae and Davidson \[2025\]Youngjin Chae and Thomas Davidson\.Large language models for text classification: From zero\-shot learning to instruction\-tuning\.*Sociological Methods & Research*, page 00491241251325243, 2025\.
- Bhardwaj and Poria \[2023\]Rishabh Bhardwaj and Soujanya Poria\.Language model unalignment: Parametric red\-teaming to expose hidden harms and biases\.*arXiv preprint arXiv:2310\.14303*, 2023\.
- Hu et al\. \[2022\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Lin et al\. \[2023\]Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, and Kush R Varshney\.Towards healthy ai: large language models need therapists too\.*arXiv preprint arXiv:2304\.00416*, 2023\.
- Phelps and Ranson \[2023\]Steve Phelps and Rebecca Ranson\.Of models and tin men–a behavioural economics study of principal\-agent problems in ai alignment using large\-language models\.*arXiv preprint arXiv:2307\.11137*, 2023\.
- Betley et al\. \[2025\]Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber\-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans\.Emergent misalignment: Narrow finetuning can produce broadly misaligned llms\.In*ICLR 2025 Workshop on Foundation Models in the Wild*, 2025\.
- Hu et al\. \[2026\]Xuhao Hu, Wang Peng, Xiaoya Lu, Dongrui Liu, Xuan\-Jing Huang, and Jing Shao\.Llms deceive unintentionally: Emergent misalignment in dishonesty from misaligned samples to biased human\-ai interactions\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 9348–9372, 2026\.
- Qu et al\. \[2025\]Yubin Qu, Song Huang, Long Li, Peng Nie, and Yongming Yao\.Beyond intentions: A critical survey of misalignment in llms\.*Computers, Materials & Continua*, 85\(1\):249–300, 2025\.
- Yang et al\. \[2025\]Kang Yang, Guanhong Tao, Xun Chen, and Jun Xu\.Alleviating the Fear of Losing Alignment in LLM Fine\-tuning \.In*2025 IEEE Symposium on Security and Privacy \(SP\)*, pages 2152–2170, Los Alamitos, CA, USA, May 2025\. IEEE Computer Society\.doi:10\.1109/SP61157\.2025\.00171\.URL[https://doi\.ieeecomputersociety\.org/10\.1109/SP61157\.2025\.00171](https://doi.ieeecomputersociety.org/10.1109/SP61157.2025.00171)\.
- Lin et al\. \[2024\]Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi\.The unlocking spell on base llms: Rethinking alignment via in\-context learning\.In*International Conference on Learning Representations*, volume 2024, pages 24907–24933, 2024\.
- Bhardwaj et al\. \[2024\]Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria\.Language models are homer simpson\! safety re\-alignment of fine\-tuned language models through task arithmetic\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.
- Qi et al\. \[2025\]Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson\.Safety alignment should be made more than just a few tokens deep\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Gong et al\. \[2025\]Yichen Gong, Delong Ran, Xinlei He, Tianshuo Cong, Anyu Wang, and Xiaoyun Wang\.Safety misalignment against large language models\.*Network and Distributed System Security Symposium \(NDSS\)*, 2025\.
- Guo et al\. \[2026\]Yongjian Guo, Puzhuo Liu, Wanlun Ma, Zehang Deng, Xiaogang Zhu, Peng Di, Xi Xiao, and Sheng Wen\.Mcpxkit: the unified toolkit for analyzing model context protocol security\.*IEEE Transactions on Dependable and Secure Computing*, pages 1–16, 2026\.doi:10\.1109/TDSC\.2026\.3695553\.
- Lyu et al\. \[2024\]Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora\.Keeping llms aligned after fine\-tuning: The crucial role of prompt templates\.*Advances in Neural Information Processing Systems*, 37:118603–118631, 2024\.
- Yang et al\. \[2026\]Zhicheng Yang, Zhijiang Guo, Yifan Song, Minrui Xu, Yongxin Wang, Yiwei Wang, Xiaodan Liang, and Jing Tang\.Prune\-opd: Efficient and reliable on\-policy distillation for long\-horizon reasoning\.*arXiv preprint arXiv:2605\.07804*, 2026\.
- Song and Zheng \[2026\]Mingyang Song and Mao Zheng\.A survey of on\-policy distillation for large language models\.*arXiv preprint arXiv:2604\.00626*, 2026\.
- Wei et al\. \[2022\]Jason Wei, Maarten Bosma, Vincent Y\. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M\. Dai, and Quoc V\. Le\.Finetuned language models are zero\-shot learners\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Guo et al\. \[2025\]Yongjian Guo, Wanlun Ma, Xi Xiao, Sheng Wen, Peng Di, and Xiaogang Zhu\.Patch the leak: Strengthening codellms against privacy extraction threats\.In*Proceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis*, ISSTA Companion ’25, page 195–199, New York, NY, USA, 2025\. Association for Computing Machinery\.ISBN 9798400714740\.doi:10\.1145/3713081\.3732931\.URL[https://doi\.org/10\.1145/3713081\.3732931](https://doi.org/10.1145/3713081.3732931)\.
- Zou et al\. \[2023\]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J\. Zico Kolter, and Matt Fredrikson\.Universal and transferable adversarial attacks on aligned language models\.In*arXiv preprint arXiv:2307\.15043*, 2023\.
- Huang et al\. \[2024\]Tiansheng Huang, Sihao Hu, and Ling Liu\.Vaccine: Perturbation\-aware alignment for large language model\.The Thirty\-Eighth Annual Conference on Neural Information Processing Systems …, 2024\.
- Ilharco et al\. \[2023\]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi\.Editing models with task arithmetic\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Hinton et al\. \[2015\]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Wang et al\. \[2026\]Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan\-Zhe Guo\.Trace: Distilling where it matters via token\-routed self on\-policy alignment\.*arXiv preprint arXiv:2605\.10194*, 2026\.
- Zhang et al\. \[2026\]Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, and Xiuyin Zhao\.Shortopd: Recovering pruned llms with short\-to\-long on\-policy distillation\.*arXiv preprint arXiv:2607\.13124*, 2026\.
- Li et al\. \[2026\]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan\-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding\.Rethinking on\-policy distillation of large language models: Phenomenology, mechanism, and recipe\.*arXiv preprint arXiv:2504\.09526*, 2026\.
- Agarwal et al\. \[2024\]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*International Conference on Learning Representations*, volume 2024, pages 21246–21263, 2024\.
- Fu et al\. \[2026\]Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao\.Revisiting on\-policy distillation: Empirical failure modes and simple fixes\.*arXiv preprint arXiv:2603\.25562*, 2026\.
- Touvron et al\. \[2023\]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al\.Llama 2: Open foundation and fine\-tuned chat models\.*arXiv preprint arXiv:2307\.09288*, 2023\.
- Team \[2024\]Qwen Team\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*, 2024\.
- Gemma Team \[2024\]Gemma Team\.Gemma 2: Improving open language models at a practical size\.*arXiv preprint arXiv:2408\.00118*, 2024\.
- Gao et al\. \[2024\]Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou\.Text\-to\-sql empowered by large language models: A benchmark evaluation\.*Proceedings of the VLDB Endowment*, 17\(5\):1132–1145, 2024\.
- Gliwa et al\. \[2019\]Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer\.SAMSum corpus: A human\-annotated dialogue dataset for abstractive summarization\.In*Proceedings of the 2nd Workshop on New Frontiers in Summarization*, 2019\.
- Lin et al\. \[2018\]Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D\. Ernst\.NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system\.In*Proceedings of the Eleventh International Conference on Language Resources and Evaluation \(LREC\)*, 2018\.
- Ji et al\. \[2023\]Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang\.BeaverTails: Towards improved safety alignment of LLM via a human\-preference dataset\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.