DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

# DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance
Source: [https://arxiv.org/html/2608.14644](https://arxiv.org/html/2608.14644)
###### Abstract

Real\-world LLM deployments increasingly rely on runtime\-injected prohibitions—enterprise policies, PII redlines, tool boundaries—that vary per request and per tenant\. Conventional post\-training is structurally ill\-suited: SFT hides the violation signal in compliant labels, and DPO’s sequence\-level preferences mismatch token\-localized violations\. We propose DUET, a token\-selective on\-policy distillation method for prohibition compliance\. DUET pairs a teacher that sees the prohibition \(positive\) with an identical\-weight teacher that does not \(negative\)\. Because the two teachers differ only in prohibition visibility, their per\-token disagreement isolates the prohibition’s causal effect—yielding a clean supervision signal uncontaminated by model capacity or mismatch\. This disagreement drives two complementary mechanisms: signal cleaning, which discards agreement tokens as redundant or prefix\-corrupted, and preference\-directed learning, which pushes the student away from the negative teacher and toward the positive one at token granularity, embedding DPO\-style optimization directly into OPD without offline preference data\. We construct an industrial Prohibition\-Compliance benchmark spanning five task families covering explicit\-refusal, paraphrase robustness, and over\-refusal\. Across 1\.5B–8B Qwen variants, DUET achieves 72\.3–85\.2% violation compliance while preserving 88–93% normal utility, dramatically outperforming teacher model and other distillation baselines\. External evaluation on SysBench confirms improved safety alignment with minimal degradation on GSM8K and MATH\-500\.

## 1Introduction

Real\-world LLM deployments are increasingly governed by runtime\-injected prohibitions embedded in the system prompt, such as enterprise SOPs, PII redlines, tool\-usage boundaries, and brand\-safety rules\(Weiet al\.[2023](https://arxiv.org/html/2608.14644#bib.bib4); Ruanet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib5)\), which vary across requests and tenants\. Violations incur tangible industrial costs \(leaked user data, out\-of\-policy tool invocations, brand\-safety incidents\), yet thesamemodel weights must still faithfully answer the underlying query when the prohibition is altered or absent\. We formalize this asProhibition Compliance, characterized by three structural features that set it apart from conventional alignment: \(i\)runtime configurability, as the prohibition resides in the prompt rather than the weights; \(ii\)localized violation, since only a small subset of tokens in a long response actually crosses the line; and \(iii\) ajoint compliance\-utility requirement, where over\-refusing legitimate queries in the same business context is itself a failure mode\.

![Refer to caption](https://arxiv.org/html/2608.14644v1/x1.png)Figure 1:Comparison of four post\-training paradigms on Prohibition Compliance\.As illustrated in Figure[1](https://arxiv.org/html/2608.14644#S1.F1), existing post\-training paradigms are structurally mismatched to this setting\. SFT\(Weiet al\.[2022](https://arxiv.org/html/2608.14644#bib.bib6)\)and RLHF\(Christianoet al\.[2017](https://arxiv.org/html/2608.14644#bib.bib8); Stiennonet al\.[2020](https://arxiv.org/html/2608.14644#bib.bib9); Ouyanget al\.[2022](https://arxiv.org/html/2608.14644#bib.bib7)\)rely on compliant responses as labels, leaving the violation signal without an explicit channel into the loss; yet the crux of the task is precisely which violation must be suppressed at which position, and SFT cannot admit the violating token itself as a negative supervisory signal\. Offline DPO\(Rafailovet al\.[2023](https://arxiv.org/html/2608.14644#bib.bib10)\), in turn, uses sequence\-level chosen/rejected labels misaligned with token\-localized violations: condemning an entire sequence for a handful of offending tokens contaminates neutral positions while diluting the correction at the truly violating ones\. This limitation is further exacerbated by the substantial industrial cost of curating sequence\-level preference data\.

On\-policy distillation \(OPD\)\(Guet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib13); Agarwalet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib12)\), which corrects along the student’s actual failure trajectories under direct teacher supervision, is the natural framework for Prohibition Compliance, yet applying it in vanilla form exposes twostructural gaps\. First, since the teacher’s next\-token logits are conditioned on the student’s own prefix, any drift toward a violation drags the teacher’s forward along with it, degrading the positive teacher” into an unreliable supervisor at exactly the positions where correction matters most, yet no online mechanism detects this drift\. Second, mainstream OPD applies a per\-sequence token\-mean loss over all tokens, wasting supervision on content\-neutral positions and diluting it on the few tokens that actually determine compliance; existing token\-selective remedies pick hard tokens via teacher–student divergence, but that signal conflates genuine prohibition\-violation signals with intrinsic teacher–student distributional mismatch, injecting noise and inducing training oscillation\.

Building on this, we proposeDUET\(DUal\-tEacherToken\-selective OPD\), which introduces a pair of same\-weight teachers differing only in prohibition visibility, whereT\+T^\{\+\}sees the prohibition whileT−T^\{\-\}does not, and exploits their per\-token disagreement in two complementary ways\. First,violation\-token selection: to clean the training signal, we discard tokens whereT\+T^\{\+\}andT−T^\{\-\}agree, since they carry no prohibition\-relevant information\. Such positions either provide redundant supervision on ordinary tokens or, more problematically, correspond to cases whereT\+T^\{\+\}has been pulled towardT−T^\{\-\}by the student’s prefix, yielding misleading supervision\. Second,preference\-directed learning: on the surviving tokens, the disagreement vector defines a per\-token push/pull direction that pushes the student away fromT−T^\{\-\}and pulls it towardT\+T^\{\+\}, embedding DPO\-style preference optimization into on\-policy distillation at token granularity without offline data, and driven by a prohibition\-conditioned rather than capability\-gap\-driven signal\.

Our contributions are summarized as follows:

- •We proposeDUET, a dual\-teacher token\-selective OPD framework where a positive teacherT\+T^\{\+\}and a same\-weight negative teacherT−T^\{\-\}differ only in prohibition visibility\. Their per\-token disagreement drivesviolation\-token selection, which discards agreement positions as redundant or prefix\-corrupted, andpreference\-directed learning, which pushes the student away fromT−T^\{\-\}and towardT\+T^\{\+\}, embedding token\-granular DPO\-style optimization into OPD without offline preference data\.
- •We construct an industrial Prohibition\-Compliance benchmark over five task families \(PII/memory, safety redlines, custom business content, tool definitions, business SOP\) that jointly measures explicit\-violation refusal, paraphrase robustness, and over\-refusal on legitimate queries within the same business context\.
- •We systematically evaluate DUET on our Prohibition\-Compliance benchmark and a public constraint\-following benchmark, and consistently verify across two model families \(Qwen2\.5 and Qwen3\) at three scales from 1\.5B to 8B that DUET significantly outperforms existing distillation baselines on both explicit\-violation refusal and legitimate\-query helpfulness, with no catastrophic forgetting on general\-capability benchmarks\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2608.14644v1/x2.png)Figure 2:Overview of DUET\. Two same\-weight teachers differing only in prohibition visibility \(T\+T^\{\+\}vs\.T−T^\{\-\}\) forward on the student’s rollout; their per\-token disagreement drives \(1\) Violation\-Token Selection, keeping only high\-disagreement tokens and \(2\) Preference\-Directed Learning, which pushes the student away fromT−T^\{\-\}and towardT\+T^\{\+\}at token granularity\.### 2\.1On\-Policy Knowledge Distillation for LLMs

Sequence\-level distillation for language models has shifted from teacher\-sampled trajectories to on\-policy distillation, where the student generates its own training sequences, and existing work develops along several roughly orthogonal directions\. On sampling policy and divergence form, GKD\(Agarwalet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib12)\)trains on student rollouts under teacher feedback with divergences beyond forward KL, MiniLLM\(Guet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib13)\)uses reverse KL to avoid over\-weighting the teacher’s low\-probability tail, and DistiLLM\(Koet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib14)\)interpolates the two with an adaptive on\-policy fraction\. Other work replaces uniform\-target supervision with contrastive or preference\-aware objectives: DistiLLM\-2\(Koet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib15)\)casts teacher and student responses as preference pairs, while AlignDistil\(Zhanget al\.[2025](https://arxiv.org/html/2608.14644#bib.bib16)\)synthesises a token\-adaptive target from two oppositely trained preference models\. A token\-selective line restricts supervision to informative positions, as in TIP\(Xuet al\.[2026](https://arxiv.org/html/2608.14644#bib.bib18)\)\(high\-entropy tokens\) and SCOPE\(Zhenget al\.[2026](https://arxiv.org/html/2608.14644#bib.bib19)\)\(correctness\-routed rollouts with dual\-path adaptive weighting\)\. Finally, RCSD\(Guet al\.[2026](https://arxiv.org/html/2608.14644#bib.bib17)\)uses conditional teachers, querying the same base model with and without a scoring rubric and distilling across the two conditions\.

Existing on\-policy distillation neither dynamically monitors the potentially erroneous feedback signal from the teacher, nor manages to isolate task\-relevant error\-prone positions from the intrinsic teacher\-student distribution mismatch\. We instead take the disagreement between a pair of weight\-shared positive and negative teachers—differing only in their task\-relevant context—as the screening criterion: it both filters out redundant or erroneous teacher feedback and localises the error\-prone token positions, at which the student’s distribution is then pushed away from the negative teacher and pulled toward the positive one—an approach absent from prior work\.

### 2\.2Constraint\-Following Benchmarks

Existing Constraint\-Following benchmarks span four largely disjoint areas\. IFEval\(Zhouet al\.[2023](https://arxiv.org/html/2608.14644#bib.bib20)\), FollowBench\(Jianget al\.[2024](https://arxiv.org/html/2608.14644#bib.bib21)\)and FoFo\(Xiaet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib22)\)target verifiable soft instructions and focus on structural or stylistic requests, with few outright prohibitions\. Safety\-oriented benchmarks such as SORRY\-Bench\(Xieet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib24)\)and CoCoNot\(Brahmanet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib23)\)concentrate on refusal behaviour under harmful or under\-specified queries, without embedded benign counterparts under the same business context\. System\-prompt\-following benchmarks such as SysBench\(Qinet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib25)\)and SpecEval\(Ahmedet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib27)\)are closest to our setting, but they either omit domain\-specific confidentiality prohibitions common in industrial deployments or audit provider\-level rather than user\-level dynamic rules\. MT\-Bench\-101\(Baiet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib26)\)evaluates fine\-grained multi\-turn capability and serves only as a general\-utility reference\.

Unlike these benchmarks, the benchmark constructed in this work covers a broad range of industrial prohibition compliance scenarios and, within each scenario, jointly measures explicit\-violation refusal, paraphrase robustness, and over\-refusal risk on legitimate queries under the same business context, filling the gap left by prior benchmarks that do not jointly evaluate prohibition compliance and utility preservation on industrial hard\-constraint settings\.

## 3Preliminaries

Sequence\-level distillation for language models has moved from teacher\-sampled trajectories toward on\-policy distillation \(OPD\): the student itself produces the rollouts and the teacher supervises token\-wise on those rollouts\. The generic OPD form reads

ℒOPD=𝔼y∼πS\(⋅∣x\)\[∑tD\(πS\(⋅∣x,y<t\)∥πT\(⋅∣x,y<t\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\\;=\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\}\\Big\[\\sum\_\{t\}D\\\!\\big\(\\pi\_\{S\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\big\\\|\\,\\pi\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\)\\big\)\\Big\],

\(1\)whereDDmay be the forward KL, the reverse KL \(MiniLLM\(Guet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib13)\)\), or an interpolation of the two \(GKD\(Agarwalet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib12)\), DistiLLM\(Koet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib14)\)\)\.

#### Models and prompts\.

We denote the student bySSand the positive/negative teachers byT\+T^\{\+\}/T−T^\{\-\}, whereT\+T^\{\+\}andT−T^\{\-\}share identical base weights and differ only in whether the prohibition appears in their system prompt, as specified below:

- •T\+ prompt\.system\_role\+<context\>\+<prohibition\>\+user\_query\+assistant thinking prefix\.
- •T−\-prompt\.system\_role\+<context\>\+user\_query\(no<prohibition\>, no prefix\)\.
- •S rollout prompt\.Same as the T\+ prompt, butwithoutthe assistant thinking prefix\.

The assistant thinking prefix is a short, fixed message appended to theT\+T^\{\+\}prompt that serves two purposes\. First, the self\-cue“I must comply with the prohibition”conditionsT\+T^\{\+\}to enhance its prohibition compliance capability\. Second,“I will carefully decide whether to answer normally or refuse the user’s request”balances the teacher across both regimes—refusing on violation\-inducing samples while answering normally on legitimate ones\.

#### Logits and effective tokens\.

For a shared rolloutyyof lengthTTover vocabularyVV, the three logits tensorszS,zT\+,zT−∈ℝB×T×Vz\_\{S\},z\_\{T^\{\+\}\},z\_\{T^\{\-\}\}\\in\\mathbb\{R\}^\{B\\times T\\times V\}are obtained by forwardingSS,T\+T^\{\+\}, andT−T^\{\-\}on their respective inputs and slicing along the response segment, so the three tensors are token\-wise aligned onyy\. LetpS=softmax​\(zS\)p\_\{S\}=\\mathrm\{softmax\}\(z\_\{S\}\), andp\+p^\{\+\},p−p^\{\-\};log⁡pS\\log p\_\{S\},log⁡p\+\\log p^\{\+\},log⁡p−\\log p^\{\-\}denote the corresponding log\-softmax tensors\. The effective\-token setℳ\\mathcal\{M\}excludes prompt positions and padding, restricting all losses to the response segment\.

#### Vocabulary\-dim pre\-shrink\.

At each position we take the top\-KVK\_\{V\}indices fromzT\+z\_\{T^\{\+\}\}andzT−z\_\{T^\{\-\}\}separately, take their union, and set the logits at unselected coordinates to−104\-10^\{4\}\. Unless otherwise noted,VVrefers to the shrunk vocabulary\.

## 4Method

As illustrated in Figure[2](https://arxiv.org/html/2608.14644#S2.F2), DUET \(DUal\-tEacherToken\-selective OPD\) puts a per\-token disagreement signal fromT\+/T−T^\{\+\}/T^\{\-\}to two complementary uses: \(1\) Violation\-Token Selection, which performs online signal cleaning by discarding tokens on which the two teachers agree, since such agreement either reflects redundant supervision unrelated to the prohibition or, worse, misleading teacher supervision from student prefix pollution; and \(2\) Preference\-direction Learning, which pushes the student at token granularity away fromT−T^\{\-\}and towardT\+T^\{\+\}on the surviving tokens, namely the student’s violation tokens localized where the positive and negative teachers disagree\.

### 4\.1Violation\-Token Selection

T\+T^\{\+\}andT−T^\{\-\}share exactly the same weights, and the two forwards are conditioned on inputs that differ only in the presence or absence of the prohibition\. Any per\-token disagreement can therefore only be caused by the prohibition itself, isolated from confounders such as intrinsic student capability gaps or teacher–student distributional mismatch\. This property turns the per\-token agreement pattern into an online signal for cleaning supervision: positions where the two teachers agree carry no prohibition\-specific information and should be discarded, while positions of disagreement pinpoint where compliance behavior actually differs\.

#### Agreement Positions: Redundant or Misleading Supervision\.

Positions whereT\+T^\{\+\}andT−T^\{\-\}agree can be discarded for one of two reasons, both of which motivate excluding them from the loss\.

1. 1\.Redundant supervision\.Whenp\+p^\{\+\}andp−p^\{\-\}are nearly identical, the position carries no prohibition\-relevant signal; spending gradient budget here merely reinforces ordinary next\-token statistics that are orthogonal to the compliance behavior we aim to sharpen\.
2. 2\.Misleading supervision\.T\+T^\{\+\}may be dragged towardT−T^\{\-\}by an erroneous student prefix, producing spurious agreement\. Supervising such positions would let analready\-degraded positive teacherpull the student further astray—precisely the failure mode of vanilla OPD\.

#### Disagreement Positions: A Three\-Stage Filter\.

Conversely, per\-token disagreement is a natural signal for identifying tokens worth supervising\. We operationalize it through the following filter\.

1. 1\.KL disagreement\.For each position\(b,t\)∈ℳ\(b,t\)\\in\\mathcal\{M\}, db,t=12​\[KL​\(pb,t\+∥pb,t−\)\+KL​\(pb,t−∥pb,t\+\)\]\.d\_\{b,t\}\\;=\\;\\tfrac\{1\}\{2\}\\\!\\left\[\\,\\mathrm\{KL\}\(p^\{\+\}\_\{b,t\}\\,\\\|\\,p^\{\-\}\_\{b,t\}\)\+\\mathrm\{KL\}\(p^\{\-\}\_\{b,t\}\\,\\\|\\,p^\{\+\}\_\{b,t\}\)\\,\\right\]\.\(2\)
2. 2\.Per\-sequence top\-KK\.For each sequencebb, retain theKb=⌈0\.1⋅\|ℳb\|⌉K\_\{b\}=\\lceil 0\.1\\cdot\|\\mathcal\{M\}\_\{b\}\|\\rceilpositions with the largestdb,td\_\{b,t\}\. Per\-sequence selection prevents long sequences from swamping the mask and short sequences from being starved\.
3. 3\.Max\-pool dilation\.Apply a 1\-D max\-pool with kernel55\(a±2\\pm 2neighborhood\) along the sequence axis and intersect withℳ\\mathcal\{M\}\. Violations often span short sub\-word clusters, so dilation captures neighboring tokens that top\-KKnarrowly missed but that still matter\.

### 4\.2Preference\-Directed Learning

We cast the token\-granularity signals from the two teachers as preference\-directed learning: at every high\-disagreement position, the student is simultaneously pulled toward the positive teacherT\+T^\{\+\}and pushed away from the negative teacherT−T^\{\-\}\. This embeds DPO\-style preference optimization directly into OPD, and it does soonline at token granularity, without ever collecting offline preference pairs\.

#### Pulling towardT\+T^\{\+\}\.

LposL\_\{\\mathrm\{pos\}\}takesp\+p^\{\+\}as the target on the selected high\-disagreement positions and pulls the student toward the compliant teacher’s distribution:

Lpos=𝔼y∼πS\(⋅∣x\)\[∑t∈𝒱∗​\(y\)DKL\(p\+\(⋅∣x,y<t\)∥pS\(⋅∣x,y<t\)\)\]\.\\displaystyle L\_\{\\mathrm\{pos\}\}\\;=\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\}\\Big\[\\sum\_\{t\\in\\mathcal\{V\}^\{\\ast\}\(y\)\}D\_\{\\mathrm\{KL\}\}\\big\(p^\{\+\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,p\_\{S\}\(\\cdot\\mid x,y\_\{<t\}\)\\big\)\\Big\]\.

\(3\)

#### Pushing away fromT−T^\{\-\}\.

LnegL\_\{\\mathrm\{neg\}\}uses the disagreement massΔb,t​\(u\)=\(pb,t−​\(u\)−pb,t\+​\(u\)\)\+\\Delta\_\{b,t\}\(u\)=\\big\(p^\{\-\}\_\{b,t\}\(u\)\-p^\{\+\}\_\{b,t\}\(u\)\\big\)\_\{\+\}as a soft\-suppression target, capturing the candidatesT−T^\{\-\}prefers overT\+T^\{\+\}:

Lneg=𝔼\(b,t\)∈𝒱†​\[∑uΔb,t​\(u\)​pS​\(u∣b,t\)\]\.L\_\{\\mathrm\{neg\}\}=\\mathbb\{E\}\_\{\(b,t\)\\in\\mathcal\{V\}^\{\\dagger\}\}\\\!\\Big\[\\sum\_\{u\}\\Delta\_\{b,t\}\(u\)\\,p\_\{S\}\(u\\mid b,t\)\\Big\]\.\(4\)

#### Pulling toward a purified target\.

BecauseT\+T^\{\+\}itself may retain residualT−T^\{\-\}\-direction contamination \(the same mechanism behind the misleading class of tokens above\), we also synthesize a purer target in logit space by extrapolatingzT\+z\_\{T^\{\+\}\}further fromzT−z\_\{T^\{\-\}\}along the disagreement direction:

zadj\\displaystyle z\_\{\\mathrm\{adj\}\}=clip​\(zT\+−γsyn​\(zT−−zT\+\),±C\),\\displaystyle\\;=\\;\\mathrm\{clip\}\\\!\\Big\(\\,z\_\{T^\{\+\}\}\-\\gamma\_\{\\mathrm\{syn\}\}\(z\_\{T^\{\-\}\}\-z\_\{T^\{\+\}\}\),\\;\\pm C\\,\\Big\),\(5\)padj\\displaystyle p\_\{\\mathrm\{adj\}\}=softmax​\(zadj\)\.\\displaystyle\\;=\\;\\mathrm\{softmax\}\(z\_\{\\mathrm\{adj\}\}\)\.Geometrically,zadjz\_\{\\mathrm\{adj\}\}produces a soft target from which theT−T^\{\-\}\-direction contamination has been actively stripped away, and a standard forward KD term aligns the student with it:

Lsyn=𝔼y∼πS\(⋅∣x\)\[∑t∈𝒱∗​\(y\)DKL\(padj\(⋅∣x,y<t\)∥pS\(⋅∣x,y<t\)\)\]\.\\displaystyle L\_\{\\mathrm\{syn\}\}\\;=\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\}\\Big\[\\sum\_\{t\\in\\mathcal\{V\}^\{\\ast\}\(y\)\}D\_\{\\mathrm\{KL\}\}\\big\(p\_\{\\mathrm\{adj\}\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,p\_\{S\}\(\\cdot\\mid x,y\_\{<t\}\)\\big\)\\Big\]\.

\(6\)Together withLposL\_\{\\mathrm\{pos\}\}andLnegL\_\{\\mathrm\{neg\}\}, this term completes the preference\-directed picture: the two independent teacher losses set the student’s coarse direction, while the synthesized dual\-teacher loss then refines it toward a target purified along the same preference axis\.

#### Utility preservation\.

To prevent over\-refusal within the same business context, we add a top\-KKKD againstT\+T^\{\+\}on the legitimate queries\. Let𝒦b,t\+\\mathcal\{K\}^\{\+\}\_\{b,t\}denote the indices of the topKtkK\_\{\\mathrm\{tk\}\}logits ofzT\+z\_\{T^\{\+\}\}at\(b,t\)\(b,t\), and let the truncated\-renormalized target be

p~b,t\+​\(u\)=pb,t\+​\(u\)​1​\[u∈𝒦b,t\+\]∑u′∈𝒦b,t\+pb,t\+​\(u′\),\\tilde\{p\}^\{\+\}\_\{b,t\}\(u\)\\;=\\;\\frac\{p^\{\+\}\_\{b,t\}\(u\)\\,\\mathbb\{1\}\[u\\in\\mathcal\{K\}^\{\+\}\_\{b,t\}\]\}\{\\sum\_\{u^\{\\prime\}\\in\\mathcal\{K\}^\{\+\}\_\{b,t\}\}p^\{\+\}\_\{b,t\}\(u^\{\\prime\}\)\},\(7\)i\.e\., we restrictT\+T^\{\+\}’s softmax to its top\-KtkK\_\{\\mathrm\{tk\}\}candidates and zero the tail\. The loss is

Lutil=𝔼y∼πS\(⋅∣x\)\[∑t∈𝒩​\(y\)DKL\(p~\+\(⋅∣x,y<t\)∥pS\(⋅∣x,y<t\)\)\]\.\\displaystyle L\_\{\\mathrm\{util\}\}\\;=\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\}\\Big\[\\sum\_\{t\\in\\mathcal\{N\}\(y\)\}D\_\{\\mathrm\{KL\}\}\\big\(\\tilde\{p\}^\{\+\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,p\_\{S\}\(\\cdot\\mid x,y\_\{<t\}\)\\big\)\\Big\]\.

\(8\)This term does not invokeT−T^\{\-\}and does not enter the𝒱∗\\mathcal\{V\}^\{\\ast\}selection\.

#### Total loss\.

Combining the three preference\-directed terms with the utility\-preservation term gives

ℒ=λpos​Lpos\+λneg​Lneg\+λsyn​Lsyn\+λutil​Lutil\.\\displaystyle\\mathcal\{L\}\\;=\\;\\lambda\_\{\\mathrm\{pos\}\}\\,L\_\{\\mathrm\{pos\}\}\\;\+\\;\\lambda\_\{\\mathrm\{neg\}\}\\,L\_\{\\mathrm\{neg\}\}\\;\+\\;\\lambda\_\{\\mathrm\{syn\}\}\\,L\_\{\\mathrm\{syn\}\}\\;\+\\;\\lambda\_\{\\mathrm\{util\}\}\\,L\_\{\\mathrm\{util\}\}\.

\(9\)The first three terms jointly implement DPO\-style preference optimization at token granularity on the high\-disagreement set𝒱∗\\mathcal\{V\}^\{\\ast\}, whileLutilL\_\{\\mathrm\{util\}\}safeguards utility\.

## 5Benchmark

None of existing Constraint\-Following benchmarks in §[2](https://arxiv.org/html/2608.14644#S2)jointly measures explicit\-violation refusal, paraphrase\-probe robustness, and over\-refusal risk on legitimate queries within a single business context\. We construct an industrial Prohibition\-Compliance benchmark spanning five task families covering explicit\-refusal, paraphrase robustness, and over\-refusal\. All prompts and other details used in the stages described below, including data construction, evaluation, and human validation, are provided in the Appendix\.

### 5\.1Task Families

The Prohibition\-Compliance benchmark includes five task families\. T1 targets PII and memory, covering direct asks for private information, indirect asks wrapped as business tasks, and paraphrase or encoding variants\. T2 targets safety redlines, covering direct violating requests and jailbreak variants such as role\-play and academic\-research wrapping\. T3 targets custom business content, covering competitors, out\-of\-scope topics, copyrighted assets, and output\-format violations\. T4 targets tool\-definition leakage, including asks for tool names, parameters, and raw tool return payloads\. T5 targets business flows and SOPs, covering asks for the full flow, risk\-control thresholds, and tier rules\.

### 5\.2Sample Structure

Each sample contains system\_role \(task family’s fixed template\), context \(contextual corpus for the scenario, e\.g\., business text, business\-flow SOP, user profile\), prohibition, user\_query, and, for some samples, tools and tool\_call\_history\. See Appendix for source\-corpus handling, per\-family rule pools, system\_role templates, and per\-task construction sub\-flows\.

Samples are further partitioned into three buckets by inducement\. Within a task family, the three buckets share system\_role and context and differ only in user\_query\. Theinduction bucketcontains queries that would naturally elicit a violating response in the absence of prohibition, including direct asks and indirect asks wrapped as business tasks\. Theboundary bucketis LLM\-rewritten from the corresponding induction query into a surface\-sanitized form that still retains leakage potential; its role is to force compliance behavior to track semantics rather than surface lexical cues, suppressing degeneration into keyword shortcuts\. Thenormal bucketcontains legitimate queries in the same business scenario whose answers do not touch the prohibition, ensuring that any compliance gain is not obtained by uniformly refusing within the same business context\.

### 5\.3Data Construction

Data construction follows a fixed five\-stage pipeline: corpus loading; scenario\-rule mapping; induction query synthesis, in which an LLM generates induction queries; boundary rewriting, in which an LLM rewriter converts each induction query into a surface\-sanitized borderline variant; and normal synthesis, which produces normal queries in the same business scenario whose answers require no access to the prohibitions\. All LLM\-based components in the data construction pipeline are instantiated with Qwen3\-235B\-A22B\(Yanget al\.[2025](https://arxiv.org/html/2608.14644#bib.bib2)\)\.

The training set contains approximately 2,500 samples, with induction plus boundary accounting for∼\\sim75% and normal for∼\\sim25%\. The evaluation set contains 700 samples, split as induction plus boundary 600 \(of which 200 have prohibitions drawn from a new pool outside the training constraint pool\) and normal 100\. To assess construction quality, we randomly sampled a subset of the constructed data for manual inspection, and 98\.4% of samples passed human validation\.

### 5\.4Evaluation

Evaluation uses an LLM\-as\-a\-judge with two bucket\-dependent prompt schemes\. On induction and boundary, the prohibition\-violation judge takes the triple \(user\_query, assistant\_response, prohibition\), emitting T if the response strictly satisfies all prohibitions and F if at least one is violated\. On normal, the non\-refusal judge takes the pair \(user\_query, assistant\_response\) and emits T if the assistant substantively addresses the query on\-topic rather than refusing, evading, returning empty content, or going off\-topic\.

## 6Experiments

### 6\.1Experimental Setup

#### Backbones and Scales\.

We validate DUET across two model families and three scale configurations: Qwen2\.5\-7B→\\toQwen2\.5\-1\.5B, Qwen3\-8B→\\toQwen3\-1\.7B, and Qwen3\-8B→\\toQwen3\-8B \(self\-distillation\)\(Qwenet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib1); Yanget al\.[2025](https://arxiv.org/html/2608.14644#bib.bib2)\)\. All LLM\-as\-a\-judge evaluations use DeepSeek\-V4\-Flash\(DeepSeek\-AIet al\.[2026](https://arxiv.org/html/2608.14644#bib.bib3)\)\.

#### Baselines\.

Beyond the raw student \(Student\) and the teacher \(Teacher, with and without an assistant thinking prefix\) as references, the main comparisons are: \(i\) OPD\-F / OPD\-R—vanilla OPD with single\-teacher forward / reverse KL\(Guet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib13); Agarwalet al\.[2024](https://arxiv.org/html/2608.14644#bib.bib12)\); \(ii\) AlignDistil\(Zhanget al\.[2025](https://arxiv.org/html/2608.14644#bib.bib16)\)—distillation from a token\-level target distribution synthesized from two oppositely trained preference models; \(iii\) TIP\(Xuet al\.[2026](https://arxiv.org/html/2608.14644#bib.bib18)\)—OPD with token selection driven by student uncertainty and teacher–student distributional gap\.

#### Training and Rollout\.

The training pipeline is implemented under the Distributed Data Parallel \(DDP\) paradigm, in which a low\-rank adaptation \(LoRA\)\(Huet al\.[2022](https://arxiv.org/html/2608.14644#bib.bib11)\)scheme with rankr=16r=16is applied uniformly to all attention and MLP projection matrices of the backbone model\. All experiments are conducted on a single node equipped with 8×\\timesNVIDIA H20 GPUs, where 2 GPUs are dedicated to rollout and the remaining 6 GPUs to policy training\. Under this setup, the full training of Qwen2\.5\-1\.5B and Qwen3\-1\.7B takes only0\.5and1\.0wall\-clock hours respectively, which corresponds to roughly75%of the time required by the fastest baseline \(TIP\) on identical hardware\.

### 6\.2Prohibition\-Compliance Evaluation

We compare DUET against representative distillation baselines on the industrial Prohibition\-Compliance benchmark, with results reported in Table[1](https://arxiv.org/html/2608.14644#S6.T1)\.

Qwen2\.5\-7B→\\toQwen2\.5\-1\.5BQwen3\-8B→\\toQwen3\-\{1\.7B, 8B\}MethodT1T2T3T4T5Viol\.Util\.Over\.Hum\.MethodT1T2T3T4T5Viol\.Util\.Over\.Hum\.Teacher†63\.687\.942\.979\.344\.359\.389\.063\.665\.0Teacher†88\.685\.749\.386\.466\.480\.246\.075\.378\.6Teacher∗40\.090\.025\.763\.636\.443\.895\.051\.154\.1Teacher∗75\.095\.725\.075\.067\.164\.586\.067\.668\.6Student35\.782\.920\.748\.646\.441\.579\.046\.950\.0Student65\.085\.720\.047\.144\.346\.886\.052\.457\.1OPD\-F51\.482\.930\.070\.745\.751\.087\.056\.157\.1OPD\-F92\.995\.058\.685\.077\.181\.384\.081\.782\.7OPD\-R51\.485\.732\.182\.148\.654\.891\.060\.064\.3OPD\-R92\.195\.754\.390\.077\.981\.088\.082\.083\.3AlignDistil33\.677\.917\.972\.945\.743\.586\.049\.651\.4AlignDistil90\.090\.727\.945\.035\.055\.571\.057\.758\.6TIP52\.983\.632\.182\.947\.954\.592\.059\.962\.9TIP91\.495\.051\.495\.074\.380\.090\.081\.480\.0DUET81\.495\.752\.987\.159\.372\.393\.075\.377\.7DUET\-1\.7B92\.195\.059\.391\.484\.383\.888\.084\.487\.3DUET\-8B92\.997\.960\.097\.182\.185\.291\.086\.088\.1Table 1:Main results on the Prohibition\-Compliance Benchmark\. The left block reports Qwen2\.5\-7B→\\toQwen2\.5\-1\.5B, and the right block reports Qwen3\-8B→\\toQwen3\-\{1\.7B, 8B\}\. T1–T5 correspond to the five task families of Prohibition\-Compliance benchmark; Viol\. denotes the violation\-refusal rate on the induction & boundary bucket, Util\. denotes the legitimate\-utility rate on the normal bucket, and Over\. denotes the overall accuracy on the full set\. Hum\. denotes the human\-rated score on a stratified sample of 30% of the data randomly drawn from each category\.†\\daggermarks the teacher run with the assistant thinking prefix,∗\\astmarks the raw teacher without prefix\. On the right block, all baselines and DUET\-1\.7B are distilled to Qwen3\-1\.7B, while DUET\-8B is a Qwen3\-8B self\-distillation variant\.Boldmarks the best andunderlinemarks the second best in each column \(Teacher rows are references and excluded from ranking\)\. DUET’s improvements over all baselines are statistically significant under McNemar’s paired test with Holm–Bonferroni correction; detailed per\-comparisonpp\-values are reported in the Appendix\.The teacher is unbalanced on the joint objective, and the thinking prefix is only a stop\-gap\.The raw teacher is much weaker on Viol\. than Util\. \(Qwen2\.5: 43\.8% vs\. 95\.0%\), lacking an inherent prohibition compliance capability\. Adding the thinking prefix raises Viol\. but collapses Util\. on Qwen3 from 86% to 46%, confirming that prompt\-side constraints alone cannot jointly achieve compliance and utility\. Prohibition alignment must be completed in the weights\.

Single\-teacher OPD fails to learn local violation refusal\.OPD\-F and OPD\-R reach a moderate Over\. on both experiments, but their violation\-refusal rate is consistently at least 17 points below DUET \(Qwen2\.5: 51\.0/54\.8 vs\. 72\.3; Qwen3\-1\.7B: 81\.3/81\.0 vs\. 83\.8\)\. This stems from the two structural gaps of vanilla OPD identified in §[1](https://arxiv.org/html/2608.14644#S1): a single teacher cannot online\-detect where it has been dragged by the student polluted prefix, and a uniform all\-token loss dilutes supervision on the few tokens that decide compliance\.

Sequence\-level preference optimization is mismatched with violation locality\.AlignDistil’s Over\. is below OPD\-F on both experiments \(Qwen2\.5: 49\.6 vs\. 56\.1; Qwen3\-1\.7B: 57\.7 vs\. 81\.7\), because sequence\-level chosen/rejected labels condemn an entire response for the sake of a few offending tokens, so the gradient contaminates neutral positions while diluting the signal at the true violation positions, and this mismatch that is amplified on token\-localized violation tasks\.

DUET jointly improves both objectives and scales with model size\.DUET’s margin over the baselines is markedly larger on Qwen2\.5 than on Qwen3, and this gap itself corroborates the two structural gaps of vanilla OPD that DUET is designed to close, with detailed case studies in Appendix\.

First,the dirtier the teacher, the larger the cleaning gain: vanilla OPD cannot detect online the teacher signals contaminated by the student’s prefix\. On Qwen2\.5 the teacher’s Viol\. is only 59\.3%, soT\+T^\{\+\}is easily dragged towardT−T^\{\-\}by the student’s erroneous prefix, producing large amounts of falsely\-consistent yet misleading supervision; DUET removes such positions online via per\-tokenT\+/T−T^\{\+\}/T^\{\-\}disagreement, yielding remarkable gains\. In contrast, Qwen3\-8B’s teacher already reaches Viol\. 80\.2%, itsT\+T^\{\+\}is close to an ideally compliant distribution, and the residualT−T^\{\-\}\-direction contamination relative toT\+T^\{\+\}is intrinsically smaller with a weaker disagreement signal, compressing the headroom that DUET can extrapolate over and naturally narrowing its lead\.

Second,the larger the teacher–student mismatch, the larger the denoising gain of token selection: vanilla OPD and divergence\-based selection methods such as TIP cannot separate violation\-related teacher–student differences from intrinsic distributional mismatch\. Qwen2\.5\-7B and Qwen2\.5\-1\.5B use different vocabularies and thus exhibit a structural teacher–student distributional gap, in which genuine prohibition\-violating tokens — which should be treated as hard tokens — are diluted and become hard to distinguish from the pervasive mismatch noise; Qwen3\-8B and Qwen3\-1\.7B share a vocabulary, so this gap is inherently smaller\. DUET’s selection signal comes from same\-weightT\+/T−T^\{\+\}/T^\{\-\}and depends solely on whether the prohibition is visible, keeping the mismatch outside the filter—so its net gain is amplified on Qwen2\.5 and naturally narrower on Qwen3\.

Table 2:Per\-category CSR and overall CSR / ISR on SysBench\. The columns \(Co / Ac / Fo / Ba / Ro / St\) are per\-category CSRs for Content / Action / Format / Background / Role / Style, while CSR and ISR denote the overall constraint\-satisfaction rate and instruction\-satisfaction rate\.†\\daggermarks the teacher as a performance ceiling\. Significance stars are attached to each baseline row and indicate the difference between DUET and that baseline \(McNemar’s paired test with Holm–Bonferroni correction, family size = 5;∗\\astp<0\.05p<0\.05,⋄\\diamondp<0\.01p<0\.01,⋆\\starp<0\.001p<0\.001\)\.
### 6\.3Constraint\-Following Evaluation: SysBench

Benchmark and metrics\.SysBench\(Qinet al\.[2025](https://arxiv.org/html/2608.14644#bib.bib25)\)is the first fine\-grained benchmark specifically designed to evaluate LLMs’ ability to follow system\-prompt constraints and prohibitions, comprising 500 system prompts spanning six constraint categories; the primary metrics areCSR\(per\-criterion pass rate\) andISR\(all\-criteria\-pass rate within a single turn\)\. No SysBench training data is used; we evaluate the model trained on Qwen2\.5\-7B→\\toQwen2\.5\-1\.5B directly in §[6\.2](https://arxiv.org/html/2608.14644#S6.SS2)\.

Analysis\.As shown in Table[2](https://arxiv.org/html/2608.14644#S6.T2), DUET significantly outperforms all distillation baselines on both CSR and ISR \(p<0\.05∼p<0\.001p<0\.05\\sim p<0\.001\), and its training data contains no SysBench\-style system prompts whatsoever\. This shows that the alignment signal isolated byT\+/T−T^\{\+\}/T^\{\-\}disagreement istask\-agnostic, general prohibition\-alignment capability, rather than memorization of a specific class of constraint templates\.

### 6\.4General Capability Preservation

GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2608.14644#bib.bib28)\)and MATH\-500\(Lightmanet al\.[2023](https://arxiv.org/html/2608.14644#bib.bib29)\)evaluate multi\-step logical reasoning and competition\-level mathematical ability, respectively\. As both benchmarks are formulated as multiple\-choice tasks, following standard evaluation protocol, we extract the selected option from each response and report accuracy\.

After DUET training, the model shows no significant degradation on either benchmark, reaching 68\.3 on GSM8K and 52\.2 on MATH\-500 on the Qwen2\.5\-7B→\\toQwen2\.5\-1\.5B distillation model from §[6\.2](https://arxiv.org/html/2608.14644#S6.SS2), without including any GSM8K or MATH\-500 data during training\. A per\-subject breakdown on MATH\-500 further shows that this small aggregate difference is not uniform—Algebra and Prealgebra remain unchanged, Geometry and Precalculus even improve slightly, and the residual gap is concentrated on Number Theory and Counting & Probability, both of which are pre\-existing weak points of the Student that demand long\-chain reasoning and are within the acceptable fluctuation range of LoRA training capacity\. Detailed per\-subject numbers are deferred to the Appendix\.

### 6\.5Ablation Study

Table 3:Ablation over each design element of DUET \(Qwen2\.5\-7B→\\toQwen2\.5\-1\.5B\)\. The first three rows probe framework\-level choices; the following four rows remove one of the four terms of the total loss in turn; DUET is the complete method\.#### Framework\-level ablations\.

Table[3](https://arxiv.org/html/2608.14644#S6.T3)reports the ablation results\. Teacher sampling \(off\-policy\) drops Over\. from 75\.3 to 62\.9, confirming that on\-policy is a necessary precondition for prohibition alignment: teacher\-sampled trajectories \(as in SFT and RLHF\) cannot expose the student’s true failure trajectories\. Keeping onlyT\+T^\{\+\}yields the worst Viol\. \(58\.8\) together with the all\-token\-supervision setting \(66\.0\), confirming that a single teacher can neither locate the position of violation tokens nor filter out misleading supervision\.

#### Loss\-term ablations\.

- •RemovingLsynL\_\{\\mathrm\{syn\}\}\(synthesized\-teacher KD\): Viol\. drops from 72\.3 to 63\.7, the largest violation\-refusal drop across all ablations—showing that the purified target obtained in Phase 2 by extrapolating along theT\+/T−T^\{\+\}/T^\{\-\}disagreement direction is the largest source of violation refusal, matching the motivation that “T\+T^\{\+\}itself still retains residualT−T^\{\-\}\-direction contamination, and the synthesized target should be introduced only after the student has stabilized, for finer alignment\.”
- •RemovingLposL\_\{\\mathrm\{pos\}\}\(T\+T^\{\+\}pull\): Over\. drops by 5\.7 points\. Without the anchor pulling the student towardT\+T^\{\+\}, the model over\-relies onLnegL\_\{\\mathrm\{neg\}\}to learn fromT−T^\{\-\}in reverse, which pulls the violation\-refusal rate down\.
- •RemovingLnegL\_\{\\mathrm\{neg\}\}\(soft suppression of violation candidates\): Viol\. drops from 72\.3 to 67\.0, indicating that squeezing the student distribution off the violation\-candidate mass\(p−−p\+\)\+\(p^\{\-\}\-p^\{\+\}\)\_\{\+\}contributes independently to refusal, complementing the pull fromLposL\_\{\\mathrm\{pos\}\}\.
- •RemovingLutilL\_\{\\mathrm\{util\}\}\(top\-K KD on the legitimate query: Util\. collapses from 93\.0 to 62\.0—the largest utility drop across all ablations\.LutilL\_\{\\mathrm\{util\}\}is the core mechanism preventing over\-refusal within the same business context\.

Taken together, removing any one of the four losses produces a drop on its corresponding target, thereby verifying the orthogonality and the design necessity\.

## 7Conclusion

We formalizedProhibition Compliance—runtime\-configurable, token\-localized rule following with a joint compliance–utility requirement—and identified two structural gaps of vanilla on\-policy distillation on this task: teacher drift under the student’s prefix and undifferentiated token\-mean supervision\. Our proposedDUETcloses both gaps with a single mechanism, a same\-weight negative teacherT−T^\{\-\}whose per\-token disagreement withT\+T^\{\+\}simultaneously cleans misleading supervision and defines a push/pull preference direction, embedding token\-granular preference optimization into OPD without any offline preference data\. Together with the industrial Prohibition\-Compliance benchmark we release, experiments across Qwen2\.5 and Qwen3 at 1\.5B–8B consistently show that DUET improves violation refusal and legitimate\-query utility over strong distillation baselines while preserving general capability\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1),[§3](https://arxiv.org/html/2608.14644#S3.p1.1),[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx2.p1.1)\.
- A\. Ahmed, K\. Klyman, Y\. Zeng, S\. Koyejo, and P\. Liang \(2025\)SpecEval: evaluating model adherence to behavior specifications\.External Links:2509\.02464,[Link](https://arxiv.org/abs/2509.02464)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- G\. Bai, J\. Liu, X\. Bu, Y\. He, J\. Liu, Z\. Zhou, Z\. Lin, W\. Su, T\. Ge, B\. Zheng, and W\. Ouyang \(2024\)MT\-bench\-101: a fine\-grained benchmark for evaluating large language models in multi\-turn dialogues\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7421–7454\.External Links:[Link](https://aclanthology.org/2024.acl-long.401/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.401)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- F\. Brahman, S\. Kumar, V\. Balachandran, P\. Dasigi, V\. Pyatkin, A\. Ravichander, S\. Wiegreffe, N\. Dziri, K\. Chandu, J\. Hessel, Y\. Tsvetkov, N\. A\. Smith, Y\. Choi, and H\. Hajishirzi \(2024\)The art of saying no: contextual noncompliance in language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 49706–49748\.External Links:[Document](https://dx.doi.org/10.52202/079017-1573),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/58e79894267cf72c66202228ad9c6057-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§6\.4](https://arxiv.org/html/2608.14644#S6.SS4.p1.1)\.
- DeepSeek\-AI, A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling, C\. Lu, C\. Zhao, C\. Deng, C\. Hou, C\. Xu, C\. Shao, C\. Ruan, C\. Sun, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Li, D\. Ji, E\. Li, F\. Wei, F\. Lin, F\. Yuan, F\. Xia, F\. Dai, G\. Hao, G\. Chen, G\. Cao, G\. Meng, G\. Li, H\. Yu, H\. Zhang, H\. Xu, H\. Li, H\. Liang, H\. Zhang, H\. Luo, H\. Wei, H\. Yuan, H\. Zhang, H\. Luo, H\. Chen, H\. Ji, H\. Zhang, H\. Ding, H\. Tang, H\. Cao, H\. Gao, H\. Qu, H\. Zeng, J\. Yang, J\. Zhu, J\. Luo, J\. Song, J\. Yu, J\. Huang, J\. Cai, J\. Liang, J\. Zhou, J\. Ye, J\. Li, J\. Xu, J\. Hu, J\. Yang, J\. Chen, J\. Yan, J\. Chen, J\. Zhou, J\. Xiang, J\. Yuan, J\. Cheng, J\. Zhou, J\. Zhu, J\. Yu, J\. Sun, J\. Ran, J\. Jiang, J\. Qiu, J\. Li, J\. Zheng, J\. Song, K\. Dong, K\. Gao, K\. Guan, K\. Zhou, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Wang, L\. Xia, L\. Zhang, L\. Zhao, L\. Guo, L\. Luo, L\. Ma, L\. Zhu, L\. Wang, L\. Cai, L\. Zhang, L\. Chen, M\. Di, M\. Xu, M\. Mei, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Zhou, M\. Han, N\. Wang, P\. Huang, P\. Wang, P\. Cong, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Li, Q\. Chen, Q\. Du, Q\. Jiang, R\. Tian, R\. Xu, R\. Lu, R\. Xu, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Chen, R\. Yin, R\. Xu, R\. Shen, R\. Zhang, R\. Chen, S\. Liu, S\. Lu, S\. Sun, S\. Zhou, S\. Chen, S\. Cai, S\. Nie, S\. Wu, S\. Chen, S\. Hu, S\. Liu, S\. Hu, S\. Ma, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. Yu, S\. Zhou, T\. Ni, T\. Yun, T\. Jin, T\. Pei, T\. Ye, T\. Lin, T\. Ji, T\. Cui, T\. Yue, T\. Yu, T\. Wang, W\. Zhang, W\. Xiao, W\. Zeng, W\. An, W\. Zhao, W\. Liu, W\. Liang, W\. Pang, W\. Luo, W\. Yao, W\. Gao, W\. Yang, W\. Huang, W\. Hou, W\. Zhang, W\. Ma, X\. Gao, X\. He, X\. Wang, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Chen, X\. Zhang, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Liu, X\. Yu, X\. Li, X\. Yang, X\. Zhang, X\. Chen, X\. Wang, X\. Su, X\. Chen, X\. Lin, X\. Fu, Y\. Yan, Y\. Wang, Y\. Ma, Y\. Luo, Y\. Zhang, Y\. Xu, Y\. Ma, Y\. Huang, Y\. Li, Y\. Li, Y\. Xu, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Qian, Y\. Shao, Y\. Yu, Y\. Zhang, Y\. Ding, Y\. Shi, Y\. Wu, Y\. Xiong, Y\. Ma, Y\. He, Y\. Tang, Y\. Zhou, Y\. Luo, Y\. Zhong, Y\. Piao, Y\. Wang, Y\. Zhang, Y\. Chen, Y\. Tan, Y\. Wei, Y\. Ma, Y\. Liu, Y\. Yang, Y\. Guo, Y\. Wu, Y\. Wu, Y\. Li, Y\. Cheng, Y\. Ou, Y\. Xu, Y\. Li, Y\. Wang, Y\. Yang, Y\. Xu, Y\. Wu, Y\. Meng, Y\. Zou, Y\. Zha, Y\. Xiong, Y\. Chen, Y\. Lin, Y\. Cao, Y\. Wang, Y\. Zhang, Y\. Yan, Y\. Lin, Y\. Gu, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. Zhou, Y\. Huang, Z\. Wu, Z\. Wang, Z\. Zhao, Z\. Ren, Z\. Zhang, Z\. Sha, Z\. Fu, Z\. Ju, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Gao, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Huang, Z\. Chen, Z\. Wu, Z\. Ren, Z\. Wu, Z\. Li, Z\. Zhang, Z\. Xu, Z\. Wang, Z\. Qu, Z\. Gu, Z\. Zhu, Z\. Li, Z\. Zhang, Z\. Xie, Z\. Gao, Z\. Wan, Z\. Pan, and Z\. Yao \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx1.p1.3)\.
- S\. Gu, J\. Chen, S\. Zhou, A\. Cohan, and R\. Ying \(2026\)Rethinking reward supervision: rubric\-conditioned self\-distillation\.External Links:2606\.19327,[Link](https://arxiv.org/abs/2606.19327)Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1),[§3](https://arxiv.org/html/2608.14644#S3.p1.1),[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx2.p1.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx3.p1.2)\.
- Y\. Jiang, Y\. Wang, X\. Zeng, W\. Zhong, L\. Li, F\. Mi, L\. Shang, X\. Jiang, Q\. Liu, and W\. Wang \(2024\)FollowBench: a multi\-level fine\-grained constraints following benchmark for large language models\.External Links:2310\.20410,[Link](https://arxiv.org/abs/2310.20410)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- J\. Ko, T\. Chen, S\. Kim, T\. Ding, L\. Liang, I\. Zharkov, and S\. Yun \(2025\)DistiLLM\-2: a contrastive approach boosts the distillation of LLMs\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=rc65N9xIrY)Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DISTILLM: towards streamlined distillation for large language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1),[§3](https://arxiv.org/html/2608.14644#S3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§6\.4](https://arxiv.org/html/2608.14644#S6.SS4.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 27730–27744\.External Links:[Document](https://dx.doi.org/10.52202/068431-2011),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p2.1)\.
- Y\. Qin, T\. Zhang, T\. Zhang, Y\. Shen, W\. Luo, sunhaoze, Y\. Zhang, Y\. Qiao, weipeng chen, Z\. Zhou, W\. Zhang, and B\. CUI \(2025\)SysBench: can LLMs follow system message?\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KZWaxtzIRx)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1),[§6\.3](https://arxiv.org/html/2608.14644#S6.SS3.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx1.p1.3)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p2.1)\.
- Y\. Ruan, H\. Dong, A\. Wang, S\. Pitis, Y\. Zhou, J\. Ba, Y\. Dubois, C\. J\. Maddison, and T\. Hashimoto \(2024\)Identifying the risks of lm agents with an lm\-emulated sandbox\.External Links:2309\.15817,[Link](https://arxiv.org/abs/2309.15817)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 3008–3021\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p2.1)\.
- A\. Wei, N\. Haghtalab, and J\. Steinhardt \(2023\)Jailbroken: how does llm safety training fail?\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 80079–80110\.External Links:[Document](https://dx.doi.org/10.52202/075280-3508),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/fd6613131889a4b656206c50a8bd7790-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p1.1)\.
- J\. Wei, M\. Bosma, V\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. Le \(2022\)Finetuned language models are zero\-shot learners\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gEZrGCozdqR)Cited by:[§1](https://arxiv.org/html/2608.14644#S1.p2.1)\.
- C\. Xia, C\. Xing, J\. Du, X\. Yang, Y\. Feng, R\. Xu, W\. Yin, and C\. Xiong \(2024\)FOFO: a benchmark to evaluate llms’ format\-following capability\.External Links:2402\.18667,[Link](https://arxiv.org/abs/2402.18667)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- T\. Xie, X\. Qi, Y\. Zeng, Y\. Huang, U\. M\. Sehwag, K\. Huang, L\. He, B\. Wei, D\. Li, Y\. Sheng, R\. Jia, B\. Li, K\. Li, D\. Chen, P\. Henderson, and P\. Mittal \(2025\)SORRY\-bench: systematically evaluating large language model safety refusal\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YfKNaRktan)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard \(2026\)TIP: token importance in on\-policy distillation\.External Links:2604\.14084,[Link](https://arxiv.org/abs/2604.14084)Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.3](https://arxiv.org/html/2608.14644#S5.SS3.p1.1),[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx1.p1.3)\.
- S\. Zhang, X\. Zhang, T\. Zhang, B\. Hu, Y\. Chen, and J\. Xu \(2025\)AlignDistil: token\-level language model alignment as adaptive policy distillation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19791–19807\.External Links:[Link](https://aclanthology.org/2025.acl-long.972/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.972),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2608.14644#S6.SS1.SSSx2.p1.1)\.
- B\. Zheng, X\. Ma, Y\. Liang, J\. Ruan, X\. Fu, K\. Lin, B\. Zhu, K\. Zeng, and X\. Cai \(2026\)SCOPE: signal\-calibrated on\-policy distillation enhancement with dual\-path adaptive weighting\.External Links:2604\.10688,[Link](https://arxiv.org/abs/2604.10688)Cited by:[§2\.1](https://arxiv.org/html/2608.14644#S2.SS1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§2\.2](https://arxiv.org/html/2608.14644#S2.SS2.p1.1)\.