PolyAlign: Conditional Human-Distribution Alignment

# PolyAlign: Conditional Human-Distribution Alignment
Source: [https://arxiv.org/html/2606.13227](https://arxiv.org/html/2606.13227)
L\. D\. M\. S\. Sai Teja1Ufaq Khan2Sathira Silva2 Xiao Wu2Muhammad Haris Khan2 1NIT Silchar, India2MBZUAI, Abu Dhabi, UAE saitejalekkala05@gmail\.com, \{ufaq\.khan, muhammad\.haris\}@mbzuai\.ac\.ae

###### Abstract

Post\-training methods such as supervised fine\-tuning \(SFT\) and preference optimization typically align language models toward a single global assistant behavior\. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings\. We study this problem as*conditional human\-distribution alignment*: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style\. We introducePolyAlign, a distribution\-aware alignment framework that organizes bilingual interaction data into bucket\-specific human reference distributions defined by language, interaction track, response family, and length\.PolyAligncombines Bucket\-Aware SFT, which balances optimization across heterogeneous buckets, with Human\-Distribution Preference Optimization \(HDPO\), which regularizes preference learning using critic\-estimated distance to bucket\-specific human support\. Across a bilingual evaluation suite covering English and Chinese single\- and multi\-turn settings,PolyAlignimproves conditional naturalness and distributional faithfulness while preserving competitive task utility\. The results111GitHub:[https://github\.com/saitejalekkala33/PolyAlign\.git](https://github.com/saitejalekkala33/PolyAlign.git)suggest that post\-training should move beyond global alignment objectives toward interaction\-aware alignment with human response distributions\.

PolyAlign: Conditional Human\-Distribution Alignment

L\. D\. M\. S\. Sai Teja1Ufaq Khan2Sathira Silva2Xiao Wu2Muhammad Haris Khan21NIT Silchar, India2MBZUAI, Abu Dhabi, UAEsaitejalekkala05@gmail\.com, \{ufaq\.khan, muhammad\.haris\}@mbzuai\.ac\.ae

![Refer to caption](https://arxiv.org/html/2606.13227v1/figures/main/teaser.png)Figure 1:PolyAlign vs\. global alignment\.Unlike standard RLHF/DPO\-style post\-training, which can collapse diverse contexts into a generic assistant style, PolyAlign aligns responses to human distributions for more natural, situation\-appropriate generation\.## 1Introduction

Large language models \(LLMs\) have become capable through large\-scale pretraining, instruction tuning, and preference\-based post\-training\. These methods have substantially improved helpfulness and instruction following, enabling modern assistant\-style systems\. Scaling studies established that sufficiently large autoregressive models can perform a wide range of tasks from prompts alone\(Brown et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib4)\), while instruction tuning showed thatSFTon diverse task instructions can substantially improve zero\-shot generalization and interaction quality\(Wei et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib31); Sanh et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib26); Chung et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib6); Wang et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib28)\)\. Another progress came from alignment methods, based on human demonstrations, preference data, and reinforcement or preference optimization\(Christiano et al\.,[2017](https://arxiv.org/html/2606.13227#bib.bib5); Stiennon et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib27); Ouyang et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib23); Bai et al\.,[2022b](https://arxiv.org/html/2606.13227#bib.bib2),[a](https://arxiv.org/html/2606.13227#bib.bib1); Rafailov et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib25)\)\. Standard post\-training often pushes diverse interactions toward a single generic assistant style, rather than the kind of response humans would naturally give in each setting\.

PolyAlign aligns model responses across multiple interaction settings by conditioning post\-training on human response distributions\. We formulate alignment here as*conditional human\-distribution alignment*, shown in Fig[1](https://arxiv.org/html/2606.13227#S0.F1)\.Thehuman distributionis the set of responses that humans would naturally produce for a given interaction setting and how they vary in content, style, and form\. This is represented using human responses and their linguistic\-feature patterns within each bucket\.

Preference optimization methods such as DPO, ORPO, SimPO, KTO, and RRHF demonstrate that post\-training objectives can strongly reshape response behavior without requiring full RL pipelines\(Rafailov et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib25); Yuan et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib34); Hong et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib13); Meng et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib22); Ethayarajh et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib9)\)\. Controllable generation has long emphasized that useful text systems should support movement across a family of target distributions rather than optimize a single generic objective\(Keskar et al\.,[2019](https://arxiv.org/html/2606.13227#bib.bib15); Dathathri et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib7); Krause et al\.,[2021](https://arxiv.org/html/2606.13227#bib.bib18); Yang and Klein,[2021](https://arxiv.org/html/2606.13227#bib.bib33); Li and Liang,[2021](https://arxiv.org/html/2606.13227#bib.bib20); Lester et al\.,[2021](https://arxiv.org/html/2606.13227#bib.bib19)\)\. The views of Distribution\-matching after training suggest that standard SFT can over\-concentrate the distribution of learned generation, motivating more explicit objectives to target response distributions\(Korbak et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib17); Li et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib21)\)\.

#### Contributions\.

- •We formulatenaturalistic alignmentas matching the human response distribution appropriate to each interaction context, rather than optimizing toward a single global assistant behavior\. This formulation captures how response style, length, discourse structure, and language use vary across interaction regimes\.
- •We introduce PolyAlign, a framework that integrates conditional distributional targets into supervised fine\-tuning through Bucket\-SFT and into preference optimization through HDPO\.
- •We develop an evaluation protocol for conditional naturalness alongside standard utility measures, making it possible to quantify the trade\-off between task performance and conditional response\-distribution fidelity\.

More broadly, our goal is to shift the alignment question from “how do we make a model better on average?” to“how do we make a model produce the right kind of answer for the right kind of interaction?”We view this as a natural next step for post\-training: moving beyond generic alignment toward conditional, naturalistic, and interaction\-aware alignment\.

## 2Related Work

Instruction tuning and preference\-based alignment\.LLM post\-training has advanced from instruction tuning, which improved few\-shot and zero\-shot behavior\(Brown et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib4); Wei et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib31); Sanh et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib26); Chung et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib6)\), to synthetic and curated alignment corpora such as Self\-Instruct, LIMA, and OpenAssistant\(Wang et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib28); Zhou et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib37); Köpf et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib16)\)\. Human\-feedback methods likewise evolved from RLHF pipelines\(Christiano et al\.,[2017](https://arxiv.org/html/2606.13227#bib.bib5); Stiennon et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib27); Ouyang et al\.,[2022](https://arxiv.org/html/2606.13227#bib.bib23); Bai et al\.,[2022b](https://arxiv.org/html/2606.13227#bib.bib2),[a](https://arxiv.org/html/2606.13227#bib.bib1)\)to offline preference objectives including RRHF, DPO, ORPO, SimPO, and KTO\(Yuan et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib34); Rafailov et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib25); Hong et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib13); Meng et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib22); Ethayarajh et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib9)\)\. PolyAlign builds on these advances but shifts the target from a single global assistant behavior to interaction\-specific human response distributions\.

Structured alignment and controllable generation\.Our work is related to approaches that treat alignment as structured rather than one\-dimensional\. Multi\-attribute frameworks such as SteerLM and HelpSteer decompose helpfulness into multiple dimensions\(Dong et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib8); Wang et al\.,[2024b](https://arxiv.org/html/2606.13227#bib.bib30),[a](https://arxiv.org/html/2606.13227#bib.bib29)\), while studies of diversified preferences show that feedback datasets can encode distinct alignment behaviors\(Zeng et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib35)\)\. Controllable generation methods similarly steer models across behavior families rather than a single generic mode, as shown by CTRL, PPLM, GeDi, FUDGE, Prefix\-Tuning, and Prompt Tuning\(Keskar et al\.,[2019](https://arxiv.org/html/2606.13227#bib.bib15); Dathathri et al\.,[2020](https://arxiv.org/html/2606.13227#bib.bib7); Krause et al\.,[2021](https://arxiv.org/html/2606.13227#bib.bib18); Yang and Klein,[2021](https://arxiv.org/html/2606.13227#bib.bib33); Li and Liang,[2021](https://arxiv.org/html/2606.13227#bib.bib20); Lester et al\.,[2021](https://arxiv.org/html/2606.13227#bib.bib19)\)\. These works motivate our formulation, but instead of relying on manually specified attributes or globally aggregated preferences, we model human response distributions conditioned on language, interaction track, and response family\.

Distribution\-aware post\-training\.Our framework also builds on distributional views of post\-training, where preference information is incorporated directly into language\-model training\(Korbak et al\.,[2023](https://arxiv.org/html/2606.13227#bib.bib17)\)and distribution\-matching objectives are used to reduce the over\-concentration often induced by standard SFT\(Li et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib21)\)\. PolyAlign extends this perspective by targeting the appropriate human response distribution for each interaction bucket\. Bucket\-SFT performs bucket\-aware supervised learning against human reference distributions, while HDPO adds bucket\-aware weighting and a distribution\-matching regularizer to offline preference optimization\. We study this setting on compact open models such as Qwen2\.5, Gemma 2, and Llama 3\.2\. The entire pipeline is given in Fig[2](https://arxiv.org/html/2606.13227#S2.F2)\.

![Refer to caption](https://arxiv.org/html/2606.13227v1/figures/main/MainPolyalign_v2.png)Figure 2:PolyAlign pipeline\.PolyAlign organizes bilingual interaction data into bucket\-specific human distributions, then aligns models through bucket\-weighted SFT, critic\-based distribution training, and HDPO\. The final model is evaluated for task utility and conditional naturalness using QA\-F1, BNG\-Macro, G\-MAUVE, and NUF\.
## 3Methodology

### 3\.1Problem Setup and Bucketed Human Reference Distributions

Rather than learning a single global assistant behavior, the goal is to align model outputs to the human response distribution that is appropriate for the interaction setting\. For this, we partition the training data into a set of bucketsℬ\\mathcal\{B\}, where each bucket corresponds to a conditional interaction regime defined by metadata such as language, interaction track, response family, and length\.

Letbi∈ℬb\_\{i\}\\in\\mathcal\{B\}denote the bucket associated with an instanceii, andnbn\_\{b\}be the number of training instances in the bucketb∈ℬb\\in\\mathcal\{B\}\. Define the total training samples asN:=∑b∈ℬnb\.N:=\\sum\_\{b\\in\\mathcal\{B\}\}n\_\{b\}\.For each bucketbb, we estimate a bucket\-specific human reference distribution over linguistic featuresΛb\\Lambda\_\{b\}\. Letz∈ℝdz\\in\\mathbb\{R\}^\{d\}denote the feature representation of a response\. We useΛb\\Lambda\_\{b\}statistics in two places: to rebalance SFT across heterogeneous interaction regimes; and to define a critic score that measures how well a candidate response matches the empirical human support of its bucket\. Lower critic scores indicate closer alignment to the target human distribution\.

### 3\.2Bucket\-Aware Supervised Fine\-Tuning

A central difficulty in this alignment is bucket imbalance: frequent interaction regimes dominate in standard SFT, even though when the objective is to balance alignment across buckets \(Table[1](https://arxiv.org/html/2606.13227#S4.T1)\)\.

Bucket\-SFT addresses this by assigning each bucket equal optimization mass to each bucket\. Letℓi​\(θ\)\\ell\_\{i\}\(\\theta\)denote the per\-example token\-normalized loss under model parametersθ\\theta\. We assign the pre\-serialization weight for eachiiin bucketbb:w~b=N\|ℬ\|​nb\.\\widetilde\{w\}\_\{b\}=\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\.The resulting Bucket\-SFT objective is

LBucket​\-​SFT​\(θ\):=∑i=1Nw~bi​ℓi​\(θ\)∑i=1Nw~bi\.L\_\{\\mathrm\{Bucket\\text\{\-\}SFT\}\}\(\\theta\):=\\frac\{\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{b\_\{i\}\}\\,\\ell\_\{i\}\(\\theta\)\}\{\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{b\_\{i\}\}\}\.\(1\)
###### Theorem 1\(Bucket\-SFT Optimizes Exact Macro Bucket Risk\)\.

For the weights inw~b\\widetilde\{w\}\_\{b\}, the Bucket\-SFT objective in Eq\. \([1](https://arxiv.org/html/2606.13227#S3.E1)\) satisfies

LBucket​\-​SFT​\(θ\)=1\|ℬ\|​∑b∈ℬ1nb​∑i:bi=bℓi​\(θ\)\.L\_\{\\mathrm\{Bucket\\text\{\-\}SFT\}\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{b\\in\\mathcal\{B\}\}\\frac\{1\}\{n\_\{b\}\}\\sum\_\{i:\\,b\_\{i\}=b\}\\ell\_\{i\}\(\\theta\)\.\(2\)Equivalently, each bucket contributes exactly the same total optimization mass:

∑i:bi=bw~bi=N\|ℬ\|for every​b∈ℬ\.\\sum\_\{i:\\,b\_\{i\}=b\}\\widetilde\{w\}\_\{b\_\{i\}\}=\\frac\{N\}\{\|\\mathcal\{B\}\|\}\\qquad\\text\{for every \}b\\in\\mathcal\{B\}\.\(3\)

Takeaway 1Bucket\-SFT does not merely reweight examples; it exactly converts supervised learning into macro bucket risk minimization, so every bucket receives equal optimization mass, Proof[G\.1](https://arxiv.org/html/2606.13227#A7.SS1)\.

### 3\.3Learning the Human Distribution

We next define the critic target used by HDPO\.For bucketbb, the criticsϕ​\(y,b\)s\_\{\\phi\}\(y,b\)should assign low scores to responses whose feature vector stays within the bucket support regionΛb\\Lambda\_\{b\}, and higher scores to responses that deviate from it\.Letzzbe the response feature vector, and letJb​\(z\)J\_\{b\}\(z\)be the set of feature indices shared byzz, the bucket support, and the bucket feature statistics, withJb​\(z\)≠∅J\_\{b\}\(z\)\\neq\\varnothing\. For eachj∈Jb​\(z\)j\\in J\_\{b\}\(z\), let\[lb​j,ub​j\]\[l\_\{bj\},u\_\{bj\}\]be the support interval and define the normalization scalesb​j:=max⁡\{ub​j−lb​j,stdb​j,ε\},ε\>0\.s\_\{bj\}:=\\max\\\{u\_\{bj\}\-l\_\{bj\},\\,\\mathrm\{std\}\_\{bj\},\\,\\varepsilon\\\},\\qquad\\varepsilon\>0\.The normalized bucket\-support distance is then

Db​\(z\):=1\|Jb​\(z\)\|​∑j∈Jb​\(z\)\[lb​j−zj\]\+\+\[zj−ub​j\]\+sb​j\.D\_\{b\}\(z\):=\\frac\{1\}\{\|J\_\{b\}\(z\)\|\}\\sum\_\{j\\in J\_\{b\}\(z\)\}\\frac\{\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}\+\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}\}\{s\_\{bj\}\}\.\(4\)
###### Theorem 2\(Bucket\-Support Distance Is a Continuous Relaxation of Human Membership\)\.

For the distance in Eq\. \([4](https://arxiv.org/html/2606.13227#S3.E4)\), the following hold:

\(i\)​Db​\(z\)≥0,\\displaystyle\\textnormal\{\(i\)\}D\_\{b\}\(z\)\\geq 0,\(ii\)​Db​\(z\)=0⇔zj∈\[lb​j,ub​j\]​∀j∈Jb​\(z\),\\displaystyle\\textnormal\{\(ii\)\}D\_\{b\}\(z\)=0\\iff z\_\{j\}\\in\[l\_\{bj\},u\_\{bj\}\]\\ \\forall j\\in J\_\{b\}\(z\),\(iii\)​Db​\(z\)​is continuous and piecewise linear in​z\.\\displaystyle\\textnormal\{\(iii\)\}D\_\{b\}\(z\)\\ \\text\{is continuous and piecewise linear in \}z\.\(5\)

Takeaway 2Bucket\-Support DistanceDb​\(z\)=0D\_\{b\}\(z\)=0when all usable active features lie insideΛb\\Lambda\_\{b\}, and increases as the response moves farther outside the bucket support, Proof[G\.2](https://arxiv.org/html/2606.13227#A7.SS2)\.

### 3\.4Human\-Distribution Preference Optimization \(HDPO\)

We now incorporate bucket\-aware human\-distribution information into preference optimization\. Consider a training triple\(x,y\+,y−\)\(x,y^\{\+\},y^\{\-\}\), wherey\+y^\{\+\}is the chosen response,y−y^\{\-\}is the rejected response, and both belong to bucketbb\. HDPO augments a standard sigmoid\-DPO objective with a critic\-based regularizer that favors responses that are both preferred and closer to the corresponding human bucket supportΛb\\Lambda\_\{b\}\.

Define the policy log\-probability marginΔπ\\Delta\_\{\\pi\}, and the reference\-model marginΔref\\Delta\_\{\\mathrm\{ref\}\}\.

Δπ:=log⁡πθ​\(y\+∣x\)−log⁡πθ​\(y−∣x\),\\Delta\_\{\\pi\}:=\\log\\pi\_\{\\theta\}\(y^\{\+\}\\mid x\)\-\\log\\pi\_\{\\theta\}\(y^\{\-\}\\mid x\),\(6\)Δref:=log⁡πref​\(y\+∣x\)−log⁡πref​\(y−∣x\)\.\\Delta\_\{\\mathrm\{ref\}\}:=\\log\\pi\_\{\\mathrm\{ref\}\}\(y^\{\+\}\\mid x\)\-\\log\\pi\_\{\\mathrm\{ref\}\}\(y^\{\-\}\\mid x\)\.\(7\)
The sigmoid\-DPO loss is

LDPO​\(θ\)=−log⁡σ​\(β​\(Δπ−Δref\)\),L\_\{\\mathrm\{DPO\}\}\(\\theta\)=\-\\log\\sigma\\\!\\Big\(\\beta\(\\Delta\_\{\\pi\}\-\\Delta\_\{\\mathrm\{ref\}\}\)\\Big\),\(8\)whereβ\>0\\beta\>0is the inverse\-temperature parameter\. To inject distributional information, we definepθ:=σ​\(β​Δπ\),p\_\{\\theta\}:=\\sigma\(\\beta\\Delta\_\{\\pi\}\),and use the critic scoressϕ​\(y\+,b\)s\_\{\\phi\}\(y^\{\+\},b\)andsϕ​\(y−,b\)s\_\{\\phi\}\(y^\{\-\},b\)to form the HDPO regularizer

RHDPO​\(θ\):=pθ​sϕ​\(y\+,b\)\+\(1−pθ\)​sϕ​\(y−,b\)\.R\_\{\\mathrm\{HDPO\}\}\(\\theta\):=p\_\{\\theta\}\\,s\_\{\\phi\}\(y^\{\+\},b\)\+\(1\-p\_\{\\theta\}\)\\,s\_\{\\phi\}\(y^\{\-\},b\)\.\(9\)
This regularizer is small when the policy places probability mass on responses that the critic judges to be closer to the target human bucket support\. The full HDPO objective then combines preference learning with critic\-guided distribution matching:

LHDPO​\(θ\)=1K​∑i=1Kwi​\(LDPO\(i\)​\(θ\)\+λhd​RHDPO\(i\)​\(θ\)\),L\_\{\\mathrm\{HDPO\}\}\(\\theta\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}w\_\{i\}\\Big\(L\_\{\\mathrm\{DPO\}\}^\{\(i\)\}\(\\theta\)\+\\lambda\_\{\\mathrm\{hd\}\}\\,R\_\{\\mathrm\{HDPO\}\}^\{\(i\)\}\(\\theta\)\\Big\),\(10\)whereKKis the number of preference pairs,λhd≥0\\lambda\_\{\\mathrm\{hd\}\}\\geq 0controls the strength of the distributional regularizer, andwi\>0w\_\{i\}\>0denotes an optional bucket\-aware training weight\.

###### Theorem 3\(Distributional Alignment of the HDPO Regularizer with Sigmoid DPO\)\.

Consider a single chosen/rejected pair\(y\+,y−\)\(y^\{\+\},y^\{\-\}\)in bucketbb\. Let

Δπ:=log⁡πθ​\(y\+∣x\)−log⁡πθ​\(y−∣x\),pθ:=sigmoid⁡\(β​Δπ\)\\displaystyle\\Delta\_\{\\pi\}=\\log\\pi\_\{\\theta\}\(y^\{\+\}\\mid x\)\-\\log\\pi\_\{\\theta\}\(y^\{\-\}\\mid x\),\\ p\_\{\\theta\}=\\operatorname\{sigmoid\}\(\\beta\\Delta\_\{\\pi\}\)and letsϕ​\(y\+,b\)s\_\{\\phi\}\(y^\{\+\},b\)andsϕ​\(y−,b\)s\_\{\\phi\}\(y^\{\-\},b\)be fixed critic scores\. For the regularizer in Eq\. \([9](https://arxiv.org/html/2606.13227#S3.E9)\),

d​RHDPOd​Δπ=β​pθ​\(1−pθ\)​\(sϕ​\(y\+,b\)−sϕ​\(y−,b\)\)\.\\frac\{dR\_\{\\mathrm\{HDPO\}\}\}\{d\\Delta\_\{\\pi\}\}=\\beta\\,p\_\{\\theta\}\(1\-p\_\{\\theta\}\)\\Big\(s\_\{\\phi\}\(y^\{\+\},b\)\-s\_\{\\phi\}\(y^\{\-\},b\)\\Big\)\.\(11\)Therefore, ifsϕ​\(y\+,b\)<sϕ​\(y−,b\),s\_\{\\phi\}\(y^\{\+\},b\)<s\_\{\\phi\}\(y^\{\-\},b\),then minimizingRHDPOR\_\{\\mathrm\{HDPO\}\}strictly favors increasing the chosen\-vs\-rejected marginΔπ\\Delta\_\{\\pi\}\. Furthermore, for the sigmoid\-DPO loss in Eq\. \([8](https://arxiv.org/html/2606.13227#S3.E8)\),

d​LDPOd​Δπ=−β​\(1−σ​\(β​\(Δπ−Δref\)\)\)<0\.\\frac\{dL\_\{\\mathrm\{DPO\}\}\}\{d\\Delta\_\{\\pi\}\}=\-\\beta\\Big\(1\-\\sigma\\big\(\\beta\(\\Delta\_\{\\pi\}\-\\Delta\_\{\\mathrm\{ref\}\}\)\\big\)\\Big\)<0\.\(12\)Hence, whenever the critic judges the chosen response to be closer to the human bucket support than the rejected response, the sigmoid\-DPO term and the HDPO regularizer are distributionally aligned: both push the policy toward largerΔπ\\Delta\_\{\\pi\}\.

Takeaway 3Under the sigmoid\-DPO variant used in PolyAlign, HDPO is locally distributionally aligned with preference learning: when the chosen response is distributionally better, both objectives push the policy toward it, Proof[G\.3](https://arxiv.org/html/2606.13227#A7.SS3)\.

## 4Experiments

### 4\.1Setup

#### Benchmark Construction\.

PolyAlign is studied in a bilingual setting that spans English and Chinese\. Our goal is to align model outputs to the appropriate human\-linguistic response distribution for the interaction setting at hand\. We organize the corpus into five interaction situations:assistant\_like,longform\_qa,open\_chat,qa\_search, andtask\_dialogue\. Each instance carries metadata for language, interaction track, response family, length bin and linguistic reference bucket, which together define the conditional regimes used throughout training and evaluation\. All instances are canonicalized and deduplicated after normalization to prevent train\-test leakage\. More details in Appendix[A](https://arxiv.org/html/2606.13227#A1)\.

DatasetLang\.Situation CategoryTrackSplit CountsCountTotalDollyenassistant\_likesingle13,525 / 713 / 75714,995584,422 / 55,575 /53,855 =693,852ELI5enlongform\_qasingle91,772 / 5,446 / 7,786105,004DailyDialogenopen\_chatmulti37,377 / 3,774 / 3,68144,832MS MARCOenqa\_searchsingle80,143 / 9,754 / 9,39999,296CoQAenqa\_searchmulti98,015 / 10,620 / 7,983116,618SQuAD v2enqa\_searchsingle117,444 / 12,832 / 11,870142,146Natural Questionsenqa\_searchsingle90,133 / 5,081 / 5,017100,231MultiWOZentask\_dialoguemulti56,013 / 7,355 / 7,36270,730COIG\-CQIAzhassistant\_likesingle9,536 / 509 / 57910,62486,336 / 10,887 /6,357 =103,580HC3\-Chinesezhlongform\_qasingle19,990 / 1,152 / 1,05822,200OASST2\-zhzhopen\_chatmulti3,809 / 527 / 2254,561CMRC2018zhqa\_searchsingle10,142 / 3,219 / 1,00214,363DRCDzhqa\_searchsingle26,936 / 3,524 / 3,49333,953DuReaderzhqa\_searchsingle15,923 / 1,956 / –17,879

Table 1:Bilingual PolyAlign corpus inventory with realized split counts shown as train / val / test\.
#### Models\.

We conduct experiments on Qwen2\.5\-1\.5B, Qwen2\.5\-3B\(Qwen Team et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib24)\), Gemma2\-2BGemma Team et al\. \([2024](https://arxiv.org/html/2606.13227#bib.bib10)\), Llama\-3\.2\-3B\(Grattafiori et al\.,[2024](https://arxiv.org/html/2606.13227#bib.bib11)\), to examine performance trends with increasing model scale, spanning approximately 1\.5B to 3B parameters\.

#### Baselines\.

We perform the experiments by considering the baselines as Normal Inference \(BaseLM\), Chain\-of\-thoughts Prompting \(CoT\), Full Supervised Fine\-Tuning of the models \(Full\-SFT\), and Direct Preference Optimization \(DPO\)\. More details on the baselines are given in the Appendix[B](https://arxiv.org/html/2606.13227#A2)\. Details of the compute is given in Appendix[C](https://arxiv.org/html/2606.13227#A3)\.

### 4\.2Evaluation

We evaluate PolyAlign along two axes:*task utility*and*conditional naturalness*to measure whether a response is useful for the user request and whether it matches the appropriate human response distribution for the current interaction situation\.

#### Metrics\.

For theTask\-Utility metrics, we report: exact match \(EM\), normalized exact match \(nEM\), token\-level F1, and ROUGE\-L\. For theConditional Naturalness metrics, we report: Bucketed Naturalness Gap \(BNG\), Conditional MAUVE, Naturalness\-Utility Frontier, and LLM\-as\-a\-Judge rubric scoring\. Further detailed information about the metrics is given in Appendix[D](https://arxiv.org/html/2606.13227#A4)\.

#### LLM\-Judge evaluation\.

We evaluate generations with an LLM\-as\-a\-judge protocolZheng et al\. \([2023](https://arxiv.org/html/2606.13227#bib.bib36)\)to measure whether model responses preserve task utility while matching the target human\-response distribution\. For each candidate, the judge is given the user request, dialogue history, available context, bucket metadata, the human reference response, and the model response; human responses are used only as references and are not scored as candidates\. We use Qwen2\.5\-7B\-InstructQwen Team et al\. \([2024](https://arxiv.org/html/2606.13227#bib.bib24)\)and Qwen3\-8BYang et al\. \([2025](https://arxiv.org/html/2606.13227#bib.bib32)\)as judges\. Each response is assigned integer scores from 1 to 5 across eight dimensions: task success, factual grounding, instruction following, reference alignment, conditional appropriateness, response shape and length, discourse naturalness, and safety, where 1 is severe failure, 3 acceptable quality, and 5 excellent quality\. We map each scoressto a 0–100 scale using100​\(s−1\)/4100\(s\-1\)/4and report weighted composite scores\. The Overall score uses weights\(0\.20,0\.15,0\.15,0\.10,0\.17,0\.08,0\.10,0\.05\)\(0\.20,0\.15,0\.15,0\.10,0\.17,0\.08,0\.10,0\.05\)over the eight dimensions; Utility emphasizes task correctness and grounding, Conditional Naturalness emphasizes bucket\-appropriate style and discourse quality, and Distribution Faithfulness measures alignment with the target language, response family, style bucket, and length bin\. The prompt and rubric is given in[H](https://arxiv.org/html/2606.13227#A8)\.

ModelMethodEnglish \- enChinese \- zhQA\-F1 \(↑\\uparrow\)BNG\-macro \(↓\\downarrow\)G\-MAUVE \(↑\\uparrow\)NUF \(↑\\uparrow\)Agg \(↑\\uparrow\)QA\-F1 \(↑\\uparrow\)BNG\-macro \(↓\\downarrow\)G\-MAUVE \(↑\\uparrow\)NUF \(↑\\uparrow\)Agg \(↑\\uparrow\)BaseLM0\.2486\.3440\.5310\.1700\.2350\.1952\.4660\.5860\.2250\.293CoT0\.0992\.5920\.3070\.2220\.2080\.0221\.6850\.2890\.0800\.117Full\-SFT0\.3175\.0120\.9470\.5470\.3710\.3580\.9170\.9180\.4580\.529DPO0\.3551\.2590\.8470\.5800\.5270\.23816\.0560\.5110\.1630\.184\\rowcolorbucketrowBucket\-SFT0\.4440\.4270\.9390\.5850\.6430\.4790\.3460\.9680\.6360\.684\\rowcolorbucketrowQwen2\.5\-1\.5BHDPO0\.4630\.4650\.8460\.6230\.6390\.4500\.3750\.8510\.8320\.694BaseLM0\.1947708\.60\.7840\.2700\.0480\.10813\.1520\.3640\.0590\.113CoT0\.0574\.8760\.1350\.0660\.0960\.0101\.9230\.3210\.0250\.072Full\-SFT0\.308411\.770\.9110\.5340\.1380\.3521\.4390\.9460\.4180\.489DPO0\.31028526\.00\.8740\.3650\.0430\.08136\.7500\.3420\.0160\.058\\rowcolorbucketrowBucket\-SFT0\.3870\.3830\.8710\.5160\.5950\.32439\.3650\.9670\.4430\.242\\rowcolorbucketrowGemma\-2\-2BHDPO0\.5410\.2750\.7880\.8530\.7310\.55523\.4940\.8520\.1880\.245BaseLM0\.2004\.8470\.5580\.3090\.2770\.2022\.4790\.6400\.5250\.374CoT0\.1012\.2910\.2280\.2130\.1970\.0531\.2150\.3220\.0660\.150Full\-SFT0\.398649\.490\.9180\.6440\.1370\.4472\.9320\.9490\.5740\.498DPO0\.2674\.0140\.7570\.5230\.3810\.32337\.5300\.7370\.0790\.148\\rowcolorbucketrowBucket\-SFT0\.4760\.2500\.9030\.6520\.6880\.5020\.5840\.9640\.6560\.669\\rowcolorbucketrowQwen2\.5\-3BHDPO0\.4180\.4900\.7790\.4740\.5670\.44923\.5370\.7480\.1820\.223BaseLM0\.2279\.7730\.8080\.3520\.2780\.29718\.230\.7250\.3420\.249CoT0\.1002\.4300\.3150\.1860\.2030\.0114\.4310\.2550\.0160\.055Full\-SFT0\.3441\.1390\.9060\.6270\.5500\.2650\.9320\.8790\.2920\.433DPO0\.35624\.4310\.7860\.6320\.2880\.10331\.8430\.5520\.0340\.088\\rowcolorbucketrowBucket\-SFT0\.3720\.3730\.8670\.4390\.5660\.4600\.6330\.9480\.6130\.636\\rowcolorbucketrowLlama\-3\.2\-3BHDPO0\.4050\.3600\.7500\.6000\.6040\.4490\.4280\.7070\.8770\.665

Table 2:Main results across bilingual benchmarks\.

## 5Results and Analysis

PolyAlign improves both task utility and distributional alignment over BaseLM, CoT, Full\-SFT, and standard DPO, given in Table[2](https://arxiv.org/html/2606.13227#S4.T2)\. Bucket\-SFT gives a robust first\-stage alignment method, consistently providing strong gains\. HDPO refines this alignment by optimizing against critic\-derived distributional preferences, giving additional improvements in model\-language combinations and demonstrating the value of preference optimization over bucket\-conditioned human support\.

#### Bucket\-conditioned supervision moves small models toward the human distribution\.

Bucket\-SFT provides a much more reliable form of post\-training because it does not collapse all human responses into a single global target\. From Table[2](https://arxiv.org/html/2606.13227#S4.T2), Qwen2\.5\-1\.5B in English, Bucket\-SFT improves score from 0\.371 under Full\-SFT to 0\.643 and reduces BNG\-macro from 5\.012 to 0\.427\. The same pattern appears in Chinese, where Agg increases from 0\.529 to 0\.684 and BNG\-macro drops from 0\.917 to 0\.346, and the similar trends were being followed in other models\. Bucket\-SFT improves distributional alignment by lowering BNG\-macro while preserving strong G\-MAUVE and NUF\. This indicates that even small models can move closer to human linguistic support when supervision is explicitly human\-distribution\-aware\.

#### Explicit human\-support conditioning is more stable than global preference optimization\.

Preference optimization alone is not sufficient when the preference signal is not explicitly grounded in the conditional human distribution\. DPO often improves task behavior in isolated cases, but it can drift away from the desired human\-like response support\. For Qwen2\.5\-1\.5B in English, DPO reaches Agg 0\.527, Bucket\-SFT improves it to 0\.643 and lowers BNG\-macro from 1\.259 to 0\.427\. In Chinese, the gap is larger: DPO obtains Agg 0\.184 with BNG\-macro 16\.056, whereas Bucket\-SFT reaches Agg 0\.684 with BNG\-macro 0\.346\. Similar failures appear for Gemma\-2\-2B and Llama\-3\.2\-3B\. The gain comes from fine\-tuning in the right direction: toward bucket\-specific human response regions\. Bucket\-SFT anchors the model to human naturalness, style, and interaction\-specific before preference optimization\.

ModelMethodEnglishChineseQwen3\-8BQwen2\.5\-7BAvg\.Qwen3\-8BQwen2\.5\-7BAvg\.Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Overall\(↑\\uparrow\)Utility\(↑\\uparrow\)Cond\.\(↑\\uparrow\)D\-Faith\.\(↑\\uparrow\)Qwen2\.5\-1\.5BBaseLM73\.769\.577\.173\.567\.463\.270\.767\.870\.666\.473\.970\.667\.164\.669\.166\.163\.360\.965\.063\.365\.262\.867\.064\.7Full\-SFT76\.171\.280\.676\.966\.262\.269\.666\.671\.166\.775\.171\.784\.682\.286\.584\.480\.477\.782\.880\.982\.579\.984\.782\.6DPO76\.071\.180\.576\.864\.360\.267\.764\.870\.265\.774\.170\.865\.263\.366\.764\.159\.957\.761\.660\.162\.560\.564\.262\.1\\rowcolorbucketrowBucket\-SFT78\.374\.082\.178\.967\.163\.370\.367\.672\.768\.776\.273\.385\.082\.786\.984\.880\.377\.882\.680\.782\.780\.284\.882\.8\\rowcolorbucketrowHDPO43\.740\.743\.839\.035\.031\.237\.033\.339\.336\.040\.436\.152\.050\.549\.745\.646\.541\.450\.246\.249\.345\.950\.045\.9Gemma\-2\-2BBaseLM65\.057\.971\.766\.254\.349\.758\.154\.359\.653\.864\.960\.255\.352\.557\.553\.752\.149\.654\.051\.353\.751\.055\.852\.5Full\-SFT72\.667\.777\.173\.263\.659\.766\.963\.968\.163\.772\.068\.579\.576\.482\.279\.474\.171\.176\.774\.576\.873\.779\.476\.9DPO69\.264\.873\.269\.459\.055\.561\.959\.164\.160\.167\.564\.247\.047\.845\.342\.845\.043\.945\.443\.646\.045\.845\.443\.2\\rowcolorbucketrowBucket\-SFT73\.168\.277\.874\.062\.458\.665\.762\.967\.863\.471\.868\.475\.771\.979\.376\.168\.765\.771\.469\.172\.268\.875\.472\.6\\rowcolorbucketrowHDPO49\.245\.350\.045\.037\.933\.440\.636\.743\.639\.345\.340\.861\.759\.960\.156\.453\.748\.157\.954\.157\.754\.059\.055\.3Qwen2\.5\-3BBaseLM39\.637\.041\.737\.147\.044\.648\.346\.443\.340\.845\.041\.862\.060\.163\.460\.459\.757\.860\.959\.660\.859\.062\.160\.0Full\-SFT80\.976\.684\.681\.569\.765\.573\.270\.575\.371\.178\.976\.086\.985\.088\.286\.482\.880\.285\.083\.284\.882\.686\.684\.8DPO69\.265\.672\.368\.959\.056\.161\.258\.764\.160\.866\.763\.874\.873\.775\.573\.769\.667\.571\.269\.772\.270\.673\.371\.7\\rowcolorbucketrowBucket\-SFT77\.873\.981\.278\.167\.363\.770\.467\.872\.668\.875\.873\.086\.784\.888\.386\.582\.480\.084\.682\.984\.682\.486\.484\.7\\rowcolorbucketrowHDPO47\.343\.448\.343\.335\.631\.537\.934\.141\.537\.543\.138\.749\.448\.246\.742\.544\.039\.347\.143\.246\.743\.746\.942\.9Llama\-3\.2\-3BBaseLM57\.453\.061\.356\.854\.150\.556\.854\.155\.851\.759\.055\.471\.569\.173\.470\.765\.063\.066\.764\.968\.366\.170\.167\.8Full\-SFT80\.775\.585\.281\.569\.264\.573\.270\.174\.970\.079\.275\.879\.776\.682\.379\.575\.172\.077\.975\.577\.474\.380\.177\.5DPO74\.870\.678\.575\.163\.960\.366\.964\.269\.365\.572\.769\.750\.949\.552\.148\.945\.644\.446\.244\.648\.346\.949\.146\.7\\rowcolorbucketrowBucket\-SFT78\.874\.083\.379\.866\.662\.470\.267\.372\.768\.276\.773\.681\.979\.484\.181\.877\.074\.479\.477\.479\.576\.981\.879\.6\\rowcolorbucketrowHDPO46\.543\.247\.142\.437\.534\.837\.233\.542\.039\.042\.238\.051\.950\.649\.545\.545\.641\.048\.744\.948\.845\.849\.145\.2

Table 3:Composite LLM\-as\-a\-judge results\. Scores are 0–100 rubric composites; higher is better\. Cond\. denotes Conditional Naturalness and D\-Faith\. denotes Distribution Faithfulness\. Avg\. is the mean across available judges\. Best and second\-best Avg\. scores within each model, language, and metric are bolded and underlined, respectively\.ModelMethodEnglishChineseAgg\(↑\\uparrow\)Binoc\.AI% \(↓\\downarrow\)Fast\-DGPTAI% \(↓\\downarrow\)RADARAI% \(↓\\downarrow\)Joint\(↑\\uparrow\)Agg\(↑\\uparrow\)Binoc\.AI% \(↓\\downarrow\)Fast\-DGPTAI% \(↓\\downarrow\)RADARAI% \(↓\\downarrow\)Joint\(↑\\uparrow\)BaseLM0\.23562\.244\.598\.527\.00\.29328\.127\.258\.139\.8Full\-SFT0\.37123\.612\.885\.445\.70\.52924\.126\.449\.858\.9DPO0\.52753\.627\.196\.846\.00\.18440\.838\.359\.927\.4\\rowcolorbucketrowBucket\-SFT0\.64338\.115\.193\.257\.00\.68425\.528\.460\.065\.1\\rowcolorbucketrowQwen2\.5\-1\.5BHDPO0\.63947\.226\.196\.051\.80\.69437\.836\.735\.366\.3BaseLM0\.04855\.335\.496\.28\.50\.11335\.335\.260\.018\.8Full\-SFT0\.13825\.814\.484\.922\.30\.48922\.727\.554\.755\.8DPO0\.04354\.331\.095\.77\.80\.05837\.043\.934\.910\.6\\rowcolorbucketrowBucket\-SFT0\.59523\.87\.489\.159\.70\.24219\.325\.057\.935\.4\\rowcolorbucketrowGemma\-2\-2BHDPO0\.73142\.719\.795\.857\.40\.24540\.839\.035\.635\.0BaseLM0\.27732\.917\.999\.935\.60\.37427\.524\.064\.746\.4Full\-SFT0\.13741\.816\.595\.421\.40\.49823\.525\.256\.656\.4DPO0\.38160\.043\.194\.935\.90\.14835\.033\.748\.323\.8\\rowcolorbucketrowBucket\-SFT0\.68839\.316\.693\.258\.10\.66924\.927\.459\.964\.7\\rowcolorbucketrowQwen2\.5\-3BHDPO56\.742\.524\.896\.450\.40\.22334\.934\.032\.633\.4BaseLM0\.27845\.020\.699\.534\.40\.24925\.717\.570\.535\.5Full\-SFT0\.55049\.321\.598\.548\.60\.43332\.534\.243\.851\.4DPO0\.28851\.526\.999\.033\.80\.08838\.346\.646\.915\.2\\rowcolorbucketrowBucket\-SFT0\.56644\.718\.094\.151\.80\.63631\.831\.761\.160\.9\\rowcolorbucketrowLlama\-3\.2\-3BHDPO0\.60450\.023\.597\.050\.30\.66550\.850\.127\.561\.5

Table 4:AI\-detection and joint quality\-human\-likeness results\. Detector scores are AI\-classification rates, where lower is better\. Joint is the harmonic mean of Agg and the mean detector human\-likeness score1−AI​rate1\-\\mathrm\{AI\\ rate\}, scaled to 0\-100\.
#### Preference optimization further refines human\-distribution alignment\.

HDPO builds on Bucket\-SFT by using critic\-derived preferences to further optimize toward responses that better match the estimated human distribution\. It improves over Bucket\-SFT for Gemma\-2\-2B in English, increasing from 0\.595 to 0\.731 and reducing BNG\-macro from 0\.383 to 0\.275\. For Qwen2\.5\-1\.5B in Chinese, HDPO slightly improves Agg from 0\.684 to 0\.694\. These results indicate that HDPO can help when the critic signal is reliable\. HDPO is especially effective on NUF, improving Qwen2\.5\-1\.5B Chinese from 0\.636 to 0\.832 and Llama\-3\.2\-3B Chinese from 0\.613 to 0\.877\. This shows that critic\-guided optimization can further improve naturalness and human\-like patterns\.

#### Human\-likeness beyond benchmark scores and Cross\-lingual implication\.

Table[4](https://arxiv.org/html/2606.13227#S5.T4)shows that PolyAlign improves both benchmark utility and human\-likeness, measured with three AI\-text detectors: BinocularsHans et al\. \([2024](https://arxiv.org/html/2606.13227#bib.bib12)\), Fast\-DetectGPTBao et al\. \([2024](https://arxiv.org/html/2606.13227#bib.bib3)\), and RADARHu et al\. \([2023](https://arxiv.org/html/2606.13227#bib.bib14)\)\. We summarize detectability withH=1−13​∑drdH=1\-\\frac\{1\}\{3\}\\sum\_\{d\}r\_\{d\}, whererdr\_\{d\}is the predicted\-AI rate, and combine it with benchmark scoreAAusingJoint=100⋅2​A​HA\+H\\mathrm\{Joint\}=100\\cdot\\frac\{2AH\}\{A\+H\}\. In English, Bucket\-SFT yields the strongest joint gains, improving Qwen2\.5\-1\.5B from 45\.7/46\.0 under Full\-SFT/DPO to 57\.0, while Gemma\-2\-2B reaches 59\.7\. In Chinese, HDPO further improves human\-likeness, raising Qwen2\.5\-1\.5B from 65\.1 to 66\.3 and Llama\-3\.2\-3B from 60\.9 to 61\.5\. Overall, PolyAlign is not English\-specific: Bucket\-SFT anchors models to human distributions, while HDPO refines language\-specific naturalness\. The Human\-Likeliness and the benchmark quality performance can be seen visually in Fig[3](https://arxiv.org/html/2606.13227#S5.F3)\. A clear\-view of UMAP projections in Fig[4](https://arxiv.org/html/2606.13227#S5.F4)of Qwen2\.5\-1\.5B’s responses gives it’s closeness to human\-responses showcasing that HDPO is clearly the best alignment rather than the DPO\.

![Refer to caption](https://arxiv.org/html/2606.13227v1/x1.png)

\(a\)BaseLM, Full\-SFT, and Bucket\-SFT\.

![Refer to caption](https://arxiv.org/html/2606.13227v1/x2.png)

\(b\)BaseLM, DPO, and HDPO\.

Figure 3:Combined benchmark quality and detector human\-likeness scores across English and Chinese models\. Bars show model\-level scores, the black line reports the method mean, and the highlighted region marks the distribution\-conditioned method in each comparison\.![Refer to caption](https://arxiv.org/html/2606.13227v1/x3.png)

\(a\) English answer embedding shifts\.![Refer to caption](https://arxiv.org/html/2606.13227v1/x4.png)

\(b\) Chinese answer embedding shifts\.Figure 4:UMAP projections of Qwen2\.5\-1\.5B answer embeddings across training methods\. Each panel compares two response distributions, with stars denoting centroids and gray lines showing paired sample shifts; smaller centroid distances indicate closer embedding\-level alignment\.
### 5\.1Human Evaluation

To complement the automatic metrics, AI\-detection analysis, and LLM\-as\-a\-judge evaluation, we conduct a focused human evaluation on held\-out English and Chinese examples\. The evaluation subsets are sampled from the same benchmark sources used in the main experiments and cover extractive QA, context\-grounded long\-form QA, and assistant\-style instruction following\. For each model method pair, we sample responses across response\-length bins, preserving the PolyAlign metadata used throughout the paper, including language, track, response family, length bin, and bucket identifier\. Human evaluators score each candidate response on three 1–5 dimensions:Usefulness, measuring whether the response satisfies the user request;Answerability, measuring whether the response is supported by the provided context or reference answer; andNaturalness, measuring fluency, idiomaticity, and appropriateness for the target interaction bucket\. We report average scores across completed annotations and use this evaluation as a human validation of the quality and conditional\-naturalness trends observed in the automatic and LLM\-judge evaluations\. More details are given in Appendix[E](https://arxiv.org/html/2606.13227#A5)\.

### 5\.2LLM\-as\-a\-Judge Evaluation

Table[3](https://arxiv.org/html/2606.13227#S5.T3)reports rubric\-based judge scores for Overall quality, Utility, Conditional Naturalness, and Distribution Faithfulness\. These results generally agree with the automatic metrics\. Bucket\-SFT remains competitive with Full\-SFT on utility\-oriented scores while improving dimensions related to conditional appropriateness and distribution faithfulness\. For Qwen2\.5\-1\.5B in English, Bucket\-SFT improves the average Overall score from 71\.1 under Full\-SFT to 72\.7 and improves Distribution Faithfulness from 71\.7 to 73\.3\. In Chinese, Bucket\-SFT also slightly improves the average Overall score over Full\-SFT, from 82\.5 to 82\.7\.These judge results support the main finding from Table[2](https://arxiv.org/html/2606.13227#S4.T2): Bucket\-SFT improves conditional alignment without sacrificing task quality\. Since the gains are moderate, we treat this evaluation as supporting evidence rather than the primary result\.

## 6Conclusion

We introduced PolyAlign, a framework for conditional human\-distribution alignment that moves beyond a single global assistant style by aligning model responses with bucket\-specific human distributions defined by language, interaction setting, response family, and length\. Our results show that Bucket\-SFT is the most reliable component, improving the balance between task utility and conditional naturalness through macro bucket\-risk optimization\. HDPO can further improve alignment when the critic signal is well calibrated, although it is more sensitive to bucket quality and model behavior\. Overall, PolyAlign shows that explicitly modeling interaction\-specific human response patterns leads to more natural, situation\-appropriate, and distribution\-faithful generations than standard Full\-SFT and DPO alone\. Future work will extend this approach to broader multilingual settings, richer bucket representations, larger models, and more extensive human evaluation\.

## Limitations

This work is an initial study of conditional human\-distribution alignment\. We focus on English and Chinese with compact open models, leaving broader multilingual, low\-resource, and larger\-scale evaluation for future work\. PolyAlign also depends on bucket construction and linguistic feature profiles, which future work can strengthen using richer metadata and learned representations\. Finally, although our metrics evaluate both utility and conditional naturalness, future work will expand human evaluation and safety analysis to better capture tone, cultural appropriateness, robustness, and fairness\.

## References

- Bai et al\. \(2022a\)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others\. 2022a\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.*arXiv preprint arXiv:2204\.05862*\.
- Bai et al\. \(2022b\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others\. 2022b\.Constitutional AI: Harmlessness from AI feedback\.*arXiv preprint arXiv:2212\.08073*\.
- Bao et al\. \(2024\)Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang\. 2024\.Fast\-DetectGPT: Efficient zero\-shot detection of machine\-generated text via conditional probability curvature\.In*International Conference on Learning Representations*\.
- Brown et al\. \(2020\)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D\. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others\. 2020\.Language models are few\-shot learners\.*Advances in Neural Information Processing Systems*, 33:1877–1901\.
- Christiano et al\. \(2017\)Paul F\. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei\. 2017\.Deep reinforcement learning from human preferences\.*Advances in Neural Information Processing Systems*, 30\.
- Chung et al\. \(2024\)Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others\. 2024\.Scaling instruction\-finetuned language models\.*Journal of Machine Learning Research*, 25\(70\):1–53\.
- Dathathri et al\. \(2020\)Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu\. 2020\.Plug and play language models: A simple approach to controlled text generation\.In*International Conference on Learning Representations*\.
- Dong et al\. \(2023\)Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev\. 2023\.SteerLM: Attribute conditioned SFT as an user\-steerable alternative to RLHF\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 11275–11288\.
- Ethayarajh et al\. \(2024\)Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela\. 2024\.KTO: Model alignment as prospect theoretic optimization\.*arXiv preprint arXiv:2402\.01306*\.
- Gemma Team et al\. \(2024\)Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others\. 2024\.Gemma 2: Improving open language models at a practical size\.*arXiv preprint arXiv:2408\.00118*\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others\. 2024\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*\.
- Hans et al\. \(2024\)Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein\. 2024\.Spotting LLMs with binoculars: Zero\-shot detection of machine\-generated text\.*arXiv preprint arXiv:2401\.12070*\.
- Hong et al\. \(2024\)Jiwoo Hong, Noah Lee, and James Thorne\. 2024\.ORPO: Monolithic preference optimization without reference model\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 11170–11189\.
- Hu et al\. \(2023\)Xiaomeng Hu, Pin\-Yu Chen, and Tsung\-Yi Ho\. 2023\.RADAR: Robust AI\-text detection via adversarial learning\.*Advances in Neural Information Processing Systems*, 36:15077–15095\.
- Keskar et al\. \(2019\)Nitish Shirish Keskar, Bryan McCann, Lav R\. Varshney, Caiming Xiong, and Richard Socher\. 2019\.CTRL: A conditional transformer language model for controllable generation\.*arXiv preprint arXiv:1909\.05858*\.
- Köpf et al\. \(2023\)Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, and 1 others\. 2023\.OpenAssistant conversations: Democratizing large language model alignment\.*Advances in Neural Information Processing Systems*, 36:47669–47681\.
- Korbak et al\. \(2023\)Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R\. Bowman, and Ethan Perez\. 2023\.Pretraining language models with human preferences\.In*International Conference on Machine Learning*, pages 17506–17533\. PMLR\.
- Krause et al\. \(2021\)Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani\. 2021\.GeDi: Generative discriminator guided sequence generation\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 4929–4952\.
- Lester et al\. \(2021\)Brian Lester, Rami Al\-Rfou, and Noah Constant\. 2021\.The power of scale for parameter\-efficient prompt tuning\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 3045–3059\.
- Li and Liang \(2021\)Xiang Lisa Li and Percy Liang\. 2021\.Prefix\-tuning: Optimizing continuous prompts for generation\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 4582–4597\.
- Li et al\. \(2024\)Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi\-Quan Luo\. 2024\.Entropic distribution matching in supervised fine\-tuning of LLMs: Less overfitting and better diversity\.In*NeurIPS 2024 Workshop on Fine\-Tuning in Modern Machine Learning: Principles and Scalability*\.
- Meng et al\. \(2024\)Yu Meng, Mengzhou Xia, and Danqi Chen\. 2024\.SimPO: Simple preference optimization with a reference\-free reward\.*Advances in Neural Information Processing Systems*, 37:124198–124235\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others\. 2022\.Training language models to follow instructions with human feedback\.*Advances in Neural Information Processing Systems*, 35:27730–27744\.
- Qwen Team et al\. \(2024\)Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others\. 2024\.[Qwen2\.5 technical report](https://arxiv.org/abs/2412.15115)\.*Preprint*, arXiv:2412\.15115\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\. 2023\.Direct preference optimization: Your language model is secretly a reward model\.*Advances in Neural Information Processing Systems*, 36:53728–53741\.
- Sanh et al\. \(2022\)Victor Sanh, Albert Webson, Colin Raffel, Stephen H\. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others\. 2022\.Multitask prompted training enables zero\-shot task generalization\.In*International Conference on Learning Representations*\.
- Stiennon et al\. \(2020\)Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F\. Christiano\. 2020\.Learning to summarize with human feedback\.*Advances in Neural Information Processing Systems*, 33:3008–3021\.
- Wang et al\. \(2023\)Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A\. Smith, Daniel Khashabi, and Hannaneh Hajishirzi\. 2023\.Self\-instruct: Aligning language models with self\-generated instructions\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 13484–13508\.
- Wang et al\. \(2024a\)Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J\. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev\. 2024a\.HelpSteer2: Open\-source dataset for training top\-performing reward models\.*arXiv preprint arXiv:2406\.08673*\.
- Wang et al\. \(2024b\)Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and 1 others\. 2024b\.HelpSteer: Multi\-attribute helpfulness dataset for SteerLM\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 3371–3384\.
- Wei et al\. \(2022\)Jason Wei, Maarten Bosma, Vincent Y\. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M\. Dai, and Quoc V\. Le\. 2022\.Finetuned language models are zero\-shot learners\.In*International Conference on Learning Representations*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yang and Klein \(2021\)Kevin Yang and Dan Klein\. 2021\.FUDGE: Controlled text generation with future discriminators\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3511–3535\.
- Yuan et al\. \(2023\)Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang\. 2023\.RRHF: Rank responses to align language models with human feedback without tears\.*Advances in Neural Information Processing Systems*, 36:10935–10950\.
- Zeng et al\. \(2024\)Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu\. 2024\.On diversified preferences of large language model alignment\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 9194–9210\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, and 1 others\. 2023\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.*Advances in Neural Information Processing Systems*, 36:46595–46623\.
- Zhou et al\. \(2023\)Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others\. 2023\.LIMA: Less is more for alignment\.*Advances in Neural Information Processing Systems*, 36:55006–55021\.

Appendix for PolyAlign

## Appendix ADataset Details and Normalization

Table[1](https://arxiv.org/html/2606.13227#S4.T1)lists the datasets used in our bilingual English–Chinese setting and their corresponding interaction situations\. The English portion covers all five situations, while the Chinese portion covers four, withtask\_dialoguecurrently instantiated only in English\. We therefore treat the corpus as a partially crossed language–situation design rather than a fully balanced factorial one\.

All datasets are normalized into a common schema containing the input, optional dialogue history, human target response, and structured metadata\. We retainlanguage,track,family,style\_bucket, andlength\_binfields for every example\. Fine\-grained conditional regimes are defined from the tuple\(language, track, family, length\_bin\)\.

Split policies are dataset\-specific\. Official train/dev/test partitions are preserved when available, while train\-only releases are partitioned deterministically\. For multi\-turn corpora, splitting is performed at the conversation or dialogue level rather than the turn level\. After normalization, all examples are canonicalized and deduplicated using normalized input–response fields, with evaluation\-safe precedence rules applied to prevent held\-out examples from leaking into optimization data\. Buckets with fewer than 20 held\-out human examples are excluded from distributional evaluation to avoid unstable reference estimates\.

Situation CategoryEnglish \(en\)Chinese \(zh\)assistant\_likeDolly\-15kCOIG\-CQIAlongform\_qaELI5\-CategoryHC3\-Chineseopen\_chatDailyDialogOASST2\-zhqa\_searchMS MARCO, CoQA,SQuAD v2, Natural QuestionsCMRC2018, DRCD,DuReadertask\_dialogueMultiWOZ–

Table 5:Core situation categories in PolyAlign\. These categories are the primary interaction regimes of interest; language, turn structure, family, and length provide additional conditioning variables\.
## Appendix BBaseline Details

All baselines are evaluated on the same test splits and model families\.

#### BaseLM\.

denotes the untuned pretrained model evaluated by direct inference from the input prompt, using conversation history and context when available\. Unless stated, decoding uses temperature=0\.1, top\_p=0\.9, repetition penalty=1\.1=1\.1, frequency penalty=0\.1=0\.1, and presence penalty=0\.0=0\.0\.

#### CoT\.

is an inference\-time prompting baseline that asks the model to produce explicit step\-by\-step reasoning followed by a final answer in json format\. The prompt follows the example language and includes the target response style and length guidance for the corresponding interaction setting\.

#### Full\-SFT\.

denotes the checkpoint obtained by standard supervised fine\-tuning on the full training mixture\. At evaluation time, the model is prompted through its chat template and asked to generate the target response directly\.

#### DPO\.

denotes the checkpoint obtained by Direct Preference Optimization on chosen\-rejected response pairs\. We evaluate the resulting model with the same chat\-template inference pipeline used for the other post\-trained checkpoints\.

#### Why these baselines?

Taken together, these baselines cover the main alternatives to our approach: no post\-training \(BaseLM\), inference\-time reasoning only \(CoT\), standard supervised post\-training \(Full\-SFT\), and standard preference\-based alignment \(DPO\)\. This makes it possible to attribute gains from PolyAlign specifically to conditional distribution\-aware alignment rather than to scale, prompting, or generic post\-training alone\.

## Appendix CImplementation Details and Compute

#### Compute environment\.

All post\-training and evaluation experiments were conducted on GPU nodes with eight AMD Instinct MI210 GPUs, each with 64 GiB of memory\. We used PyTorch with ROCm support, Hugging Face Transformers, and LLaMA\-Factory for full\-parameter supervised fine\-tuning and preference optimization\. Unless otherwise specified, training used single\-node distributed execution with one process per GPU\.

#### Training pipeline\.

For each model and language, all methods use the same normalized PolyAlign data schema and evaluation format\. Full\-SFT and Bucket\-SFT are trained as supervised fine\-tuning runs; Bucket\-SFT additionally uses bucket\-level example weights derived from the macro\-bucket objective\. DPO and HDPO are initialized from the corresponding supervised checkpoint\. HDPO consists of three stages: training a bucket\-conditioned critic, scoring candidate preference pairs with the critic, and optimizing the final policy with the HDPO objective\. This design keeps the prompt format, data representation, and evaluation split fixed across methods, so differences in performance are attributable to the post\-training objective\.

#### Generation and automatic evaluation\.

For automatic evaluation, we generate responses from the final checkpoint of each method using the model’s chat template\. Unless otherwise stated, generation uses low\-temperature decoding with temperature0\.10\.1, top\-pp0\.90\.9, repetition penalty1\.11\.1, frequency penalty0\.10\.1, and presence penalty0\.00\.0\. Generated responses are evaluated against the same held\-out test instances and bucket metadata used for the corresponding language and model\. Distributional metrics such as BNG and conditional MAUVE use the bucket\-specific human reference statistics defined by the PolyAlign data construction\.

#### LLM\-as\-a\-judge evaluation\.

For rubric\-based judge evaluation, we use Qwen3\-8B and Qwen2\.5\-7B\-Instruct as local vLLM\-served judges\. Since both judge models fit on a single MI210 GPU, we use replica parallelism rather than tensor parallelism: each GPU hosts one independent judge replica, and evaluation examples are sharded across replicas\. This setup is well suited to the judge workload, which consists of many independent scoring requests\. Judge responses are parsed into the eight rubric dimensions described in Appendix[H](https://arxiv.org/html/2606.13227#A8)and aggregated into Overall, Utility, Conditional Naturalness, and Distribution Faithfulness scores\.

#### Runtime reporting\.

Table[6](https://arxiv.org/html/2606.13227#A3.T6)summarizes the compute configuration, and Table[7](https://arxiv.org/html/2606.13227#A3.T7)reports wall\-clock training time for the full experimental grid\. BaseLM and CoT are inference\-only baselines and therefore do not appear in the training\-runtime table\. The reported times exclude queueing, checkpoint transfer, and evaluation post\-processing\.

ItemConfigurationGPU node8×\\timesAMD Instinct MI210 GPUsGPU memory64 GiB per GPUTraining frameworkPyTorch/ROCm with LLaMA\-FactoryDistributed trainingSingle\-node 8\-GPU trainingJudge servingvLLM with one judge replica per GPUJudge modelsQwen3\-8B and Qwen2\.5\-7B\-Instruct

Table 6:Compute configuration used for post\-training and evaluation\.ModelStageRuntimeStepsEnglish DatasetQwen2\.5\-1\.5BFull\-SFT5h 35m9132Bucket\-SFT6h 05m9132DPO policy14h 20m6685HDPO critic8h 55m68134HDPO policy15h 05m6685Gemma\-2\-2BFull\-SFT7h 05m9132Bucket\-SFT7h 45m9132DPO policy17h 50m6987HDPO critic9h 10m68134HDPO policy18h 54m6987Qwen2\.5\-3BFull\-SFT8h 10m9132Bucket\-SFT8h 30m9132DPO policy20h 05m6712HDPO critic9h 05m68134HDPO policy20h 42m6712Llama\-3\.2\-3BFull\-SFT8h 30m9132Bucket\-SFT8h 45m9132DPO policy23h 11m7176HDPO critic1h 35m8517HDPO policy21h 36m7021Chinese DatasetQwen2\.5\-1\.5BFull\-SFT55m1349Bucket\-SFT1h 05m1349DPO policy2h 05m840HDPO critic1h 30m10792HDPO policy2h 05m840Gemma\-2\-2BFull\-SFT1h 20m1349Bucket\-SFT1h 25m1349DPO policy2h 50m915HDPO critic1h 40m10792HDPO policy3h 02m915Qwen2\.5\-3BFull\-SFT1h 30m1349Bucket\-SFT1h 35m1349DPO policy2h 40m745HDPO critic1h 35m10792HDPO policy2h 43m745Llama\-3\.2\-3BFull\-SFT1h 35m1349Bucket\-SFT1h 40m1349DPO policy2h 55m767HDPO critic1h 35m10792HDPO policy3h 03m767Table 7:Wall\-clock training time for the full experimental grid\. Times exclude queueing, checkpoint transfer, and evaluation post\-processing\.

## Appendix DFurther details on Evaluation metrics

Lets∈𝒮s\\in\\mathcal\{S\}denote an observed language\-situation stratum, and letb∈ℬb\\in\\mathcal\{B\}denote an eligible fine\-grained bucket within that stratum\. For each bucketbb, letHbH\_\{b\}andGbG\_\{b\}denote the sets of human and model\-generated responses\. We report all distributional metrics as macro averages over eligible buckets\.

#### Utility\.

We report exact match \(EM\), normalized exact match \(nEM\), token\-level F1, and ROUGE\-L\. These are for reference\-grounded regimes, and we interpret them as complementary rather than sufficient evidence in open\-ended settings\.

#### Bucketed Naturalness Gap \(BNG\)\.

Letϕk​\(y\)\\phi\_\{k\}\(y\)denote thekk\-th linguistic feature extracted from responseyy, and letσb,k\\sigma\_\{b,k\}be the empirical standard deviation of featurekkunder the human bucketHbH\_\{b\}\. For a bucketbbwith feature setℱb\\mathcal\{F\}\_\{b\}, we defineBNG​\(b\)=1\|ℱb\|​∑k∈ℱbW1​\(ϕk​\(Gb\),ϕk​\(Hb\)\)max⁡\(σb,k,ε\),\\mathrm\{BNG\}\(b\)=\\frac\{1\}\{\|\\mathcal\{F\}\_\{b\}\|\}\\sum\_\{k\\in\\mathcal\{F\}\_\{b\}\}\\frac\{W\_\{1\}\\\!\\left\(\\phi\_\{k\}\(G\_\{b\}\),\\phi\_\{k\}\(H\_\{b\}\)\\right\)\}\{\\max\(\\sigma\_\{b,k\},\\varepsilon\)\},whereW1W\_\{1\}is the 1\-Wasserstein distance andε\>0\\varepsilon\>0\. BNG measures how far the generated feature distribution drifts from the human feature distribution inside the correct bucket\.

#### Conditional MAUVE\.

To evaluate distributional similarity directly, we compute MAUVE between human and generated responses within each bucket:C​\-​MAUVE=1\|ℬ\|​∑b∈ℬMAUVE​\(Hb,Gb\)\.\\mathrm\{C\\text\{\-\}MAUVE\}=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{b\\in\\mathcal\{B\}\}\\mathrm\{MAUVE\}\(H\_\{b\},G\_\{b\}\)\.We report global MAUVE computed over the pooled setsHbH\_\{b\}andGbG\_\{b\}\.

#### Turn Dynamics Match \(TDM\)\.

For multi\-turn settings, we additionally evaluate whether model responses reproduce human turn\-level interaction dynamics\. For a prompt–response pair\(x,y\)\(x,y\), we define three per\-example quantities:r​\(x,y\)=\|y\|max⁡\(\|x\|,1\),a​\(x,y\)=cos⁡\(tfidf​\(x\),tfidf​\(y\)\),c​\(x,y\)=cos⁡\(zx,zy\),r\(x,y\)=\\frac\{\|y\|\}\{\\max\(\|x\|,1\)\},\\;a\(x,y\)=\\cos\\\!\\bigl\(\\mathrm\{tfidf\}\(x\),\\mathrm\{tfidf\}\(y\)\\bigr\),\\;c\(x,y\)=\\cos\(z\_\{x\},z\_\{y\}\),whererris the response\-length ratio,aais lexical accommodation, andccis semantic coupling;zxz\_\{x\}andzyz\_\{y\}are latent representations obtained from a SVD projection of TF\-IDF features\. For each multi\-turn bucketbb, we compute standardized Wasserstein gaps for these three quantities:Δbu=W1​\(u​\(Gb\),u​\(Hb\)\)max⁡\(Std​\[u​\(Hb\)\],ε\),u∈\{r,a,c\}\.\\Delta\_\{b\}^\{u\}=\\frac\{W\_\{1\}\\\!\\left\(u\(G\_\{b\}\),u\(H\_\{b\}\)\\right\)\}\{\\max\(\\mathrm\{Std\}\[u\(H\_\{b\}\)\],\\varepsilon\)\},\\qquad u\\in\\\{r,a,c\\\}\.We further compare latent turn\-transition operators\. LetQbQ\_\{b\},RbHR\_\{b\}^\{H\}, andRbGR\_\{b\}^\{G\}denote the centered latent prompt, human\-response, and generated\-response matrices for bucketbb\. We defineΔbtr\.\\Delta\_\{b\}^\{\\mathrm\{tr\}\}\.Each gap is converted to a bounded similaritys​\(Δ\)=1/\(1\+Δ\)s\(\\Delta\)=1/\(1\+\\Delta\), and TDM is defined asTDM​\(b\)\.\\mathrm\{TDM\}\(b\)\.

Δbtr=‖Qb⊤​RbGnb−Qb⊤​RbHnb‖Fmax⁡\(‖Qb⊤​RbHnb‖F,ε\)\\Delta\_\{b\}^\{\\mathrm\{tr\}\}=\\frac\{\\left\\\|\\frac\{Q\_\{b\}^\{\\top\}R\_\{b\}^\{G\}\}\{n\_\{b\}\}\-\\frac\{Q\_\{b\}^\{\\top\}R\_\{b\}^\{H\}\}\{n\_\{b\}\}\\right\\\|\_\{F\}\}\{\\max\\\!\\left\(\\left\\\|\\frac\{Q\_\{b\}^\{\\top\}R\_\{b\}^\{H\}\}\{n\_\{b\}\}\\right\\\|\_\{F\},\\varepsilon\\right\)\}\(13\)
TDM​\(b\)=14​\(s​\(Δbr\)\+s​\(Δba\)\+s​\(Δbc\)\+s​\(Δbtr\)\)\\mathrm\{TDM\}\(b\)=\\frac\{1\}\{4\}\\Bigl\(s\(\\Delta\_\{b\}^\{r\}\)\+s\(\\Delta\_\{b\}^\{a\}\)\+s\(\\Delta\_\{b\}^\{c\}\)\+s\(\\Delta\_\{b\}^\{\\mathrm\{tr\}\}\)\\Bigr\)\(14\)

#### Naturalness–Utility Frontier \(NUF\)\.

The primitive metrics above characterize different aspects of conditional alignment, but PolyAlign ultimately aims to improve*both*usefulness and conditional naturalness\. We therefore summarize this trade\-off with the Naturalness–Utility Frontier \(NUF\)\.

For each bucketbb, letub=clip​\(F1b,0,1\)u\_\{b\}=\\mathrm\{clip\}\(\\mathrm\{F1\}\_\{b\},0,1\)denote the normalized score, whereF1b\\mathrm\{F1\}\_\{b\}is bucket\-level QA\-F1\. We normalize BNG byBNG~b=11\+BNG​\(b\),\\tilde\{\\mathrm\{BNG\}\}\_\{b\}=\\frac\{1\}\{1\+\\mathrm\{BNG\}\(b\)\},clip the bounded text\-distribution metric asC​\-​MAUVE~b=clip​\(C​\-​MAUVEb,0,1\),\\widetilde\{\\mathrm\{C\\text\{\-\}MAUVE\}\}\_\{b\}=\\mathrm\{clip\}\(\\mathrm\{C\\text\{\-\}MAUVE\}\_\{b\},0,1\),andTDM~b=clip​\(TDMb,0,1\)\.\\widetilde\{\\mathrm\{TDM\}\}\_\{b\}=\\mathrm\{clip\}\(\\mathrm\{TDM\}\_\{b\},0,1\)\.We define the bucket\-level naturalness score as the bounded geometric meannbn\_\{b\}and for multi\-turn, we extend this tonbmtn\_\{b\}^\{\\text\{mt\}\}\.

nb=\(BNG~b⋅C​\-​MAUVE~b\)1/2n\_\{b\}=\\Bigl\(\\tilde\{\\mathrm\{BNG\}\}\_\{b\}\\cdot\\widetilde\{\\mathrm\{C\\text\{\-\}MAUVE\}\}\_\{b\}\\Bigr\)^\{1/2\}\(15\)
nbmt=\(BNG~b⋅C​\-​MAUVE~b⋅TDM~b\)1/3n\_\{b\}^\{\\text\{mt\}\}=\\Bigl\(\\tilde\{\\mathrm\{BNG\}\}\_\{b\}\\cdot\\widetilde\{\\mathrm\{C\\text\{\-\}MAUVE\}\}\_\{b\}\\cdot\\widetilde\{\\mathrm\{TDM\}\}\_\{b\}\\Bigr\)^\{1/3\}\(16\)

Each bucket is thus mapped to a point\(ub,nb\)\(u\_\{b\},n\_\{b\}\)in utility\-naturalness space\. We define the Pareto frontier as the set of non\-dominated buckets and summarize it with frontier hypervolume relative to the reference point\(0,0\)\(0,0\):NUF​\-​HV=Hypervolume​\(\{\(ub,nb\)\}b∈ℬ,\(0,0\)\)\.\\mathrm\{NUF\\text\{\-\}HV\}=\\mathrm\{Hypervolume\}\\bigl\(\\\{\(u\_\{b\},n\_\{b\}\)\\\}\_\{b\\in\\mathcal\{B\}\},\(0,0\)\\bigr\)\.

## Appendix EMore details on Human Evaluation

We complement automatic metrics, AI\-detection, and LLM\-as\-a\-judge evaluation with human evaluation on held\-out English and Chinese examples spanning QA, context\-grounded QA, and assistant\-style instruction following\. Human evaluators score in Table[8](https://arxiv.org/html/2606.13227#A6.T8), each candidate on 1–5Usefulness,Answerability, andNaturalness, validating whether model responses remain useful, grounded, and bucket\-appropriate across PolyAlign metadata bins\.

## Appendix FDeclaration of LLM usage

During the preparation of this manuscript, ChatGPT was used only for language polishing, grammar checking, and improving sentence clarity and readability\. The tool was not used to generate experimental results, design the methodology, conduct analysis, create citations, or formulate the scientific claims of the paper\. All technical content, interpretations, figures, tables, and references were reviewed, verified, and finalized by the authors\.

MethodnnUseAnsNatAvg\.BaseLM193\.473\.893\.843\.74Full\-SFT204\.454\.654\.554\.55DPO203\.053\.602\.953\.20Bucket\-SFT204\.604\.955\.004\.85Table 8:Chinese human evaluation results from completed annotations\. Scores are on a 1\-5 scale\. Usefulness measures task satisfaction, Answerability measures context/reference support, and Naturalness measures fluent and bucket\-appropriate Chinese generation\.
## Appendix GProofs of Theorems

### G\.1Proof of Theorem[1](https://arxiv.org/html/2606.13227#Thmtheorem1): Bucket\-SFT Optimizes Exact Macro Bucket Risk

###### Proof\.

By definition, every example in bucketbbreceives the same pre\-serialization weight

w~b=N\|ℬ\|​nb\.\\widetilde\{w\}\_\{b\}=\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\.Therefore, for any fixed bucketb∈ℬb\\in\\mathcal\{B\},

∑i:bi=bw~bi=∑i:bi=bN\|ℬ\|​nb=nb⋅N\|ℬ\|​nb=N\|ℬ\|\.\\sum\_\{i:\\,b\_\{i\}=b\}\\widetilde\{w\}\_\{b\_\{i\}\}=\\sum\_\{i:\\,b\_\{i\}=b\}\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}=n\_\{b\}\\cdot\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}=\\frac\{N\}\{\|\\mathcal\{B\}\|\}\.This proves the equal\-mass identity\.

Now sum over all buckets:

∑i=1Nw~bi=∑b∈ℬ∑i:bi=bw~bi=∑b∈ℬN\|ℬ\|=N\.\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{b\_\{i\}\}=\\sum\_\{b\\in\\mathcal\{B\}\}\\sum\_\{i:\\,b\_\{i\}=b\}\\widetilde\{w\}\_\{b\_\{i\}\}=\\sum\_\{b\\in\\mathcal\{B\}\}\\frac\{N\}\{\|\\mathcal\{B\}\|\}=N\.Hence

LBucket​\-​SFT​\(θ\)=1N​∑i=1Nw~bi​ℓi​\(θ\)\.L\_\{\\mathrm\{Bucket\\text\{\-\}SFT\}\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{b\_\{i\}\}\\,\\ell\_\{i\}\(\\theta\)\.Grouping by bucket gives

LBucket​\-​SFT​\(θ\)=1N​∑b∈ℬ∑i:bi=bN\|ℬ\|​nb​ℓi​\(θ\)=1\|ℬ\|​∑b∈ℬ1nb​∑i:bi=bℓi​\(θ\),L\_\{\\mathrm\{Bucket\\text\{\-\}SFT\}\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{b\\in\\mathcal\{B\}\}\\sum\_\{i:\\,b\_\{i\}=b\}\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\\,\\ell\_\{i\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{b\\in\\mathcal\{B\}\}\\frac\{1\}\{n\_\{b\}\}\\sum\_\{i:\\,b\_\{i\}=b\}\\ell\_\{i\}\(\\theta\),which is exactly the macro average of bucket\-wise empirical risks\. ∎

### G\.2Proof of Theorem[2](https://arxiv.org/html/2606.13227#Thmtheorem2): Bucket\-Support Distance Is a Continuous Relaxation of Human Membership

###### Proof\.

Fix bucketbband a feature vectorzzwithJb​\(z\)≠∅J\_\{b\}\(z\)\\neq\\varnothing\.

For eachj∈Jb​\(z\)j\\in J\_\{b\}\(z\), define

db​j​\(zj\):=\[lb​j−zj\]\+\+\[zj−ub​j\]\+sb​j\.d\_\{bj\}\(z\_\{j\}\):=\\frac\{\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}\+\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}\}\{s\_\{bj\}\}\.Sincesb​j\>0s\_\{bj\}\>0by definition and each positive\-part term is nonnegative,

db​j​\(zj\)≥0for every​j∈Jb​\(z\)\.d\_\{bj\}\(z\_\{j\}\)\\geq 0\\qquad\\text\{for every \}j\\in J\_\{b\}\(z\)\.Averaging overjjgives

Db​\(z\)=1\|Jb​\(z\)\|​∑j∈Jb​\(z\)db​j​\(zj\)≥0,D\_\{b\}\(z\)=\\frac\{1\}\{\|J\_\{b\}\(z\)\|\}\\sum\_\{j\\in J\_\{b\}\(z\)\}d\_\{bj\}\(z\_\{j\}\)\\geq 0,which proves \(i\)\.

For \(ii\), first suppose thatzj∈\[lb​j,ub​j\]z\_\{j\}\\in\[l\_\{bj\},u\_\{bj\}\]for everyj∈Jb​\(z\)j\\in J\_\{b\}\(z\)\. Then for every suchjj,

\[lb​j−zj\]\+=0,\[zj−ub​j\]\+=0,\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}=0,\\qquad\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}=0,sodb​j​\(zj\)=0d\_\{bj\}\(z\_\{j\}\)=0\. HenceDb​\(z\)=0D\_\{b\}\(z\)=0\.

Conversely, supposeDb​\(z\)=0D\_\{b\}\(z\)=0\. SinceDb​\(z\)D\_\{b\}\(z\)is the average of nonnegative numbers, every summand must be zero:

db​j​\(zj\)=0for every​j∈Jb​\(z\)\.d\_\{bj\}\(z\_\{j\}\)=0\\qquad\\text\{for every \}j\\in J\_\{b\}\(z\)\.Becausesb​j\>0s\_\{bj\}\>0, this implies

\[lb​j−zj\]\+\+\[zj−ub​j\]\+=0for every​j∈Jb​\(z\)\.\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}\+\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}=0\\qquad\\text\{for every \}j\\in J\_\{b\}\(z\)\.A sum of two nonnegative terms is zero iff both terms are zero, so

\[lb​j−zj\]\+=0and\[zj−ub​j\]\+=0for every​j∈Jb​\(z\),\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}=0\\quad\\text\{and\}\\quad\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}=0\\qquad\\text\{for every \}j\\in J\_\{b\}\(z\),which is equivalent to

lb​j≤zj≤ub​jfor every​j∈Jb​\(z\)\.l\_\{bj\}\\leq z\_\{j\}\\leq u\_\{bj\}\\qquad\\text\{for every \}j\\in J\_\{b\}\(z\)\.This proves \(ii\)\.

Finally, each map

zj↦\[lb​j−zj\]\+andzj↦\[zj−ub​j\]\+z\_\{j\}\\mapsto\\left\[l\_\{bj\}\-z\_\{j\}\\right\]\_\{\+\}\\quad\\text\{and\}\\quad z\_\{j\}\\mapsto\\left\[z\_\{j\}\-u\_\{bj\}\\right\]\_\{\+\}is continuous and piecewise linear\. Dividing by the positive constantsb​js\_\{bj\}preserves these properties, and finite sums and averages preserve them as well\. HenceDb​\(z\)D\_\{b\}\(z\)is continuous and piecewise linear inzz, proving \(iii\)\. ∎

### G\.3Proof of Theorem[3](https://arxiv.org/html/2606.13227#Thmtheorem3): Distributional Alignment of the HDPO Regularizer with Sigmoid DPO

###### Proof\.

Write

s\+:=sϕ​\(y\+,b\),s−:=sϕ​\(y−,b\),pθ:=σ​\(β​Δπ\)\.s^\{\+\}:=s\_\{\\phi\}\(y^\{\+\},b\),\\qquad s^\{\-\}:=s\_\{\\phi\}\(y^\{\-\},b\),\\qquad p\_\{\\theta\}:=\\sigma\(\\beta\\Delta\_\{\\pi\}\)\.Then the HDPO regularizer is

RHDPO​\(θ\)=pθ​s\+\+\(1−pθ\)​s−=s−\+pθ​\(s\+−s−\)\.R\_\{\\mathrm\{HDPO\}\}\(\\theta\)=p\_\{\\theta\}s^\{\+\}\+\(1\-p\_\{\\theta\}\)s^\{\-\}=s^\{\-\}\+p\_\{\\theta\}\(s^\{\+\}\-s^\{\-\}\)\.Differentiate with respect toΔπ\\Delta\_\{\\pi\}\. Since

dd​Δπ​σ​\(β​Δπ\)=β​σ​\(β​Δπ\)​\(1−σ​\(β​Δπ\)\)=β​pθ​\(1−pθ\),\\frac\{d\}\{d\\Delta\_\{\\pi\}\}\\sigma\(\\beta\\Delta\_\{\\pi\}\)=\\beta\\,\\sigma\(\\beta\\Delta\_\{\\pi\}\)\\bigl\(1\-\\sigma\(\\beta\\Delta\_\{\\pi\}\)\\bigr\)=\\beta\\,p\_\{\\theta\}\(1\-p\_\{\\theta\}\),we obtain

d​RHDPOd​Δπ=β​pθ​\(1−pθ\)​\(s\+−s−\)\.\\frac\{dR\_\{\\mathrm\{HDPO\}\}\}\{d\\Delta\_\{\\pi\}\}=\\beta\\,p\_\{\\theta\}\(1\-p\_\{\\theta\}\)\\,\(s^\{\+\}\-s^\{\-\}\)\.This proves the derivative formula\.

Now supposes\+<s−s^\{\+\}<s^\{\-\}\. Becauseβ\>0\\beta\>0andpθ​\(1−pθ\)\>0p\_\{\\theta\}\(1\-p\_\{\\theta\}\)\>0for all finiteΔπ\\Delta\_\{\\pi\}, it follows that

d​RHDPOd​Δπ<0\.\\frac\{dR\_\{\\mathrm\{HDPO\}\}\}\{d\\Delta\_\{\\pi\}\}<0\.Therefore increasingΔπ\\Delta\_\{\\pi\}decreasesRHDPOR\_\{\\mathrm\{HDPO\}\}, so minimizing the regularizer favors larger chosen\-vs\-rejected policy margin\.

Next consider the sigmoid DPO loss

LDPO​\(Δπ;Δref\)=−log⁡σ​\(β​\(Δπ−Δref\)\)\.L\_\{\\mathrm\{DPO\}\}\(\\Delta\_\{\\pi\};\\Delta\_\{\\mathrm\{ref\}\}\)=\-\\log\\sigma\\\!\\bigl\(\\beta\(\\Delta\_\{\\pi\}\-\\Delta\_\{\\mathrm\{ref\}\}\)\\bigr\)\.Let

u:=β​\(Δπ−Δref\)\.u:=\\beta\(\\Delta\_\{\\pi\}\-\\Delta\_\{\\mathrm\{ref\}\}\)\.Then

LDPO=−log⁡σ​\(u\)\.L\_\{\\mathrm\{DPO\}\}=\-\\log\\sigma\(u\)\.Using

dd​u​\[−log⁡σ​\(u\)\]=−\(1−σ​\(u\)\),\\frac\{d\}\{du\}\[\-\\log\\sigma\(u\)\]=\-\(1\-\\sigma\(u\)\),and the chain ruled​ud​Δπ=β\\frac\{du\}\{d\\Delta\_\{\\pi\}\}=\\beta, we get

d​LDPOd​Δπ=−β​\(1−σ​\(u\)\)=−β​\(1−σ​\(β​\(Δπ−Δref\)\)\)<0\.\\frac\{dL\_\{\\mathrm\{DPO\}\}\}\{d\\Delta\_\{\\pi\}\}=\-\\beta\\bigl\(1\-\\sigma\(u\)\\bigr\)=\-\\beta\\Bigl\(1\-\\sigma\\bigl\(\\beta\(\\Delta\_\{\\pi\}\-\\Delta\_\{\\mathrm\{ref\}\}\)\\bigr\)\\Bigr\)<0\.Thus increasingΔπ\\Delta\_\{\\pi\}also decreases the sigmoid DPO loss\.

Hence, whenevers\+<s−s^\{\+\}<s^\{\-\}, both the HDPO regularizer and the sigmoid DPO term are reduced by increasingΔπ\\Delta\_\{\\pi\}\. They are therefore distributionally aligned during optimization\. ∎

### G\.4Proof of Theorem[4](https://arxiv.org/html/2606.13227#Thmtheorem4)

###### Theorem 4\(Zero HDPO Critic Loss Implies Exact Calibration and Margin\-Consistent Ranking\)\.

Consider the HDPO critic objective

Lcritic​\(ϕ\)=λreg⋅1M​∑m=1M\|D^ϕ​\(ym,bm\)−Dbm​\(ym\)\|\+λrank⋅1P​∑p=1P\[κ−D^ϕ​\(yp−,bp\)\+D^ϕ​\(yp\+,bp\)\]\+,L\_\{\\mathrm\{critic\}\}\(\\phi\)=\\lambda\_\{\\mathrm\{reg\}\}\\cdot\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\left\|\\widehat\{D\}\_\{\\phi\}\(y\_\{m\},b\_\{m\}\)\-D\_\{b\_\{m\}\}\(y\_\{m\}\)\\right\|\+\\lambda\_\{\\mathrm\{rank\}\}\\cdot\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\\left\[\\kappa\-\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\+\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\\right\]\_\{\+\},whereλreg\>0\\lambda\_\{\\mathrm\{reg\}\}\>0,λrank\>0\\lambda\_\{\\mathrm\{rank\}\}\>0, andκ\>0\\kappa\>0is the ranking margin\. IfLcritic​\(ϕ\)=0L\_\{\\mathrm\{critic\}\}\(\\phi\)=0, then:

D^ϕ​\(ym,bm\)=Dbm​\(ym\)for every regression sample​m,\\displaystyle\\widehat\{D\}\_\{\\phi\}\(y\_\{m\},b\_\{m\}\)=D\_\{b\_\{m\}\}\(y\_\{m\}\)\\qquad\\text\{for every regression sample \}m,D^ϕ​\(yp\+,bp\)\+κ≤D^ϕ​\(yp−,bp\)for every pairwise ranking sample​p\.\\displaystyle\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\+\\kappa\\leq\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\\qquad\\text\{for every pairwise ranking sample \}p\.Thus, zero critic loss implies simultaneous exact distance calibration and strict chosen\-over\-rejected margin ordering\.

###### Proof\.

Write

Am:=\|D^ϕ​\(ym,bm\)−Dbm​\(ym\)\|for​m=1,…,M,A\_\{m\}:=\\left\|\\widehat\{D\}\_\{\\phi\}\(y\_\{m\},b\_\{m\}\)\-D\_\{b\_\{m\}\}\(y\_\{m\}\)\\right\|\\qquad\\text\{for \}m=1,\\dots,M,and

Bp:=\[κ−D^ϕ​\(yp−,bp\)\+D^ϕ​\(yp\+,bp\)\]\+for​p=1,…,P\.B\_\{p\}:=\\left\[\\kappa\-\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\+\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\\right\]\_\{\+\}\\qquad\\text\{for \}p=1,\\dots,P\.ThenAm≥0A\_\{m\}\\geq 0andBp≥0B\_\{p\}\\geq 0for allm,pm,p, and the critic loss is

Lcritic​\(ϕ\)=λreg⋅1M​∑m=1MAm\+λrank⋅1P​∑p=1PBp\.L\_\{\\mathrm\{critic\}\}\(\\phi\)=\\lambda\_\{\\mathrm\{reg\}\}\\cdot\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}A\_\{m\}\+\\lambda\_\{\\mathrm\{rank\}\}\\cdot\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}B\_\{p\}\.Sinceλreg\>0\\lambda\_\{\\mathrm\{reg\}\}\>0andλrank\>0\\lambda\_\{\\mathrm\{rank\}\}\>0, and both averages are nonnegative, the equalityLcritic​\(ϕ\)=0L\_\{\\mathrm\{critic\}\}\(\\phi\)=0implies

1M​∑m=1MAm=0and1P​∑p=1PBp=0\.\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}A\_\{m\}=0\\qquad\\text\{and\}\\qquad\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}B\_\{p\}=0\.Because eachAm≥0A\_\{m\}\\geq 0, the first equality impliesAm=0A\_\{m\}=0for everymm, i\.e\.,

\|D^ϕ​\(ym,bm\)−Dbm​\(ym\)\|=0∀m\.\\left\|\\widehat\{D\}\_\{\\phi\}\(y\_\{m\},b\_\{m\}\)\-D\_\{b\_\{m\}\}\(y\_\{m\}\)\\right\|=0\\qquad\\forall m\.Therefore

D^ϕ​\(ym,bm\)=Dbm​\(ym\)∀m\.\\widehat\{D\}\_\{\\phi\}\(y\_\{m\},b\_\{m\}\)=D\_\{b\_\{m\}\}\(y\_\{m\}\)\\qquad\\forall m\.
Likewise, because eachBp≥0B\_\{p\}\\geq 0, the second equality impliesBp=0B\_\{p\}=0for everypp, i\.e\.,

\[κ−D^ϕ​\(yp−,bp\)\+D^ϕ​\(yp\+,bp\)\]\+=0∀p\.\\left\[\\kappa\-\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\+\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\\right\]\_\{\+\}=0\\qquad\\forall p\.For any scalaruu,\[u\]\+=0\\left\[u\\right\]\_\{\+\}=0iffu≤0u\\leq 0\. Hence

κ−D^ϕ​\(yp−,bp\)\+D^ϕ​\(yp\+,bp\)≤0∀p,\\kappa\-\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\+\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\\leq 0\\qquad\\forall p,which rearranges to

D^ϕ​\(yp\+,bp\)\+κ≤D^ϕ​\(yp−,bp\)∀p\.\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\+\},b\_\{p\}\)\+\\kappa\\leq\\widehat\{D\}\_\{\\phi\}\(y\_\{p\}^\{\-\},b\_\{p\}\)\\qquad\\forall p\.Thus zero total critic loss implies both exact regression calibration and margin\-consistent ranking\. ∎

### G\.5Proof of Theorem[5](https://arxiv.org/html/2606.13227#Thmtheorem5)

###### Theorem 5\(HDPO Weighting Amplifies Scarce and Distribution\-Critical Pairs\)\.

Define the bucket scarcity score

rb:=max⁡\{0,N\|ℬ\|​nb−1\},gb:=\{rb/maxb′∈ℬ⁡rb′if​maxb′⁡rb′\>0,0otherwise,r\_\{b\}:=\\max\\\!\\left\\\{0,\\ \\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\-1\\right\\\},\\qquad g\_\{b\}:=\\begin\{cases\}r\_\{b\}/\\max\_\{b^\{\\prime\}\\in\\mathcal\{B\}\}r\_\{b^\{\\prime\}\}&\\text\{if \}\\max\_\{b^\{\\prime\}\}r\_\{b^\{\\prime\}\}\>0,\\\\ 0&\\text\{otherwise,\}\\end\{cases\}and analogously define a normalized language rarity scorehℓ∈\[0,1\]h\_\{\\ell\}\\in\[0,1\]from inverse language frequency\. For pairii, let

δi:=\[si−−si\+\]\+,\\delta\_\{i\}:=\\left\[s\_\{i\}^\{\-\}\-s\_\{i\}^\{\+\}\\right\]\_\{\+\},wheresi\+s\_\{i\}^\{\+\}andsi−s\_\{i\}^\{\-\}are the chosen and rejected critic scores, and letρti\\rho\_\{t\_\{i\}\}denote the pair\-type bias\. Define the pre\-rounding HDPO weight

w~i:=clip⁡\(1\+α​gbi\+β​δi\+γ​hℓi\+η​ρti,wmin,wmax\),\\widetilde\{w\}\_\{i\}:=\\operatorname\{clip\}\\\!\\Big\(1\+\\alpha g\_\{b\_\{i\}\}\+\\beta\\delta\_\{i\}\+\\gamma h\_\{\\ell\_\{i\}\}\+\\eta\\rho\_\{t\_\{i\}\},\\ w\_\{\\min\},\\ w\_\{\\max\}\\Big\),withα,β,γ,η≥0\\alpha,\\beta,\\gamma,\\eta\\geq 0and0<wmin≤wmax0<w\_\{\\min\}\\leq w\_\{\\max\}\. Then:

\(i\)w~i∈\[wmin,wmax\]for every​i,\\displaystyle\\text\{\(i\)\}\\quad\\widetilde\{w\}\_\{i\}\\in\[w\_\{\\min\},w\_\{\\max\}\]\\quad\\text\{for every \}i,\(ii\)w~i​is nondecreasing in each of​gbi,δi,hℓi,ρti,\\displaystyle\\text\{\(ii\)\}\\quad\\widetilde\{w\}\_\{i\}\\text\{ is nondecreasing in each of \}g\_\{b\_\{i\}\},\\ \\delta\_\{i\},\\ h\_\{\\ell\_\{i\}\},\\ \\rho\_\{t\_\{i\}\},\(iii\)gb=0whenever​nb≥N/\|ℬ\|\.\\displaystyle\\text\{\(iii\)\}\\quad g\_\{b\}=0\\quad\\text\{whenever \}n\_\{b\}\\geq N/\|\\mathcal\{B\}\|\.Moreover, for any fixed weightswi\>0w\_\{i\}\>0used by training, the normalized weighted preference objective

Lpref​\(θ\):=∑i=1Kwi​ℓi​\(θ\)∑i=1KwiL\_\{\\mathrm\{pref\}\}\(\\theta\):=\\frac\{\\sum\_\{i=1\}^\{K\}w\_\{i\}\\,\\ell\_\{i\}\(\\theta\)\}\{\\sum\_\{i=1\}^\{K\}w\_\{i\}\}satisfies

∇θLpref​\(θ\)=1∑j=1Kwj​∑i=1Kwi​∇θℓi​\(θ\)\.\\nabla\_\{\\theta\}L\_\{\\mathrm\{pref\}\}\(\\theta\)=\\frac\{1\}\{\\sum\_\{j=1\}^\{K\}w\_\{j\}\}\\sum\_\{i=1\}^\{K\}w\_\{i\}\\,\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\.Hence HDPO reallocates gradient mass toward scarce buckets, rare languages, and critic\-hard pairs in direct proportion to their assigned weights\.

###### Proof\.

For part \(i\),w~i\\widetilde\{w\}\_\{i\}is obtained by applying the clipping operator

clip⁡\(⋅,wmin,wmax\),\\operatorname\{clip\}\(\\cdot,w\_\{\\min\},w\_\{\\max\}\),so by definition

w~i∈\[wmin,wmax\]for every​i\.\\widetilde\{w\}\_\{i\}\\in\[w\_\{\\min\},w\_\{\\max\}\]\\qquad\\text\{for every \}i\.
For part \(ii\), define the affine score

ai:=1\+α​gbi\+β​δi\+γ​hℓi\+η​ρti\.a\_\{i\}:=1\+\\alpha g\_\{b\_\{i\}\}\+\\beta\\delta\_\{i\}\+\\gamma h\_\{\\ell\_\{i\}\}\+\\eta\\rho\_\{t\_\{i\}\}\.Sinceα,β,γ,η≥0\\alpha,\\beta,\\gamma,\\eta\\geq 0, the mapaia\_\{i\}is nondecreasing in each scalar argumentgbi,δi,hℓi,ρtig\_\{b\_\{i\}\},\\delta\_\{i\},h\_\{\\ell\_\{i\}\},\\rho\_\{t\_\{i\}\}\. The clipping functionu↦clip⁡\(u,wmin,wmax\)u\\mapsto\\operatorname\{clip\}\(u,w\_\{\\min\},w\_\{\\max\}\)is also nondecreasing\. Therefore the composition

w~i=clip⁡\(ai,wmin,wmax\)\\widetilde\{w\}\_\{i\}=\\operatorname\{clip\}\(a\_\{i\},w\_\{\\min\},w\_\{\\max\}\)is nondecreasing in each argument\.

For part \(iii\), recall

rb=max⁡\{0,N\|ℬ\|​nb−1\}\.r\_\{b\}=\\max\\\!\\left\\\{0,\\ \\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\-1\\right\\\}\.Ifnb≥N/\|ℬ\|n\_\{b\}\\geq N/\|\\mathcal\{B\}\|, then

N\|ℬ\|​nb≤1,\\frac\{N\}\{\|\\mathcal\{B\}\|\\,n\_\{b\}\}\\leq 1,so

rb=max⁡\{0,nonpositive quantity\}=0\.r\_\{b\}=\\max\\\{0,\\text\{nonpositive quantity\}\\\}=0\.Ifmaxb′⁡rb′\>0\\max\_\{b^\{\\prime\}\}r\_\{b^\{\\prime\}\}\>0, then

gb=rbmaxb′⁡rb′=0\.g\_\{b\}=\\frac\{r\_\{b\}\}\{\\max\_\{b^\{\\prime\}\}r\_\{b^\{\\prime\}\}\}=0\.Ifmaxb′⁡rb′=0\\max\_\{b^\{\\prime\}\}r\_\{b^\{\\prime\}\}=0, then by definition all normalized scores are set to zero\. Hencegb=0g\_\{b\}=0whenevernb≥N/\|ℬ\|n\_\{b\}\\geq N/\|\\mathcal\{B\}\|\.

For the gradient identity, let

W:=∑j=1Kwj\.W:=\\sum\_\{j=1\}^\{K\}w\_\{j\}\.Since the weights are fixed with respect toθ\\theta,WWis constant inθ\\theta\. Then

Lpref​\(θ\)=1W​∑i=1Kwi​ℓi​\(θ\),L\_\{\\mathrm\{pref\}\}\(\\theta\)=\\frac\{1\}\{W\}\\sum\_\{i=1\}^\{K\}w\_\{i\}\\,\\ell\_\{i\}\(\\theta\),and therefore

∇θLpref​\(θ\)=1W​∑i=1Kwi​∇θℓi​\(θ\)\.\\nabla\_\{\\theta\}L\_\{\\mathrm\{pref\}\}\(\\theta\)=\\frac\{1\}\{W\}\\sum\_\{i=1\}^\{K\}w\_\{i\}\\,\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\.Thus each example contributes to the total gradient in direct proportion to its fixed weight\. Combined with parts \(i\)–\(iii\), this means that higher scarcity, larger positive critic gap, and higher language rarity can only increase, never decrease, the gradient mass assigned to that example, up to the clipping ceiling\. ∎

## Appendix HLLM\-as\-a\-Judge Prompt and Rubric

LLM\-as\-a\-Judge PromptYou are a careful bilingual NLP evaluator\. Judge the candidate response against the user request, context, dialogue history, human reference, and PolyAlign bucket metadata\.Evaluation constraints\.Do not reveal chain\-of\-thought\. Return one valid JSON object only\. Use the full input text provided by the caller; do not ignore or summarize long context\.PolyAlign evaluation objective\.The candidate should preserve task utility while matching the human response distribution appropriate to the current metadata bucket\. In other words, the response should be the right kind of answer for the right language, interaction track, response family, style bucket, and length bin\.Scoring scale\.Score every dimension with an integer from 1 to 5:1 = severe failure, 2 = weak, 3 = acceptable, 4 = strong, and 5 = excellent\.Required JSON output\.Return JSON with exactly the following top\-level keys:\{"scores": \{"task\_success": int,"factual\_grounding": int,"instruction\_following": int,"reference\_alignment": int,"conditional\_appropriateness": int,"response\_shape\_and\_length": int,"discourse\_naturalness": int,"safety": int\},"major\_errors": \[string\],"rationale": string\}Input fields\.Metadata:\{language, dataset, track, family, style\_bucket, length\_bin, bucket\_id, source\_id, model\_key, stage\}Conversation history:<history\> \.\.\. </history\>Context:<context\> \.\.\. </context\>User request:<instruction\> \.\.\. </instruction\>Human reference response:<reference\> \.\.\. </reference\>Candidate response to judge:<candidate\> \.\.\. </candidate\>Final instruction\.Now judge only the candidate response\. Return valid JSON only\.