哪些目标需要调节？在可转向多元对齐中预测目标冲突与涵盖权衡

# Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment
Source: [https://arxiv.org/html/2609.26929](https://arxiv.org/html/2609.26929)
David TsoiAffiliation:Department of Computer Science, University of StuttgartEmail:[david\-alexander\.tsoi@ims\.uni\-stuttgart\.de](mailto:[email protected])Esra DönmezAffiliation:Institute for Natural Language Processing, University of StuttgartAffiliation:Interchange Forum for Reflecting on Intelligent Systems, University of StuttgartEmail:[esra\.doenmez@ims\.uni\-stuttgart\.de](mailto:[email protected])

###### Abstract

People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone\. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently\. Multi\-Objective Direct Preference Optimization \(MODPO\) does this by using an objective weight to span a continuum of trade\-offs\. We study two questions: when can one model improve two objectives simultaneously, and how can many trade\-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre\-training measurements predict whether objectives align or conflict for human\-annotated data, but not for AI\-annotated data, where response length and repetition confound reward\-model scores\. For broader trade\-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training\. These findings yield practical guidance for building steerable models that serve diverse preferences\.

## 1Introduction

People do not share one set of values\. They disagree about what makes a response good, and the qualities they care about trade off against one another: a longer answer can be more helpful but less concise, and a maximally cautious answer can be less useful\. When we align a model, these qualities become*objectives*we optimize, and different stakeholders, holding different values, weight the same objectives differently over the same preferences\. Aligning to a single averaged preference hides this diversity and picks winners among objectives that people would weight differently\([Bakker et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib13);[Plank, 2022](https://arxiv.org/html/2609.26929#bib.bib9)\)\.*Pluralistic alignment*asks instead how one system can serve a plurality of stakeholders\([Sorensen et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib10)\)\.

One practical route is*steerable*pluralism: rather than fixing one compromise, build a model that can be steered to weight competing objectives differently\([Sorensen et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib10)\)\. Multi\-Objective Direct Preference Optimization\([Zhou et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib5), MODPO;\)is an efficient way to build such a model\. It extends Direct Preference Optimization\([Rafailov et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib16), DPO;\)with an*objective weight*that steers the balance between two objectives, such as helpfulness, honesty, or safety\. Training at a range of weights yields a family of models spanning a continuum of trade\-offs, so a stakeholder can pick the compromise their context calls for\.

Two questions decide whether MODPO can be used this way in practice, and neither is answered by reporting scores at a single weight\.

#### RQ1: When can one model satisfy two objectives at once?

Some objectives reinforce each other and some conflict\. If MODPO can improve two objectives together, a practitioner should train one model that serves both; if they conflict, they must instead steer along a trade\-off and choose a point on it\. We ask whether this can be predicted*before*training from properties of the preference data and the base model\. We study two cheap measurements: thepreference\-pair conflict rate, how often the two objectives disagree on which response is better\([Li et al\., 2025](https://arxiv.org/html/2609.26929#bib.bib2)\), a direct measure of that disagreement in the data\([Plank, 2022](https://arxiv.org/html/2609.26929#bib.bib9)\); and theSFT reference\-model score correlation, whether the SFT reference model’s better answers for one objective are also better for the other\.

#### RQ2: How can we cover many trade\-offs without training a model for each?

Serving a continuum of stakeholder preferences should not require training one model per compromise\. Given a family of MODPO models already trained at a few weights, we ask how closely two training\-free reuse methods match a model trained directly at a new*target*weight:nearest\-model selection, using the trained model whose weight is closest to the target, andparameter merging, blending the LoRA adapters of the two neighbouring trained models\([Jang et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib15);[Rame et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib17)\)\.

#### Contributions\.

\(i\) We frame MODPO as a tool for steerable pluralistic alignment and ask, concretely, when it can satisfy two objectives jointly and when it must trade them off\. \(ii\) We test two pre training measurements of objective compatibility across seven objective pairs\. They predict the outcome well for the human\-annotated HelpSteer data but fail for the AI\-annotated UltraFeedback data\. \(iii\) We trace that failure to a*sociotechnical*cause: the apparent conflict is strongly confounded by response length and repetition in the reward\-model scores, a warning for how objective behavior is evaluated\. \(iv\) We show that nearest\-model selection and parameter merging can cover the trade\-off continuum without retraining, but neither reliably matches direct training\. We distill each finding into a practical recipe\.

## 2Related Work

#### Pluralistic alignment\.

A growing line of work argues that alignment should reflect diverse and potentially conflicting human values rather than collapse them into a single averaged preference\([Sorensen et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib10);[Conitzer et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib14);[Kirk et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib12)\)\. This concern is reinforced by evidence that reward\-model quality varies across socially consequential domains and that reward models can systematically prefer socially undesirable responses\([Ghazaryan and Dönmez, 2026](https://arxiv.org/html/2609.26929#bib.bib37)\)\.[Sorensen et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib10)distinguish several forms of pluralism; we focus on the*steerable*setting, in which a single model can be adjusted to weight its objectives differently for different stakeholders\. Existing approaches include training on population\-level agreement\([Bakker et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib13)\), optimizing max\-min objectives over diverse preferences\([Chakraborty et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib11)\), and coordinating multiple LLMs\([Feng et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib6)\)\. We study MODPO as a concrete and efficient approach to steerable pluralism, asking when a single model can satisfy two objectives jointly and when those objectives instead require a trade\-off\.

#### Steering multiple objectives\.

DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib16)\)aligns a model to preferences without a separate reward model or reinforcement learning\. Several methods extend alignment to several objectives with a controllable balance: MODPO folds the balance into the DPO loss\([Zhou et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib5)\); directional preference alignment attaches a preference direction to a multi\-objective reward\([Wang et al\., 2024a](https://arxiv.org/html/2609.26929#bib.bib7)\); Rewards\-in\-Context conditions on the balance in the prompt\([Yang et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib18)\); and controlled or multi\-objective decoding sets it at generation time\([Mudgal et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib19);[Shi et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib20)\)\. These works report scores at chosen weights; we instead ask whether the*outcome*at each weight can be predicted before training\.

#### Disagreement in preference data\.

Human annotators disagree, and this variation is signal, not just noise\([Plank, 2022](https://arxiv.org/html/2609.26929#bib.bib9);[Bakker et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib13)\)\. For multiple objectives,[Li et al\. \(2025\)](https://arxiv.org/html/2609.26929#bib.bib2)name*preference conflict*\(the two objectives prefer different responses for the same pair\), show it is common, and reduce it by rewriting responses;[Xu et al\. \(2026\)](https://arxiv.org/html/2609.26929#bib.bib25)study a related reward consistency\. We do not try to remove conflict\. We measure it, and whether it and a base\-model correlation predict which objectives MODPO can jointly satisfy\.

#### Reusing and merging models\.

Weight\-space methods combine trained models without retraining: Model Soups\([Wortsman et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib21)\), Task Arithmetic\([Ilharco et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib22)\), and AdapterSoup\([Chronopoulou et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib8)\)\. For alignment specifically, Rewarded Soups\([Rame et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib17)\)and Personalized Soups\([Jang et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib15)\)merge models trained on different rewards to reach new preference balances\. Our parameter\-merging method blends the LoRA adapters\([Hu et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib23)\)of the two trained MODPO models nearest a target weight, and we compare it against nearest\-model selection and direct training\.

#### Evaluating with reward models\.

Reward models enable scalable, consistent scoring across objectives, and multi\-objective models such as ArmoRM\([Wang et al\., 2024a](https://arxiv.org/html/2609.26929#bib.bib7)\)let us evaluate both objectives on the same generations\. Because scores can be influenced by proxy optimization, response length, category, and repetition\([Gao et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib28);[Singhal et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib29);[Dubois et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib30);[Zheng et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib31);[Lambert et al\., 2025](https://arxiv.org/html/2609.26929#bib.bib1);[Welleck et al\., 2020](https://arxiv.org/html/2609.26929#bib.bib32)\), we report length and repetition alongside reward scores and re\-decode with repetition control\. We show that these checks are important for interpreting whether two objectives are genuinely compatible\.

## 3Background

We align to two objectives at a time and always fix the first as helpfulness\.

#### Preference data and conflict\.

A*response pair*\(x,yi,yj\)\(x,y\_\{i\},y\_\{j\}\)is a promptxxwith two responses\. Each objectiveooscores responses; the higher\-scoring response is preferred, giving a*preference pair*\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\)with preferredywy\_\{w\}and dispreferredyly\_\{l\}\(ties are dropped\)\. One response pair thus yields one preference pair*per*objective, and the two need not agree\. When two objectives prefer different responses for the same pair, they are in*conflict*\([Li et al\., 2025](https://arxiv.org/html/2609.26929#bib.bib2)\)\. Over a dataset, the fraction of jointly labelled pairs that conflict is the*preference\-pair conflict rate*\. It is a property of the data, measures how much the two objectives disagree, and is computed before any training\.

#### SFT reference model\.

Preference optimization does not start from the pretrained model directly: it starts from a*reference model*πsft\\pi\_\{\\text\{sft\}\}, obtained by*supervised fine\-tuning*\(SFT\), i\.e\. plain next\-token likelihood training on demonstration responses \(here, following[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib5), the preferred responses of the helpfulness preference pairs\)\. SFT uses no preference signal: it never sees the dispreferred response and never sees the second objective\. This one model plays three roles in what follows\. It initializes every MODPO run; it is the fixed anchor MODPO keeps the trained policy close to \(viaβ\\betabelow\); and it is the model whose generations we score before training in §[4\.1](https://arxiv.org/html/2609.26929#S4.SS1)\. Because all trained models share it as a starting point, it is also the baseline that their scores are read against: a model that already answers in the dataset’s domain and format, but has not yet been optimized for either objective of a pair\.

#### DPO\.

DPO treats a model as a policyπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)and trains it against the reference policyπsft\\pi\_\{\\text\{sft\}\}, which stays frozen\([Rafailov et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib16)\)\. WritingΔθ​\(x,y\)=log⁡πθ​\(y∣x\)−log⁡πsft​\(y∣x\)\\Delta\_\{\\theta\}\(x,y\)=\\log\\pi\_\{\\theta\}\(y\\mid x\)\-\\log\\pi\_\{\\text\{sft\}\}\(y\\mid x\)and the preference margindθ=Δθ​\(x,yw\)−Δθ​\(x,yl\)d\_\{\\theta\}=\\Delta\_\{\\theta\}\(x,y\_\{w\}\)\-\\Delta\_\{\\theta\}\(x,y\_\{l\}\), the loss is

ℒDPO=−𝔼\(x,yw,yl\)​\[log⁡σ⁡\(β​dθ\)\],\\mathcal\{L\}\_\{\\text\{DPO\}\}=\-\\,\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\}\\big\[\\log\\sigma\(\\beta\\,d\_\{\\theta\}\)\\big\],\(1\)whereσ\\sigmais the logistic function andβ\\betalimits how far the policy drifts from the reference\. DPO optimizes one objective\.

#### MODPO\.

MODPO trains one policy for a weighted mix of two objectives\([Zhou et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib5)\)\. An*objective weight*w∈\(0,1\]w\\in\(0,1\]sets the balance: the preference pairs come from the first objective \(helpfulness\), while the second enters through a marginmϕ=rϕ​\(x,yw\)−rϕ​\(x,yl\)m\_\{\\phi\}=r\_\{\\phi\}\(x,y\_\{w\}\)\-r\_\{\\phi\}\(x,y\_\{l\}\)from a reward modelrϕr\_\{\\phi\}trained on it\. The DPO logit becomes

z=1w​\(β​dθw−\(1−w\)​mϕ\),z=\\tfrac\{1\}\{w\}\\big\(\\beta\\,d\_\{\\theta\_\{w\}\}\-\(1\-w\)\\,m\_\{\\phi\}\\big\),\(2\)with loss−𝔼⁡\[log⁡σ⁡\(z\)\]\-\\mathbb\{E\}\[\\log\\sigma\(z\)\]\. Each weightwwgives a separate model at a*known*point on the trade\-off, so a set of weights is a steerable family\. Flipping the sign of the margin reverses which direction of the second objective is preferred \(e\.g\. shorter vs\. longer responses\)\.

#### LoRA\.

We train with Low\-Rank Adaptation\([Hu et al\., 2022](https://arxiv.org/html/2609.26929#bib.bib23)\): the pretrained weightsW0W\_\{0\}are frozen and a low\-rank updateΔ​W=B​A\\Delta W=BAis learned, applied ash=W0​u\+\(α/r\)​B​A​uh=W\_\{0\}u\+\(\\alpha/r\)\\,BA\\,u\. A trained model is then the base model plus a small adapter, so storing many objective\-weight variants is cheap and their adapters can be merged \(§[4](https://arxiv.org/html/2609.26929#S4)\)\.

## 4Method

We run two studies111[https://github\.com/esradonmez/modpo\-objective\-trade\-off](https://github.com/esradonmez/modpo-objective-trade-off)\. The*trade\-off study*\(RQ1\) asks whether pre\-training measurements predict which MODPO models improve two objectives together and which trade them off\. The*coverage study*\(RQ2\) asks how well trained models can be reused to reach new trade\-offs without retraining\. We report two kinds of score and keep them distinct:*dataset scores*, which ship with the data and are used only to build preference pairs, and*reward\-model scores*, which the evaluator assigns to generated responses and which we compare\.

### 4\.1Measuring objective compatibility before training

For each objective pair we compute two cheap measurements before any MODPO training\.

#### Preference\-pair conflict rate\.

Over the training data, we take the response pairs that yield a preference pair for*both*objectives and measure how often the two prefer different responses \(§[3](https://arxiv.org/html/2609.26929#S3)\)\. This is a direct count of how often the objectives disagree in the data\. A high rate means they pull toward opposite responses, which is what produces the conflicting gradients that can force a trade\-off\.

#### SFT reference\-model score correlation\.

We generate one response per evaluation prompt from the reference model, score it for both objectives, and take the Pearson correlation of the two score vectors\. The correlation is high when the reference model’s higher\-scoring responses for one objective are also its higher\-scoring responses for the other\. We compute it on*generated*responses with the same prompts and evaluator later used for the MODPO models, so it is comparable to their scores\. A correlation of the raw dataset scores would not be, since those come from a different annotator \(humans for HelpSteer, GPT\-4 for UltraFeedback\)\.

A high correlation and a low conflict rate both point to objectives that move together, so we expect them to agree\. The interesting cases are where they do not, and testing those is crucial for understanding alignment objectives\.

### 4\.2Probing the conflict rate

To test the conflict rate more directly, we split the jointly labelled pairs into a*non\-conflict*set \(rate00\) and a*conflict*set \(rate11\), keep the helpfulness preference pairs fixed, and retrain only the second\-objective margin reward model on each subset\. Because the subsets also differ in size and composition, we read these results as descriptive, not causal\.

### 4\.3Covering the trade\-off continuum at inference time

A*target*trade\-off is an objective weightw⋆w^\{\\star\}chosen after training\. We compare two training\-free reuse methods against a model trained directly atw⋆w^\{\\star\}\.

#### Nearest\-model selection\.

From a setGGof trained weights, pickwg=arg⁡minw∈G​\|w−w⋆\|w\_\{g\}=\\arg\\min\_\{w\\in G\}\|w\-w^\{\\star\}\|\(ties broken toward the lower weight\) and generate with that model\. We compare a*fine*set \(more weights\) and a*coarse*set \(fewer\)\.

#### Parameter merging\.

Using the fine set, take the closest lower and upper trained weightswℓ<w⋆<wuw\_\{\\ell\}<w^\{\\star\}<w\_\{u\}, setλ=\(w⋆−wℓ\)/\(wu−wℓ\)\\lambda=\(w^\{\\star\}\-w\_\{\\ell\}\)/\(w\_\{u\}\-w\_\{\\ell\}\), and merge their LoRA adapters tensor by tensor,Θmerge=\(1−λ\)​Θℓ\+λ​Θu\\Theta\_\{\\text\{merge\}\}=\(1\-\\lambda\)\\,\\Theta\_\{\\ell\}\+\\lambda\\,\\Theta\_\{u\}\([Jang et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib15);[Rame et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib17)\)\. No new training is done\. Merging is an interpolation heuristic, not an optimization procedure forw⋆w^\{\\star\}: the MODPO objective atw⋆w^\{\\star\}is never evaluated, so nothing guarantees the merged adapter behaves like one trained there\. How closely it does is what we measure\.

## 5Experimental Setup

#### Datasets\.

We use two datasets that score more than two objectives so that several objective pairs share one dataset\. HelpSteer\([Wang et al\., 2024c](https://arxiv.org/html/2609.26929#bib.bib3)\)provides*human*\-assigned scores \(00–44\) for helpfulness, correctness, coherence, complexity, and verbosity over10,45910\{,\}459prompts\. UltraFeedback\([Cui et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib24)\)provides*GPT\-4*\-assigned scores \(11–55\) for helpfulness, honesty, instruction\-following, and truthfulness over63,96763\{,\}967prompts\. The coverage study uses PKU\-SafeRLHF\-10K\([Ji et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib26);[Dai et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib27)\)with the helpfulness/harmlessness pair \(an objective pair where the trade\-off is real and most consequential\) reproducing the MODPO safety setup so our numbers can be checked against[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib5)\.

#### Objectives\.

We fix*helpfulness*as the first objective in every pair and choose the second objectives to span ones that plausibly reinforce helpfulness and ones that plausibly compete with it: from HelpSteer,*correctness*and*coherence*should reinforce it,*complexity*is more independent, and*verbosity*can compete with it \(a concise answer may still be helpful\); from UltraFeedback,*honesty*,*instruction\-following*, and*truthfulness*are broadly aligned with helpfulness but need not coincide with it response by response\. This gives four HelpSteer pairs and three UltraFeedback pairs; for HelpSteer verbosity we train once toward higher and once toward lower verbosity \(positive / negative margin sign\), for eight settings in all\. The spread lets us test the measurements on pairs expected to agree and pairs expected to conflict, rather than on a single case\.

#### Models and training\.

We useLlama\-2\-7b\([Touvron et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib33)\)for HelpSteer andAlpaca\-7b\([Taori et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib34)\)for UltraFeedback and PKU\-SafeRLHF, all trained with LoRA\. Following[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib5), each setting trains a margin reward model on the second objective and one MODPO model per objective weight\. In the trade\-off study we additionally train an SFT reference model on the preferred responses of the helpfulness pairs and use weightsw∈\{0\.1,0\.2,0\.4,0\.6,0\.8,\(0\.9\)\}w\\in\\\{0\.1,0\.2,0\.4,0\.6,0\.8,\(0\.9\)\\\}\. The coverage study reuses theAlpaca\-7bcheckpoint as its reference model without further SFT, again following[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib5), and trains the*fine*weight set\{0\.0,0\.2,0\.4,0\.6,0\.8,1\.0\}\\\{0\.0,0\.2,0\.4,0\.6,0\.8,1\.0\\\}, the*coarse*set\{0\.0,0\.5,1\.0\}\\\{0\.0,0\.5,1\.0\\\}, and direct models at the target weightsw⋆∈\{0\.1,0\.3,0\.5,0\.7,0\.9\}w^\{\\star\}\\in\\\{0\.1,0\.3,0\.5,0\.7,0\.9\\\}; atw=0\.0w\{=\}0\.0all weight is on harmlessness, so the DPO margin adapter is loaded instead of a MODPO adapter\. We report the*2048\-token*training\-length limit as primary \(it retains almost all pairs\) and contrast a*512\-token*limit that retains far fewer\. Training uses learning rate10−410^\{\-4\},β=0\.1\\beta=0\.1, AdamW with a cosine schedule, and a single seed per model; full hyperparameters are in Appendix[A](https://arxiv.org/html/2609.26929#A1)\.

#### Evaluation\.

We generate one greedy response per prompt on400400held\-out prompts \(trade\-off study\) or700700response pairs \(coverage study\)\. Trade\-off responses are scored per objective with ArmoRM\([Wang et al\., 2024b](https://arxiv.org/html/2609.26929#bib.bib4)\); PKU responses are scored with the Beaver reward \(helpfulness, higher better\) and cost \(harmlessness, lower better\) models\([Dai et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib27)\)\. We report mean scores, mean response length in words, and the share of responses with a repeated44\-gram, with95%95\\%paired bootstrap confidence intervals\([Efron and Tibshirani, 1993](https://arxiv.org/html/2609.26929#bib.bib35)\)\. Because greedy decoding can produce long, repetitive text, we additionally re\-decode the 2048\-token models with a repetition penalty and44\-gram blocking to test whether score changes track length and repetition\.

Figure 1:The two pre\-training measurements for all seven objective pairs \(2048\-token limit; both verbosity margin signs are plotted, giving eight points\)\. Objectives that move together sit toward the bottom right \(high correlation, low conflict\); HelpSteer correctness and coherence are the clearest such cases, while HelpSteer verbosity toward concision \(*HS verb\. neg\.*\) has by far the highest conflict rate\. The two measurements agree for the three UltraFeedback pairs too: high correlation and a conflict rate below every HelpSteer pair but correctness and coherence\. Verbosity is where the two measurements part company; UltraFeedback is where the*results*will\.Objective pairCorr\.ConflictBoth↑\\uparrow?*HelpSteer*correctness0\.9810\.9810\.0270\.027allwwcoherence0\.9110\.9110\.0670\.067allwwcomplexity0\.0950\.0950\.2020\.2022 of 5wwverbosity \(\+\+\)−0\.162\-0\.1620\.2260\.226w=0\.9w\{=\}0\.9verbosity \(−\-\)0\.1620\.1620\.7740\.774none†*UltraFeedback*honesty0\.8540\.8540\.1810\.181noneinstruction\-following0\.8790\.8790\.1640\.164nonetruthfulness0\.8100\.8100\.1920\.192noneTable 1:Pre\-training measurements \(SFT reference\-model score correlation and preference\-pair conflict rate, 2048\-token limit\) and at which objective weights*both*objectives improve over the reference under greedy decoding at the same limit\.†verbosity\(−\-\) improves both only atw=0\.4w\{=\}0\.4under the 512\-token limit\. Appendix[B](https://arxiv.org/html/2609.26929#A2)gives the 512\-token measurements and Appendix[C](https://arxiv.org/html/2609.26929#A3)the underlying scores\. HelpSteer follows the measurements; the high\-correlation UltraFeedback pairs do not\.

## 6Results

### 6\.1When can one model satisfy two objectives? \(RQ1\)

#### The measurements work for HelpSteer\.

[Table1](https://arxiv.org/html/2609.26929#S5.T1)and Figure[1](https://arxiv.org/html/2609.26929#S5.F1)display the two pre\-training measurements; Figure[2](https://arxiv.org/html/2609.26929#S6.F2)shows the resulting MODPO scores for one pair from each dataset\. For HelpSteer the measurements match both intuition and each other\.*Correctness*and*coherence*have very high correlations \(0\.980\.98,0\.910\.91\) and low conflict rates \(0\.030\.03,0\.070\.07\)\. Under the 2048\-token limit, MODPO improves*both*objectives at every weight \(the “allww” entries in[Table1](https://arxiv.org/html/2609.26929#S5.T1)\): in Figure[2](https://arxiv.org/html/2609.26929#S6.F2)\(left\) every 2048\-token model sits up and to the right of the reference, with correctness rising from0\.6360\.636to0\.7060\.706atw=0\.1w\{=\}0\.1as helpfulness rises from0\.6210\.621to0\.7020\.702; the 512\-token condition shows weaker and less consistent gains\.*Complexity*has a weak correlation \(0\.0950\.095\) and moderate conflict \(0\.2020\.202\): it improves at every weight but helpfulness only at two, a partial match\.*Verbosity toward concision*has the highest conflict rate \(0\.7740\.774\) and shows a clean trade\-off: low weights favor concision, high weights favor helpfulness, exactly as the conflict rate predicts\.

#### The measurements fail for UltraFeedback\.

The three UltraFeedback pairs have high correlations \(0\.810\.81–0\.880\.88\) and only moderate conflict, yet every model*degrades*performance on the second objective at every weight and never improves both\. Figure[2](https://arxiv.org/html/2609.26929#S6.F2)\(right\) highlights the contrast: every UltraFeedback model scores*below*the reference on truthfulness\. A high pre\-training correlation did not deliver the joint improvement it did for HelpSteer\.

Figure 2:MODPO scores \(ArmoRM mean over 400 prompts\) for an objective pair that MODPO can satisfy jointly and one it cannot, at two training\-length limits \(512, 2048\)\. Empty squares mark the reference model; labels are objective weightsww\.Left:HelpSteer correctness\. Under the 2048\-token limit every MODPO model improves*both*objectives relative to its reference model; the 512\-token condition shows weaker and less consistent gains\.Right:UltraFeedback truthfulness\. Despite a high pre\-training correlation, every model scores*lower*on truthfulness than the reference\. The gap between the 512 and 2048 clouds shows how much the training\-length limit moves the scores\.
#### The apparent conflict is strongly confounded by length and repetition\.

Almost all MODPO models generate much longer and more repetitive responses than the reference \(up to∼970\{\\sim\}970words with a repeated\-44\-gram share above0\.950\.95for UltraFeedback\)\. When we re\-decode the same 2048\-token models with repetition control, mean length roughly halves and the repeated\-44\-gram share drops by6060–7070points \([Table2](https://arxiv.org/html/2609.26929#S6.T2);38\.738\.7–81\.681\.6across individual pairs, Appendix[D](https://arxiv.org/html/2609.26929#A4)\)\. For HelpSteer this barely moves the scores\. For UltraFeedback both objective scores rise by\+0\.10\+0\.10to\+0\.15\+0\.15, and the resulting scores are substantially more consistent with the high reference\-model correlations\. This does not isolate the cause: re\-decoding changes length, repetition, wording, and content together, and the same reward model scores both conditions, so we cannot separate a reward\-model bias against repetition from genuinely better responses\. What it does show is that the apparent UltraFeedback conflict is strongly confounded by generation length and repetition, and was driven at least partly by decoding behavior rather than by objective incompatibility alone; an independent evaluator would be needed to settle this\. A plausible contributing factor is that UltraFeedback’s GPT\-4 scores track length more strongly than HelpSteer’s human scores, and that HelpSteer separates verbosity into its own objective\. The training\-length limit adds to this: keeping more pairs shortens HelpSteer generations but lengthens UltraFeedback ones, which is why the same limit helps one dataset and hurts the other \(Figure[2](https://arxiv.org/html/2609.26929#S6.F2)\)\.

Δ\\DeltahelpΔ\\Deltaobj2Δ\\DeltawordsΔ\\DeltarepHelpSteer \(mean\)−0\.001\-0\.001−0\.008\-0\.008−48\.8%\-48\.8\\%−69\.7\-69\.7UltraFeedback \(mean\)\+0\.137\+0\.137\+0\.125\+0\.125−58\.9%\-58\.9\\%−62\.7\-62\.7Table 2:Change in scores, response length, and repeated\-44\-gram share \(percentage points\) when the 2048\-token models are re\-decoded with repetition control\. HelpSteer is unmoved; UltraFeedback rises sharply, showing its greedy\-decoding scores were confounded by length and repetition\.
#### The margin sign steers which direction of an objective wins\.

By Eq\. \([2](https://arxiv.org/html/2609.26929#S3.E2)\), flipping the sign of the margin reverses which direction of the second objective is preferred\. For verbosity this is the difference between steering toward longer and toward shorter responses\. This is the explicit, per\-objective control that steerable pluralism calls for\. Figure[3](https://arxiv.org/html/2609.26929#S6.F3)shows the positive sign: MODPO reliably raises verbosity but helps helpfulness at a single weight \(w=0\.9w\{=\}0\.9, the one setting where this pair improves both\), while the negative sign gives the clean trade\-off above\. The negative sign is also the pair with the highest conflict rate, so its trade\-off is what the measurement predicts\. Direction is therefore a second control knob, orthogonal to the weight\.

Figure 3:HelpSteer helpfulness–verbosity under the positive margin sign \(steering toward higher verbosity\)\. MODPO raises verbosity at every weight but improves helpfulness only atw=0\.9w\{=\}0\.9, where helpfulness receives the greatest weight and verbosity the least\. Flipping the sign steers toward concision and yields the trade\-off in the text\.
#### Splitting on conflict changes the outcome but does not decide it\.

Training the margin reward model only on conflicting pairs shifts the results but not all the way to a trade\-off\. For UltraFeedback instruction\-following \(the lowest conflict rate of the three UltraFeedback pairs; 512\-token limit\), the conflict set comes*close*to a trade\-off but stops short, because the pair still has a high correlation\. HelpSteer verbosity toward concision points the same way from the other side: the split is computed in the dataset’s score direction, so under the negative margin sign the “conflict” set is the one whose pairs*agree*with helpfulness, and it duly improves helpfulness at every weight while worsening concision at the higher ones\. Conflict alone is thus not enough; the correlation still matters, in line with[Li et al\. \(2025\)](https://arxiv.org/html/2609.26929#bib.bib2)\. Full scores are in Appendix[E](https://arxiv.org/html/2609.26929#A5)\.

### 6\.2Covering the continuum without retraining \(RQ2\)

Across the trained models, mean reward increases and mean harmlessness worsens aswwrises, an empirical trade\-off frontier in this single\-seed run \(Figure[4](https://arxiv.org/html/2609.26929#S6.F4)\), consistent with[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.26929#bib.bib5)\. No single model dominates, so reaching a new target is non\-trivial\.[Table3](https://arxiv.org/html/2609.26929#S6.T3)and Figure[5](https://arxiv.org/html/2609.26929#S6.F5)report the mean absolute difference from a model trained directly at the target\. Parameter merging has the smallest average gap when reward and cost are considered together \(combined mean1\.141\.14, vs\.1\.691\.69for fine selection and2\.402\.40for coarse\), although it is not consistently closest at individual target weights and is slightly worse than fine selection on mean reward gap \(1\.111\.11vs\.1\.061\.06\)\. Its95%95\\%confidence interval for the paired difference includes zero in55of1010comparisons, vs\.2/102/10for fine selection: evidence of no detected difference, not of equivalence\. Merging is an interpolation heuristic, not an optimization procedure for the target weight: its score trajectory did not reach the directly trained model atw⋆=0\.9w^\{\\star\}\{=\}0\.9, producing a large reward gap, while the selection methods instead spike at thew⋆=0\.1w^\{\\star\}\{=\}0\.1endpoint\. Nearest\-model selection is limited by design: it picks on weight distance, not on scores, so the nearest weight is often not nearest in reward or cost\.Both methods cover the continuum usefully, but neither reliably reproduces a directly trained model\.

Figure 4:Reward vs\. cost for the fine weight set on PKU\-SafeRLHF \(labels are objective weights; thew=0w\{=\}0point is the DPO margin adapter, since atw=0w\{=\}0all weight is on harmlessness\)\. Higher reward is more helpful, lower cost is more harmless\. Across these trained models mean reward increases and mean cost worsens withww\(a single\-seed run\), so no single model dominates and reaching a new target requires selecting or merging models\.Figure 5:Absolute score difference from a directly trained target model at each target weight \(lower is closer\)\. Parameter merging \(interpolated model, dashed\) has the smallest combined gap but is not uniformly best: it is slightly worse than fine selection on mean reward, the selection methods spike at thew⋆=0\.1w^\{\\star\}\{=\}0\.1endpoint, and merging shows a large reward gap atw⋆=0\.9w^\{\\star\}\{=\}0\.9\. Merging is an interpolation heuristic, not an optimization procedure for the target weight\.FineCoarseMergeReward gap \(mean∗\)1\.06\\mathbf\{1\.06\}2\.012\.011\.111\.11Cost gap \(mean∗\)2\.322\.322\.792\.791\.18\\mathbf\{1\.18\}CI incl\.002/102/101/8†1/8^\{\\dagger\}𝟓/𝟏𝟎\\mathbf\{5/10\}Table 3:Coverage study: mean absolute difference from direct training in Beaver reward and cost, and how many comparisons have a95%95\\%confidence interval for the paired difference that includes zero \(no detected difference, not equivalence\)\. Fine selection has the smallest mean reward gap; merging the smallest cost gap and combined gap\.∗mean over target weights excludingw⋆=0\.5w^\{\\star\}\{=\}0\.5, where the coarse set already contains the direct model\.†the coarse count is out of eight for the same reason: its two comparisons atw⋆=0\.5w^\{\\star\}\{=\}0\.5are zero by construction\. Per\-target values are in Appendix[F](https://arxiv.org/html/2609.26929#A6)\.

## 7Discussion

#### Cheap measurements predict compatibility, but only for trustworthy data \(RQ1\)\.

The conflict rate and the reference\-model correlation predict MODPO’s behavior well on human\-annotated HelpSteer: aligned objectives \(correctness, coherence\) improve together, and the most disagreeing pair \(concision vs\. helpfulness\) trades off\. They fail on AI\-annotated UltraFeedback, but the failure is instructive\. Rather than a clean objective incompatibility, the reward\-model scores were strongly confounded by length and repetition, and once we controlled for those the results became substantially more consistent with the measurements\. We cannot conclude from this alone that the objectives are compatible; that would take an independent evaluator\. For steerable pluralism this is a warning: whether two objectives*look*compatible can depend heavily on how outputs are decoded and scored, so a system that decided which objectives to serve from a cheap measurement alone could draw the wrong line between compatible and conflicting objectives\.

#### Recipe for RQ1\.

To serve two objectives together, pick a pair with high correlation and low conflict, use the training\-length limit that keeps the most pairs, and then*check*whether the models generate very long or repetitive text and whether a different decoding setting changes the scores\. To build a steerable trade\-off, pick a pair with low correlation and high conflict, and use the margin sign to choose which direction of the objective to steer toward\.

#### Coverage is cheap but approximate \(RQ2\)\.

A continuum of trade\-offs can be covered by reusing a few trained models: parameter merging has the smallest combined gap from direct training and nearest\-model selection is a simple fallback\. Neither matches direct training: selection ignores scores, and merging is an interpolation heuristic rather than an optimization procedure for the target weight\. This mirrors the trade\-off in the wider merging literature\([Jang et al\., 2024](https://arxiv.org/html/2609.26929#bib.bib15);[Rame et al\., 2023](https://arxiv.org/html/2609.26929#bib.bib17)\): merging buys cheap coverage of the preference space at some cost in fidelity\.

#### Recipe for RQ2\.

Train a family of MODPO models across a range of weights\. If it already covers the trade\-offs you need, use nearest\-model selection; otherwise consider filling the gaps by merging the two nearest models\. When a target’s exact behavior matters, train directly at that weight\.

#### When is a trade\-off the right frame?

Not every pair of objectives needs a steerable trade\-off\. Different stakeholders weight objectives differently, but when the objectives themselves reinforce each other \(as correctness and helpfulness do here\), a single well\-trained model serves both, and offering a dial would be a false choice\. A trade\-off should be exposed only when the objectives genuinely compete for the same responses, which is what a high conflict rate, read together with the correlation and after controlling for length and repetition confounds, is meant to detect\. This suggests a modest but useful role for the measurements in pluralistic systems: not to make the choice automatically, but to flag which pairs are candidates for steerable trade\-offs and which are better served jointly, leaving the normative choice of*which*trade\-offs to expose to the people affected\([Dönmez and Falenska, 2026](https://arxiv.org/html/2609.26929#bib.bib36)\)\.

#### Takeaway\.

An objective weight is a genuine steering dial, but it is not a promise: the same weight behaves differently depending on the data kept and the decoding used\. Steerable pluralism with MODPO therefore needs its models characterized empirically by their generated length, repetition, and scored behavior, not just by their nominal weights\.

## 8Conclusion

We studied MODPO as a tool for steerable pluralistic alignment \(one model family that can be steered across a continuum of objective trade\-offs to suit different stakeholders\) and asked when it can satisfy two objectives at once and how to cover the continuum cheaply\. Across seven objective pairs, two cheap pre\-training measurements predicted MODPO’s behavior for human\-annotated HelpSteer but not for AI\-annotated UltraFeedback, where response length and repetition strongly confounded the reward\-model scores; controlling for them made the results much more consistent with the measurements, though an independent evaluator would be needed to confirm compatibility\. Whether two objectives can be served jointly is therefore not something to read off a single number: it depends on the data, the decoding, and the evaluator together\. For covering new trade\-offs without retraining, nearest\-model selection and parameter merging both help, but neither reliably matches direct training\. We hope the resulting recipes help practitioners build models that serve diverse stakeholder preferences, and measure honestly whether they do\.

## Limitations

Our study has several limitations\. \(i\) We steer only*two*objectives at a time \(helpfulness and a second objective\); behavior with three or more as well as with different pairings may differ, and serving many stakeholders often means balancing many objectives at once\. \(ii\) The coverage study reuses models within a fixed trained set and a fixed set of target weights, and parameter merging is tested only at the midpointλ=0\.5\\lambda=0\.5; other targets and merge positions are untested\. \(iii\) All trained\-model results use a*single*seed, so we cannot quantify seed variation\. \(iv\) All scores come from reward models, a proxy for human judgment\. Every comparison we draw is within\-evaluator: each MODPO model is read against the SFT reference trained on the same data and scored by the same model, with paired confidence intervals, so a constant evaluator bias shifts the score levels rather than the direction of a comparison\. What this does not cover is absolute level and transfer, since ArmoRM is trained on data that includes HelpSteer and UltraFeedback and the Beaver models on the BeaverTails family that includes PKU\-SafeRLHF\. The one result that turns on the evaluator itself is the re\-decoding comparison \(§[6\.1](https://arxiv.org/html/2609.26929#S6.SS1)\), where generation and scoring change together; we mark it as unsettled there\. \(v\) We use one 7B base model per dataset, so results may transfer differently to other models or sizes\. \(vi\) The conflict\-split experiments change the training\-set size and composition, so they are suggestive of a conflict\-rate effect rather than causal\. Promising next steps are a second \(human or LLM\-judge\) evaluator to confirm objective scores independently of length and repetition, which matters most for UltraFeedback; a size\-controlled study of the conflict rate; and a selection rule 2that uses reward and cost scores rather than weight distance alone\.

## Ethics Statement

This work studies how to balance competing objectives in a single steerable model, using only publicly available datasets and models\. Part of our motivation is pluralistic: making the trade\-off between objectives \(which different stakeholders would weight differently\)*explicit*and inspectable, rather than hiding it inside one aggregate reward\. We note two risks\. First, steerability cuts both ways: the same objective weight that balances helpfulness against harmlessness can be pushed toward helpfulness to produce less safe outputs \([Figure4](https://arxiv.org/html/2609.26929#S6.F4)\), and our PKU\-SafeRLHF setup includes adversarial prompts\. Deciding*which*trade\-offs a deployed system should expose is a governance question beyond our scope\. Second, reward\-model scores carry the biases of their training data and, as we show, of response length and repetition, so they should not be read as ground\-truth measures of any human value\([Ghazaryan and Dönmez, 2026](https://arxiv.org/html/2609.26929#bib.bib37)\)\. We report these confounds openly and recommend human or independent evaluation before deployment\.

## Acknowledgments

We acknowledge the support of the Ministerium für Wissenschaft, Forschung und Kunst BadenWürttemberg \(MWK, Ministry of Science, Research and the Arts Baden\-Württemberg under Az\. 33\-7533\-9 19/54/5\) in Künstliche Intelligenz & Gesellschaft: Reflecting Intelligent Systems for Diversity, Demography and Democracy \(IRIS3D\) and the support by the Interchange Forum for Reflecting on Intelligent Systems \(IRIS\) at the University of Stuttgart\. We would like to thank the Institute of Natural Language Processing \(IMS\) for providing computational support for this project\.

## References

- M\. A\. Bakker, M\. J\. Chadwick, H\. R\. Sheahan, M\. H\. Tessler, L\. Campbell\-Gillingham, J\. Balaguer, N\. McAleese, A\. Glaese, J\. Aslanides, M\. M\. Botvinick, and C\. SummerfieldFine\-tuning language models to find agreement among humans with diverse preferences\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088,[Link](https://dl.acm.org/doi/10.5555/3600270.3603036)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px3.p1.1)\.
- Chakrabortyet al\.\(2024\)S\. Chakraborty, J\. Qiu, H\. Yuan, A\. Koppel, D\. Manocha, F\. Huang, A\. Bedi, and M\. WangMaxMin\-RLHF: alignment with diverse human preferences\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=8tzjEMF0Vq)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1)\.
- Chronopoulouet al\.\(2023\)A\. Chronopoulou, M\. Peters, A\. Fraser, and J\. DodgeAdapterSoup: weight averaging to improve generalization of pretrained language models\.InFindings of the Association for Computational Linguistics: EACL 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 2054–2063\.External Links:[Link](https://aclanthology.org/2023.findings-eacl.153/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.153)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1)\.
- Conitzeret al\.\(2024\)V\. Conitzer, R\. Freedman, J\. Heitzig, W\. H\. Holliday, B\. M\. Jacobs, N\. Lambert, M\. Mosse, E\. Pacuit, S\. Russell, H\. Schoelkopf, E\. Tewolde, and W\. S\. ZwickerPosition: social choice should guide AI alignment in dealing with diverse human feedback\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 9346–9360\.External Links:[Link](https://proceedings.mlr.press/v235/conitzer24a.html)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1)\.
- Cuiet al\.\(2024\)G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin, Z\. Liu, and M\. SunULTRAFEEDBACK: boosting language models with scaled ai feedback\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.External Links:[Link](https://dl.acm.org/doi/10.5555/3692070.3692454)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px1.p1.1)\.
- Daiet al\.\(2024\)J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. YangSafe RLHF: safe reinforcement learning from human feedback\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TyFrPOKYXw)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px4.p1.1)\.
- Dönmez and Falenska \(2026\)E\. Dönmez and A\. FalenskaStructuring the space of sociotechnical alignment\.External Links:2607\.01250,[Link](https://arxiv.org/abs/2607.01250)Cited by:[§7](https://arxiv.org/html/2609.26929#S7.SS0.SSS0.Px5.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, P\. Liang, and T\. HashimotoLength\-controlled alpacaeval: a simple debiasing of automatic evaluators\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=CybBmzWBX0)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Efron and Tibshirani \(1993\)B\. Efron and R\. J\. TibshiraniAn introduction to the bootstrap\.Chapman & Hall/CRC,New York\.Cited by:[Appendix A](https://arxiv.org/html/2609.26929#A1.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px4.p1.1)\.
- Fenget al\.\(2024\)S\. Feng, T\. Sorensen, Y\. Liu, J\. Fisher, C\. Y\. Park, Y\. Choi, and Y\. TsvetkovModular pluralism: pluralistic alignment via multi\-LLM collaboration\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4151–4171\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.240/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.240)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.External Links:[Link](https://dl.acm.org/doi/10.5555/3618408.3618845)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Ghazaryan and Dönmez \(2026\)G\. Ghazaryan and E\. DönmezMisaligned by reward: socially undesirable preferences in llms\.External Links:2605\.05003,[Link](https://arxiv.org/abs/2605.05003)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1),[Ethics Statement](https://arxiv.org/html/2609.26929#Sx2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2609.26929#S3.SS0.SSS0.Px5.p1.1)\.
- Ilharcoet al\.\(2023\)G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, L\. Schmidt, H\. Hajishirzi, and A\. FarhadiEditing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2212.04089)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1)\.
- Janget al\.\(2024\)J\. Jang, S\. Kim, B\. Y\. Lin, Y\. Wang, J\. Hessel, L\. Zettlemoyer, H\. Hajishirzi, Y\. Choi, and P\. AmmanabroluPersonalized soups: personalized large language model alignment via post\-hoc parameter merging\.InAdaptive Foundation Models: Evolving AI for Personalized and Efficient Learning,External Links:[Link](https://openreview.net/forum?id=EMrnoPRvxe)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1),[§4\.3](https://arxiv.org/html/2609.26929#S4.SS3.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.26929#S7.SS0.SSS0.Px3.p1.1)\.
- Jiet al\.\(2023\)J\. Ji, M\. Liu, J\. Dai, X\. Pan, C\. Zhang, C\. Bian, B\. Chen, R\. Sun, Y\. Wang, and Y\. YangBEAVERTAILS: towards improved safety alignment of llm via a human\-preference dataset\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.External Links:[Link](https://dl.acm.org/doi/abs/10.5555/3666122.3667194)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px1.p1.1)\.
- Kirket al\.\(2024\)H\. R\. Kirk, A\. Whitefield, P\. Röttger, A\. M\. Bean, K\. Margatina, R\. Mosquera, J\. M\. Ciro, M\. Bartolo, A\. Williams, H\. He, B\. Vidgen, and S\. A\. HaleThe PRISM alignment dataset: what participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=DFr5hteojx)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1755–1797\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.96/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96),ISBN 979\-8\-89176\-195\-7Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Liet al\.\(2025\)M\. Li, Y\. Zhang, W\. Wang, W\. Shi, Z\. Liu, F\. Feng, and T\. ChuaSelf\-improvement towards Pareto optimality: mitigating preference conflicts in multi\-objective alignment\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11010–11031\.External Links:[Link](https://aclanthology.org/2025.findings-acl.574/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.574),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2609.26929#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2609.26929#S3.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.26929#S6.SS1.SSS0.Px5.p1.1)\.
- Mudgalet al\.\(2024\)S\. Mudgal, J\. Lee, H\. Ganapathy, Y\. Li, T\. Wang, Y\. Huang, Z\. Chen, H\. Cheng, M\. Collins, T\. Strohman, J\. Chen, A\. Beutel, and A\. BeiramiControlled decoding from language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.External Links:[Link](https://dl.acm.org/doi/10.5555/3692070.3693554)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1)\.
- Plank \(2022\)B\. PlankThe “problem” of human label variation: on ground truth in data, modeling and evaluation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 10671–10682\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.731/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.731)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.26929#S1.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px3.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.p2.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.26929#S3.SS0.SSS0.Px3.p1.1)\.
- Rameet al\.\(2023\)A\. Rame, G\. Couairon, C\. Dancette, J\. Gaya, M\. Shukor, L\. Soulier, and M\. CordRewarded soups: towards pareto\-optimal alignment by interpolating weights fine\-tuned on diverse rewards\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=lSbbC2VyCu)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1),[§4\.3](https://arxiv.org/html/2609.26929#S4.SS3.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.26929#S7.SS0.SSS0.Px3.p1.1)\.
- Shiet al\.\(2024\)R\. Shi, Y\. Chen, Y\. Hu, A\. Liu, H\. Hajishirzi, N\. A\. Smith, and S\. S\. DuDecoding\-time language model alignment with multiple objectives\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385,[Link](https://dl.acm.org/doi/10.5555/3737916.3739465)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1)\.
- Singhalet al\.\(2024\)P\. Singhal, T\. Goyal, J\. Xu, and G\. DurrettA long way to go: investigating length correlations in RLHF\.InFirst Conference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2310.03716)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Sorensenet al\.\(2024\)T\. Sorensen, J\. Moore, J\. Fisher, M\. L\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri, T\. Althoff, and Y\. ChoiPosition: a roadmap to pluralistic alignment\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 46280–46302\.External Links:[Link](https://proceedings.mlr.press/v235/sorensen24a.html)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.p1.1),[§1](https://arxiv.org/html/2609.26929#S1.p2.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px1.p1.1)\.
- Taoriet al\.\(2023\)R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. HashimotoStanford alpaca: an instruction\-following LLaMA model\.GitHub\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px3.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.External Links:[Link](https://arxiv.org/abs/2307.09288)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2024a\)H\. Wang, Y\. Lin, W\. Xiong, R\. Yang, S\. Diao, S\. Qiu, H\. Zhao, and T\. ZhangArithmetic control of LLMs for diverse user preferences: directional preference alignment with multi\-objective rewards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 8642–8655\.External Links:[Link](https://aclanthology.org/2024.acl-long.468/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.468)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Wanget al\.\(2024b\)H\. Wang, W\. Xiong, T\. Xie, H\. Zhao, and T\. ZhangInterpretable preferences via multi\-objective reward modeling and mixture\-of\-experts\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10582–10592\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.620/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.620)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024c\)Z\. Wang, Y\. Dong, J\. Zeng, V\. Adams, M\. N\. Sreedhar, D\. Egert, O\. Delalleau, J\. Scowcroft, N\. Kant, A\. Swope, and O\. KuchaievHelpSteer: multi\-attribute helpfulness dataset for SteerLM\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3371–3384\.External Links:[Link](https://aclanthology.org/2024.naacl-long.185/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.185)Cited by:[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px1.p1.1)\.
- Wellecket al\.\(2020\)S\. Welleck, I\. Kulikov, S\. Roller, E\. Dinan, K\. Cho, and J\. WestonNeural text generation with unlikelihood training\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SJeYe0NtvH)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Wortsmanet al\.\(2022\)M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith, and L\. SchmidtModel soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),pp\. 23965–23998\.External Links:[Link](https://arxiv.org/abs/2203.05482)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px4.p1.1)\.
- Xuet al\.\(2026\)Z\. Xu, Y\. Tong, X\. Zhang, J\. Zhou, and X\. WangUnderstanding conflicts in multi\-objective alignment through reward consistency\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 5450–5472\.External Links:[Link](https://aclanthology.org/2026.findings-acl.269/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.269),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)R\. Yang, X\. Pan, F\. Luo, S\. Qiu, H\. Zhong, D\. Yu, and J\. ChenRewards\-in\-context: multi\-objective alignment of foundation models with dynamic preference adjustment\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.External Links:[Link](https://dl.acm.org/doi/abs/10.5555/3692070.3694392)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.External Links:[Link](https://dl.acm.org/doi/10.5555/3666122.3668142)Cited by:[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px5.p1.1)\.
- Zhouet al\.\(2024\)Z\. Zhou, J\. Liu, J\. Shao, X\. Yue, C\. Yang, W\. Ouyang, and Y\. QiaoBeyond one\-preference\-fits\-all alignment: multi\-objective direct preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10586–10613\.External Links:[Link](https://aclanthology.org/2024.findings-acl.630/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.630)Cited by:[§1](https://arxiv.org/html/2609.26929#S1.p2.1),[§2](https://arxiv.org/html/2609.26929#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.26929#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.26929#S3.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.26929#S5.SS0.SSS0.Px3.p1.1),[§6\.2](https://arxiv.org/html/2609.26929#S6.SS2.p1.1)\.

## Appendix ATraining and Evaluation Details

#### Data splits\.

HelpSteer uses the official95/595/5split \(48,88848\{,\}888training /2,4742\{,\}474validation preference pairs\); UltraFeedback has no official validation split, so we use a random90/1090/10split with seed00\(345,414345\{,\}414/38,38238\{,\}382pairs\)\. PKU\-SafeRLHF\-10K is split into9,0009\{,\}000training and1,0001\{,\}000validation response pairs\. Evaluation uses the first400400unique validation prompts for the trade\-off study and700700validation response pairs \(665665unique prompts\) for the coverage study\.

#### Training\.

All stages use learning rate10−410^\{\-4\}, KL coefficientβ=0\.1\\beta=0\.1, AdamW with a cosine schedule, weight decay0\.050\.05, and effective batch size88\(one example, eight gradient\-accumulation steps\); the checkpoint with the lowest validation loss is kept\. Trade\-off\-study LoRA uses rank/alpha/dropout16/32/0\.0516/32/0\.05onq\_proj,v\_projin bf16; the coverage study uses64/1/064/1/0onq,k,v,o\_projin fp16\. HelpSteer trains1,0001\{,\}000steps per stage; UltraFeedback trains300300\(SFT, margin\) and400400\(MODPO\); the coverage study trains33epochs per stage with a10241024\-token limit\. We use split seed00, training seed303303\(trade\-off\) or4242\(coverage\), and generation seed00, with a single seed per model\.

#### Generation\.

Primary decoding is greedy with no repetition penalty ornn\-gram blocking\. The trade\-off study generates up to40964096tokens \(15361536prompt /25602560response\); the coverage study up to10241024\. The repetition\-control setting adds repetition penalty1\.101\.10andno\_repeat\_ngram\_size=4=4\.

#### Confidence intervals\.

We use the percentile bootstrap over paired score differences with10,00010\{,\}000resamples, seed10001000, and95%95\\%intervals\([Efron and Tibshirani, 1993](https://arxiv.org/html/2609.26929#bib.bib35)\), paired by validation prompt \(trade\-off\) or response pair \(coverage\)\.

#### Compute\.

Training and evaluation used NVIDIA RTX A6000 GPUs for approximately1,2521\{,\}252GPU\-hours in total \(∼985\{\\sim\}985for training,∼267\{\\sim\}267for generation and scoring\)\.

## Appendix BObjective Compatibility Measurements

[Table4](https://arxiv.org/html/2609.26929#A2.T4)gives both pre\-training measurements at both training\-length limits, together with the data they are computed from\. The 512\-token limit keeps roughly a third of the HelpSteer pairs and half of the UltraFeedback pairs; the 2048\-token limit keeps essentially all of them, which is why we report it as primary\.

DataSFT corr\.Conflict ratePairLimitcorr\.5125122048204851251220482048Joint pairsRetainedHS correctness5125120\.8530\.8530\.9820\.9820\.9810\.9810\.0220\.0220\.0270\.0277,0337\{,\}0330\.3150\.3152048204821,18421\{,\}1840\.9960\.996HS coherence5125120\.6350\.6350\.9120\.9120\.9110\.9110\.0600\.0600\.0670\.0675,5085\{,\}5080\.3170\.3172048204816,34016\{,\}3400\.9960\.996HS complexity5125120\.2360\.2360\.1780\.1780\.0950\.0950\.1480\.1480\.2020\.2024,1684\{,\}1680\.3430\.3432048204811,25611\{,\}2560\.9960\.996HS verbosity \(\+\+\)5125120\.2560\.256−0\.068\-0\.068−0\.162\-0\.1620\.1590\.1590\.2260\.2265,6545\{,\}6540\.3350\.3352048204815,66315\{,\}6630\.9960\.996HS verbosity \(−\-\)512512−0\.256\-0\.2560\.0680\.0680\.1620\.1620\.8410\.8410\.7740\.7745,6545\{,\}6540\.3350\.3352048204815,66315\{,\}6630\.9960\.996UF honesty5125120\.6350\.6350\.8700\.8700\.8540\.8540\.1860\.1860\.1810\.18193,21893\{,\}2180\.5030\.50320482048182,000182\{,\}0000\.9950\.995UF instr\.\-following5125120\.6980\.6980\.8880\.8880\.8790\.8790\.1720\.1720\.1640\.164112,391112\{,\}3910\.5130\.51320482048216,157216\{,\}1570\.9950\.995UF truthfulness5125120\.6150\.6150\.8320\.8320\.8100\.8100\.1930\.1930\.1920\.19287,40287\{,\}4020\.5230\.52320482048164,654164\{,\}6540\.9940\.994Table 4:Pre\-training measurements and the data behind them, for both training\-length limits\.*Data corr\.*is the Pearson correlation of the dataset’s own scores;*SFT corr\.*the correlation of ArmoRM scores on reference\-model generations \(§[4\.1](https://arxiv.org/html/2609.26929#S4.SS1)\)\.*Joint pairs*counts response pairs that yield a preference pair for both objectives, and*Retained*the share of valid response pairs the limit keeps\. The 2048\-token columns are the ones reported in[Table1](https://arxiv.org/html/2609.26929#S5.T1)\.
## Appendix CPer\-Model Trade\-off Scores

[Table5](https://arxiv.org/html/2609.26929#A3.T5)lists every score behind[Figure2](https://arxiv.org/html/2609.26929#S6.F2),[Figure3](https://arxiv.org/html/2609.26929#S6.F3), and theBoth↑\\uparrow?column of[Table1](https://arxiv.org/html/2609.26929#S5.T1): all eight settings, both training\-length limits, and both decoding conditions\.[Table6](https://arxiv.org/html/2609.26929#A3.T6)gives the matching length and repetition diagnostics\.

PairLimitSFTw=\.1w\{=\}\.1w=\.2w\{=\}\.2w=\.4w\{=\}\.4w=\.6w\{=\}\.6w=\.8w\{=\}\.8w=\.9w\{=\}\.9HS correctness512512\.628\.628/\.641\.641\.677\.677/\.681\.681\.680\.680/\.683\.683\.674\.674/\.677\.677\.621\.621/\.626\.626\.623\.623/\.627\.627\.614\.614/\.619\.61920482048\.621\.621/\.636\.636\.702\.702/\.706\.706\.699\.699/\.703\.703\.683\.683/\.688\.688\.675\.675/\.680\.680\.668\.668/\.674\.674–HS coherence512512\.628\.628/\.737\.737\.684\.684/\.756\.756\.673\.673/\.749\.749\.651\.651/\.729\.729\.633\.633/\.713\.713\.628\.628/\.710\.710\.619\.619/\.698\.69820482048\.621\.621/\.730\.730\.703\.703/\.770\.770\.692\.692/\.761\.761\.697\.697/\.763\.763\.676\.676/\.749\.749\.676\.676/\.748\.748–HS complexity512512\.628\.628/\.386\.386\.575\.575/\.439\.439\.606\.606/\.446\.446\.588\.588/\.443\.443\.609\.609/\.446\.446\.599\.599/\.436\.436\.604\.604/\.438\.43820482048\.621\.621/\.382\.382\.626\.626/\.455\.455\.626\.626/\.451\.451\.646\.646/\.456\.456\.635\.635/\.452\.452\.653\.653/\.440\.440–HS verbosity \(\+\+\)512512\.628\.628/\.427\.427\.631\.631/\.632\.632\.626\.626/\.630\.630\.605\.605/\.645\.645\.575\.575/\.653\.653\.603\.603/\.635\.635\.616\.616/\.636\.63620482048\.621\.621/\.420\.420\.618\.618/\.625\.625\.606\.606/\.634\.634\.608\.608/\.641\.641\.602\.602/\.637\.637\.619\.619/\.622\.622\.661\.661/\.602\.602HS verbosity \(−\-\)512512\.628\.628/\.427\.427\.599\.599/\.334\.334\.620\.620/\.359\.359\.644\.644/\.384\.384\.660\.660/\.442\.442\.649\.649/\.597\.597\.624\.624/\.620\.62020482048\.621\.621/\.420\.420\.573\.573/\.282\.282\.582\.582/\.289\.289\.628\.628/\.343\.343\.684\.684/\.428\.428\.696\.696/\.520\.520\.690\.690/\.547\.547UF honesty512512\.432\.432/\.518\.518\.438\.438/\.456\.456\.452\.452/\.471\.471\.450\.450/\.467\.467\.440\.440/\.453\.453\.441\.441/\.456\.456\.446\.446/\.455\.45520482048\.430\.430/\.532\.532\.391\.391/\.394\.394\.388\.388/\.392\.392\.395\.395/\.398\.398\.410\.410/\.412\.412\.428\.428/\.424\.424–UF instr\.\-following512512\.432\.432/\.509\.509\.464\.464/\.504\.504\.470\.470/\.507\.507\.469\.469/\.497\.497\.459\.459/\.485\.485\.441\.441/\.467\.467\.436\.436/\.459\.45920482048\.430\.430/\.515\.515\.375\.375/\.395\.395\.391\.391/\.409\.409\.418\.418/\.437\.437\.410\.410/\.432\.432–\.413\.413/\.428\.428UF truthfulness512512\.432\.432/\.581\.581\.461\.461/\.511\.511\.459\.459/\.512\.512\.462\.462/\.514\.514\.452\.452/\.498\.498\.459\.459/\.503\.503\.441\.441/\.484\.48420482048\.430\.430/\.602\.602\.463\.463/\.507\.507\.451\.451/\.480\.480\.422\.422/\.444\.444\.421\.421/\.438\.438\.409\.409/\.419\.419–*2048\-token models re\-decoded with repetition control*HS correctness20482048\.600\.600/\.607\.607\.678\.678/\.673\.673\.677\.677/\.671\.671\.681\.681/\.675\.675\.678\.678/\.671\.671\.665\.665/\.661\.661–HS coherence20482048\.600\.600/\.722\.722\.667\.667/\.754\.754\.669\.669/\.754\.754\.668\.668/\.752\.752\.665\.665/\.750\.750\.668\.668/\.751\.751–HS complexity20482048\.600\.600/\.378\.378\.657\.657/\.463\.463\.659\.659/\.467\.467\.660\.660/\.461\.461\.664\.664/\.462\.462\.663\.663/\.444\.444–HS verbosity \(\+\+\)20482048\.600\.600/\.386\.386\.652\.652/\.602\.602\.648\.648/\.605\.605\.653\.653/\.608\.608\.648\.648/\.593\.593\.656\.656/\.580\.580\.670\.670/\.558\.558HS verbosity \(−\-\)20482048\.600\.600/\.386\.386\.537\.537/\.274\.274\.547\.547/\.284\.284\.592\.592/\.330\.330\.645\.645/\.426\.426\.668\.668/\.505\.505\.674\.674/\.522\.522UF honesty20482048\.442\.442/\.535\.535\.544\.544/\.553\.553\.545\.545/\.553\.553\.548\.548/\.548\.548\.550\.550/\.551\.551\.554\.554/\.552\.552–UF instr\.\-following20482048\.442\.442/\.504\.504\.546\.546/\.536\.536\.548\.548/\.544\.544\.552\.552/\.546\.546\.556\.556/\.549\.549–\.552\.552/\.545\.545UF truthfulness20482048\.442\.442/\.595\.595\.543\.543/\.570\.570\.548\.548/\.574\.574\.551\.551/\.561\.561\.553\.553/\.554\.554\.550\.550/\.543\.543–Table 5:Mean ArmoRM scores over the same400400prompts, as helpfulness/second\-objective\. Leading zeros are dropped;*–*marks a weight that was not trained\. For HS verbosity \(−\-\) a lower second score is better\. The lower block re\-decodes the 2048\-token models with repetition penalty1\.101\.10and44\-gram blocking; its aggregate is[Table2](https://arxiv.org/html/2609.26929#S6.T2)\.PairLimitSFTw=\.1w\{=\}\.1w=\.2w\{=\}\.2w=\.4w\{=\}\.4w=\.6w\{=\}\.6w=\.8w\{=\}\.8w=\.9w\{=\}\.9HS correctness512512174174/\.438\.438321321/\.720\.720338338/\.795\.795374374/\.825\.825588588/\.868\.868550550/\.863\.863597597/\.865\.86520482048202202/\.398\.398257257/\.767\.767263263/\.755\.755315315/\.790\.790348348/\.800\.800317317/\.713\.713–HS coherence512512174174/\.438\.438313313/\.838\.838376376/\.855\.855438438/\.843\.843513513/\.865\.865520520/\.855\.855560560/\.868\.86820482048202202/\.398\.398234234/\.728\.728269269/\.765\.765272272/\.728\.728292292/\.740\.740299299/\.738\.738–HS complexity512512174174/\.438\.438788788/\.927\.927628628/\.890\.890736736/\.897\.897622622/\.910\.910647647/\.915\.915626626/\.880\.88020482048202202/\.398\.398544544/\.863\.863551551/\.870\.870477477/\.845\.845505505/\.845\.845371371/\.713\.713–HS verbosity \(\+\+\)512512174174/\.438\.438478478/\.895\.895492492/\.875\.875589589/\.902\.902716716/\.915\.915634634/\.885\.885592592/\.873\.87320482048202202/\.398\.398528528/\.845\.845578578/\.868\.868610610/\.882\.882645645/\.858\.858597597/\.840\.840429429/\.805\.805HS verbosity \(−\-\)512512174174/\.438\.4388282/\.233\.2339090/\.263\.263102102/\.345\.345182182/\.450\.450416416/\.790\.790536536/\.875\.87520482048202202/\.398\.3984141/\.138\.1384646/\.145\.1458484/\.278\.278125125/\.445\.445217217/\.640\.640275275/\.703\.703UF honesty512512329329/\.527\.527727727/\.892\.892687687/\.895\.895677677/\.897\.897695695/\.917\.917683683/\.920\.920670670/\.920\.92020482048234234/\.480\.480897897/\.965\.965889889/\.958\.958862862/\.960\.960808808/\.960\.960786786/\.950\.950–UF instr\.\-following512512329329/\.527\.527590590/\.823\.823584584/\.835\.835607607/\.858\.858642642/\.900\.900687687/\.917\.917706706/\.927\.92720482048234234/\.480\.480972972/\.973\.973922922/\.953\.953824824/\.960\.960828828/\.960\.960–830830/\.970\.970UF truthfulness512512329329/\.527\.527640640/\.853\.853629629/\.855\.855623623/\.877\.877656656/\.885\.885618618/\.905\.905686686/\.907\.90720482048234234/\.480\.480599599/\.885\.885683683/\.915\.915804804/\.938\.938787787/\.958\.958820820/\.968\.968–*2048\-token models re\-decoded with repetition control*HS correctness204820488888/\.013\.013164164/\.020\.020161161/\.010\.010161161/\.015\.015164164/\.008\.008149149/\.010\.010–HS coherence204820488888/\.013\.013152152/\.023\.023149149/\.015\.015154154/\.020\.020144144/\.010\.010152152/\.013\.013–HS complexity204820488888/\.013\.013210210/\.015\.015204204/\.018\.018196196/\.033\.033197197/\.030\.030156156/\.015\.015–HS verbosity \(\+\+\)204820488888/\.013\.013210210/\.030\.030223223/\.030\.030224224/\.048\.048220220/\.040\.040198198/\.025\.025172172/\.028\.028HS verbosity \(−\-\)204820488888/\.013\.0133333/\.003\.0033535/\.000\.0005252/\.003\.0039999/\.003\.003141141/\.008\.008154154/\.013\.013UF honesty20482048114114/\.080\.080329329/\.305\.305338338/\.313\.313356356/\.348\.348345345/\.320\.320327327/\.298\.298–UF instr\.\-following20482048114114/\.080\.080397397/\.383\.383383383/\.350\.350354354/\.385\.385346346/\.355\.355–329329/\.338\.338UF truthfulness20482048114114/\.080\.080270270/\.233\.233283283/\.265\.265305305/\.315\.315331331/\.310\.310347347/\.353\.353–Table 6:Generation diagnostics for the same models, as mean words/share of responses with a repeated44\-gram\. Greedy decoding above, repetition control below\. These are the numbers behind the length and repetition confound in §[6\.1](https://arxiv.org/html/2609.26929#S6.SS1)\.
## Appendix DRepetition Control, Per Pair

[Table7](https://arxiv.org/html/2609.26929#A4.T7)breaks[Table2](https://arxiv.org/html/2609.26929#S6.T2)out by objective pair\. The split is clean: no HelpSteer pair moves by more than0\.0360\.036on either objective, no UltraFeedback pair by less than0\.1030\.103, while the length and repetition drops are comparable across the two datasets\.

Δ\\DeltahelpΔ\\Deltaobj2Δ\\DeltawordsΔ\\DeltarepHS correctness−0\.010\-0\.010−0\.020\-0\.020−46\.0%\-46\.0\\%−75\.3\-75\.3HS coherence−0\.021\-0\.021−0\.006\-0\.006−44\.7%\-44\.7\\%−72\.4\-72\.4HS complexity\+0\.023\+0\.023\+0\.009\+0\.009−60\.5%\-60\.5\\%−80\.5\-80\.5HS verbosity \(\+\+\)\+0\.036\+0\.036−0\.036\-0\.036−62\.9%\-62\.9\\%−81\.6\-81\.6HS verbosity \(−\-\)−0\.032\-0\.032\+0\.012\+0\.012−30\.0%\-30\.0\\%−38\.7\-38\.7UF honesty\+0\.145\+0\.145\+0\.147\+0\.147−59\.9%\-59\.9\\%−64\.2\-64\.2UF instr\.\-following\+0\.149\+0\.149\+0\.124\+0\.124−58\.7%\-58\.7\\%−60\.1\-60\.1UF truthfulness\+0\.116\+0\.116\+0\.103\+0\.103−58\.2%\-58\.2\\%−63\.8\-63\.8HelpSteer \(mean\)−0\.001\-0\.001−0\.008\-0\.008−48\.8%\-48\.8\\%−69\.7\-69\.7UltraFeedback \(mean\)\+0\.137\+0\.137\+0\.125\+0\.125−58\.9%\-58\.9\\%−62\.7\-62\.7Table 7:Per\-pair version of[Table2](https://arxiv.org/html/2609.26929#S6.T2): change in scores, mean words, and repeated\-44\-gram share \(percentage points\) when the 2048\-token models are re\-decoded with repetition control\. Positive score changes are in the preferred direction\.
## Appendix EConflict\-Split Results

[Table8](https://arxiv.org/html/2609.26929#A5.T8)gives the scores for the conflict\-split experiment of §[4\.2](https://arxiv.org/html/2609.26929#S4.SS2)\. The joint and non\-conflict sets stay close to the main results, as expected, since non\-conflict pairs make up most of the joint set; the conflict set is where the scores move\.

SettingSetSFTw=0\.1w\{=\}0\.1w=0\.2w\{=\}0\.2w=0\.4w\{=\}0\.4w=0\.8w\{=\}0\.8UF instr\.\-following,512512joint\.432\.432/\.509\.509\.445\.445/\.482\.482\.430\.430/\.463\.463\.456\.456/\.486\.486\.457\.457/\.483\.483non\-conflict\.432\.432/\.509\.509\.451\.451/\.475\.475\.458\.458/\.486\.486\.454\.454/\.480\.480\.446\.446/\.470\.470conflict\.432\.432/\.509\.509\.412\.412/\.511\.511\.430\.430/\.522\.522\.427\.427/\.499\.499\.424\.424/\.453\.453HS verbosity \(−\-\),512512joint\.628\.628/\.427\.427\.604\.604/\.348\.348\.604\.604/\.364\.364\.631\.631/\.398\.398\.647\.647/\.588\.588non\-conflict\.628\.628/\.427\.427\.595\.595/\.371\.371\.600\.600/\.393\.393\.621\.621/\.422\.422\.669\.669/\.562\.562conflict\.628\.628/\.427\.427\.585\.585/\.653\.653\.617\.617/\.647\.647\.594\.594/\.650\.650\.614\.614/\.626\.626HS verbosity \(−\-\),20482048joint\.621\.621/\.420\.420\.609\.609/\.317\.317\.608\.608/\.317\.317\.626\.626/\.345\.345\.693\.693/\.518\.518non\-conflict\.621\.621/\.420\.420\.581\.581/\.305\.305\.555\.555/\.277\.277\.613\.613/\.343\.343\.670\.670/\.462\.462conflict\.621\.621/\.420\.420\.691\.691/\.419\.419\.694\.694/\.426\.426\.695\.695/\.441\.441\.701\.701/\.518\.518Table 8:Conflict\-split results \(§[4\.2](https://arxiv.org/html/2609.26929#S4.SS2)\): mean helpfulness/second\-objective scores for margin reward models trained on the joint set, the non\-conflict set \(conflict rate00\), and the conflict set \(rate11\)\. The split is computed in the dataset’s score direction, so under the negative verbosity sign the “conflict” set is the one whose pairs agree with helpfulness\.SettingSetSFTw=0\.1w\{=\}0\.1w=0\.2w\{=\}0\.2w=0\.4w\{=\}0\.4w=0\.8w\{=\}0\.8UF instr\.\-following,512512joint329329/\.527\.527714714/\.868\.868738738/\.880\.880673673/\.887\.887643643/\.905\.905non\-conflict329329/\.527\.527676676/\.917\.917685685/\.920\.920667667/\.943\.943685685/\.915\.915conflict329329/\.527\.527212212/\.430\.430270270/\.475\.475471471/\.650\.650732732/\.920\.920HS verbosity \(−\-\),512512joint174174/\.438\.438100100/\.250\.250119119/\.307\.307138138/\.338\.338435435/\.760\.760non\-conflict174174/\.438\.438155155/\.292\.292186186/\.340\.340184184/\.388\.388280280/\.705\.705conflict174174/\.438\.438756756/\.925\.925606606/\.945\.945707707/\.900\.900595595/\.887\.887HS verbosity \(−\-\),20482048joint202202/\.398\.3985858/\.217\.2176666/\.212\.2129595/\.263\.263240240/\.598\.598non\-conflict202202/\.398\.3986464/\.217\.2174646/\.155\.1559898/\.292\.292192192/\.490\.490conflict202202/\.398\.398109109/\.390\.390116116/\.417\.417135135/\.445\.445195195/\.605\.605Table 9:Length and repetition diagnostics for the conflict\-split models of[Table8](https://arxiv.org/html/2609.26929#A5.T8), as mean words/share of responses with a repeated44\-gram\.
## Appendix FCoverage Study

[Table10](https://arxiv.org/html/2609.26929#A6.T10)gives the per\-target gaps that[Table3](https://arxiv.org/html/2609.26929#S6.T3)averages, and[Table11](https://arxiv.org/html/2609.26929#A6.T11)the reward and cost scores they are computed from\.

Reward gapCost gapw⋆w^\{\\star\}FineCoarseMergeFineCoarseMerge0\.10\.12\.0182\.0182\.0182\.0180\.6730\.6734\.9004\.9004\.9004\.9002\.8222\.8220\.30\.30\.2530\.2532\.2192\.2190\.1660\.1663\.4443\.4442\.4002\.4001\.1671\.1670\.50\.51\.0381\.038–0\.0510\.0512\.1192\.119–0\.1460\.1460\.70\.70\.9280\.9280\.0840\.0841\.5461\.5460\.7280\.7282\.6452\.6450\.2730\.2730\.90\.91\.0501\.0503\.7323\.7322\.0592\.0590\.2180\.2181\.2261\.2260\.4480\.448mean∗1\.0621\.0622\.0132\.0131\.1111\.1112\.3232\.3232\.7932\.7931\.1781\.178Table 10:Per\-target absolute difference from a model trained directly atw⋆w^\{\\star\}, the values plotted in[Figure5](https://arxiv.org/html/2609.26929#S6.F5)and averaged in[Table3](https://arxiv.org/html/2609.26929#S6.T3)\.*–*marks the targets where coarse selection returns the direct model itself\.∗mean excludingw⋆=0\.5w^\{\\star\}\{=\}0\.5\.DirectLowerMergeUpperw⋆w^\{\\star\}rew\.costrew\.costrew\.costrew\.cost0\.10\.1−0\.22\-0\.222\.532\.53−2\.24\-2\.24−2\.37\-2\.37−0\.90\-0\.90−0\.29\-0\.290\.990\.992\.102\.100\.30\.31\.241\.245\.545\.540\.990\.992\.102\.101\.411\.414\.374\.372\.422\.425\.825\.820\.50\.53\.463\.467\.947\.942\.422\.425\.825\.823\.513\.518\.098\.094\.474\.479\.869\.860\.70\.73\.543\.5410\.5810\.584\.474\.479\.869\.865\.095\.0910\.8610\.865\.805\.8011\.7611\.760\.90\.94\.754\.7511\.9711\.975\.805\.8011\.7611\.766\.816\.8112\.4212\.428\.488\.4813\.2013\.20Table 11:Mean Beaver reward and cost over the700700coverage prompts for the directly trained model, the two neighbouring trained models, and their merge\. Every merged model lands between its neighbours, which is why merging cannot reach a direct target outside that interval \(reward atw⋆=0\.7,0\.9w^\{\\star\}\{=\}0\.7,0\.9; cost atw⋆=0\.1w^\{\\star\}\{=\}0\.1\)\.