利用多维用户偏好数据实现大语言模型对齐的高样本效率个性化

arXiv cs.LG 论文

摘要

该论文提出多维偏好引导的在线直接偏好优化方法(MPG-DPO),利用低成本的多维自动评估构建个性化偏好锥,以提升大语言模型在线偏好对齐的样本效率。理论分析与多模型实验表明其在相同查询预算下优于现有方法。

arXiv:2611.22210v1 Announce Type: new Abstract: Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.
查看原文
查看缓存全文

缓存时间: 2026/11/19 03:37

# Sample-efficient Personalization of Large Language Model Alignmentwith Multidimensional User Preference Data
Source: [https://arxiv.org/html/2611.22210](https://arxiv.org/html/2611.22210)
Zhen\-Yu Zhang zhen\-yu\.zhang@riken\.jpJerry Huang jerry\.huang@mila\.quebecAffiliation:Center for Advanced Intelligence Project, RIKEN, JapanAffiliation:Université de Montréal, CanadaAffiliation:Mila \- Quebec AI Institute, CanadaGang Niu gang\.niu@riken\.jpAffiliation:Center for Advanced Intelligence Project, RIKENTakashi Ishida ishi@k\.u\-tokyo\.ac\.jpAffiliation:Center for Advanced Intelligence Project, RIKEN, JapanAffiliation:Graduate School of Frontier Sciences, The University of Tokyo, JapanMasashi Sugiyama sugi@k\.u\-tokyo\.ac\.jpAffiliation:Center for Advanced Intelligence Project, RIKEN, JapanAffiliation:Graduate School of Frontier Sciences, The University of Tokyo, Japan

###### Abstract

Preference alignment has become a central tool to align*large language models*\(LLMs\) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback\. But*costly*,*time\-consuming*annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison\. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering\. In this paper, we propose*multidimensional preference\-guided online direct preference optimization*\(MPG\-DPO\), which leverages low\-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment\. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement\. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone\-dominated responses vanishes as learning progresses\. Experiments across multiple LLM families show that MPG\-DPO outperforms strong contenders under matched query budgets\.

††heading:1 2024 1\-4/00 10/00 Anonymous Authors††shortheadings:multidimensional preference\-guided online direct preference optimization / Zhang, Huang, Niu, Ishida, Sugiyama††firstpage:1## 1Introduction

Preference alignment has achieved remarkable success in aligning*large language models*\(LLMs\) with human preferences and desired behaviors\([Christiano et al\., 2017](https://arxiv.org/html/2611.22210#bib.bib1);[Ouyang et al\., 2022](https://arxiv.org/html/2611.22210#bib.bib3)\)\. To continually push the boundaries of model capabilities, preference alignment is typically extended to an*online*process, where models repeatedly generate new responses for human evaluation\([Guo et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib16);[Calandriello et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib13);[Xie et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib4)\)\. However, the high cost and long turnaround of the annotations required pose a major barrier to scaling this continuous feedback loop\. Therefore, improving sample efficiency in online preference alignment is of critical importance\.

![Refer to caption](https://arxiv.org/html/2611.22210v1/preference_cone.png)Figure 1:Summary of the proposedMPG\-DPOapproach\.Given an input prompt, candidate responses are sampled and filtered using the preference cone, after which a pair is selected to maximize the weaker of two opposing preference margins\. The selected responses are evaluated and used to update the cone and policy\.Existing approaches can be broadly divided into two categories\. The first focuses on online exploration to improve policy coverage, increasing the sampling probability of responses that are poorly covered by previous data\([Xie et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib4);[Cen et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib5);[Chen et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib9)\)\. The second line of work relies on active preference alignment, which first generates a pool of candidate pairs and then selects the most informative pair for comparison in each interaction\([Das et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib11);[Lin et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib12);[Belakaria et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib20)\)\. These active methods typically rank candidates using information criteria measured from the model perspective, such as reward uncertainty or gradient\-based metrics\.

While effective at optimizing response generation or pair selection, these methods mainly rely on*model\-perspective*information, often querying feedback even when past comparisons made by the user already settle the ordering within a pair\. For example, in code generation, previous choices may show that the user is willing to trade some clarity for a sufficiently large gain in accuracy\. A new pair may still exhibit high reward uncertainty even when one response offers too little accuracy gain to justify its loss of clarity, making the comparison carry little*user\-perspective*information despite appearing uncertain to the model\. In a related direction, constitutional AI reduces repeated human supervision via high\-level guiding principles\([Bai et al\., 2022](https://arxiv.org/html/2611.22210#bib.bib17);[Huang et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib19)\), but these principles are*hand\-written*and shared across users\. To capture personalized trade\-offs, specific dimensions like accuracy and clarity can fortunately be evaluated automatically, allowing them to naturally serve as low\-cost multidimensional scores\. This raises a question:*How can low\-cost multidimensional scores be leveraged to capture personalized preferences and improve sample efficiency?*

Using such multidimensional information is non\-trivial because the scores are*heterogeneous*and can hardly be directly aggregated\. The insightful AMPLe\([Oh et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib8)\)uses multi\-objective rewards through Bayesian scalarization, but its uncertainty\-driven queries may compare responses already dominated by other alternatives\. We instead treat each historical comparison as revealing a*preferred direction*of the user in the sub\-objective space, and these directions form a personalized preference*Pareto cone*\([Censor, 1977](https://arxiv.org/html/2611.22210#bib.bib22);[Rame et al\., 2023](https://arxiv.org/html/2611.22210#bib.bib2);[Ma et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib23)\)\. The key challenge is to estimate this cone from historical data and use it to identify response pairs whose preference remains unresolved\.

In this paper, we use low\-cost multidimensional evaluations to improve sample efficiency in online preference alignment\. Each response is scored on attributes such as correctness and generation cost by low\-cost or task\-specific verifiers that are substantially cheaper than human evaluators\. We construct a personalized preference cone from historical user comparisons and leverage its geometry to distinguish comparisons that are already resolved from those that represent genuinely unresolved trade\-offs\. We then select two competitive responses that*exhibit the strongest unresolved disagreement among the retained preference directions*\. This preserves the exploration of potentially valuable responses while directing costly feedback toward comparisons that provide new information\.

We propose*multidimensional preference\-guided online direct preference optimization*\(MPG\-DPO\), which uses the user preference cone to guide response selection at each round of online DPO\. Specifically, at each round, the current policy generates a pool of on\-policy responses, and a ridge\-regression estimator predicts their multidimensional score vectors together with confidence bounds\. We eliminate responses that are confidently dominated by another response in the sampled pool and select a pair for which different preference directions favor different responses\. After obtaining the preference label, we update the preference cone and the policy\. Theoretical analysis shows that, with high probability, only a slowly growing number of selections are cone\-dominated, so wasted queries vanish as learning progresses\. Experiments show the effectiveness of the proposed method\.

## 2Preliminaries

In this section, we introduce the preliminaries and notations of our proposed algorithm\.

### 2\.1Online Preference Alignment

In this part, we introduce the notations for online preference alignment\. LetΠ\\Pibe a class of LLM policies, where eachπ∈Π\\pi\\in\\Pimaps a prompt𝐱∈𝒳\\mathbf\{x\}\\in\\mathcal\{X\}to a distributionπ\(⋅∣𝐱\)\\pi\(\\cdot\\mid\\mathbf\{x\}\)over the response space𝒴\\mathcal\{Y\}\. At each roundtt, a prompt𝐱t∼ρ\\mathbf\{x\}\_\{t\}\\sim\\rhois drawn from a distributionρ\\rhoover𝒳\\mathcal\{X\}\. Given𝐱t\\mathbf\{x\}\_\{t\}, we sample two responsesyt∼πt\(1\)\(⋅∣𝐱t\)y\_\{t\}\\sim\\pi\_\{t\}^\{\(1\)\}\(\\cdot\\mid\\mathbf\{x\}\_\{t\}\)andyt′∼πt\(2\)\(⋅∣𝐱t\)y\_\{t\}^\{\\prime\}\\sim\\pi\_\{t\}^\{\(2\)\}\(\\cdot\\mid\\mathbf\{x\}\_\{t\}\), whereπt\(1\),πt\(2\)∈Π\\pi\_\{t\}^\{\(1\)\},\\pi\_\{t\}^\{\(2\)\}\\in\\Piare the sampling policies used at roundtt\. A preference oracle then returns a labelot∈\{−1,\+1\}o\_\{t\}\\in\\\{\-1,\+1\\\}, whereot=\+1o\_\{t\}=\+1indicates thatyty\_\{t\}is preferred toyt′y\_\{t\}^\{\\prime\}, denoted byyt≻yt′y\_\{t\}\\succ y\_\{t\}^\{\\prime\}, andot=−1o\_\{t\}=\-1indicates the reverse preference\.

For any policyπ∈Π\\pi\\in\\Pi, we define its KL\-regularized value as

Jβ\(π\)=𝔼𝐱∼ρ,y∼π\(⋅∣𝐱\)\[r∗\(𝐱,y\)−βlogπ⁡\(y∣𝐱\)πref​\(y∣𝐱\)\],J\_\{\\beta\}\(\\pi\)=\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\rho,\\,y\\sim\\pi\(\\cdot\\mid\\mathbf\{x\}\)\}\\left\[r^\{\*\}\(\\mathbf\{x\},y\)\-\\beta\\log\\frac\{\\pi\(y\\mid\\mathbf\{x\}\)\}\{\\pi\_\{\\rm ref\}\(y\\mid\\mathbf\{x\}\)\}\\right\],wherer∗r^\{\*\}is the reward function,πref\\pi\_\{\\rm ref\}is a fixed reference policy andβ\>0\\beta\>0is the KL\-regularization coefficient\. Letπβ∗∈arg​maxπ∈Π⁡Jβ​\(π\)\\pi\_\{\\beta\}^\{\*\}\\in\\argmax\_\{\\pi\\in\\Pi\}J\_\{\\beta\}\(\\pi\)denote an optimal KL\-regularized policy\.

The objectiveJβJ\_\{\\beta\}depends on the unknown reward function, whereas the available feedback consists of preference labels\.*Direct preference optimization*\(DPO\) instead learns directly from preference pairs without fitting a separate reward model\([Rafailov et al\., 2023](https://arxiv.org/html/2611.22210#bib.bib7)\)\. We consider the online DPO setting\([Calandriello et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib13);[Guo et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib16)\), in which preference pairs are collected during the iterative training process\. Given the comparison\(𝐱t,yt,yt′\)\(\\mathbf\{x\}\_\{t\},y\_\{t\},y\_\{t\}^\{\\prime\}\)and its observed preference labeloto\_\{t\}at roundtt, we perform the stochastic\-gradient update

πt\+1=πt−ηt​∇π\[−log⁡σ⁡\(β​ot​\[log⁡π⁡\(yt∣𝐱t\)πref​\(yt∣𝐱t\)−log⁡π⁡\(yt′∣𝐱t\)πref​\(yt′∣𝐱t\)\]\)\]\|π=πt,\\pi\_\{t\+1\}=\\pi\_\{t\}\-\\eta\_\{t\}\\left\.\\nabla\_\{\\pi\}\\left\[\-\\log\\sigma\\left\(\\beta\\ o\_\{t\}\\left\[\\log\\frac\{\\pi\(y\_\{t\}\\mid\\mathbf\{x\}\_\{t\}\)\}\{\\pi\_\{\\rm ref\}\(y\_\{t\}\\mid\\mathbf\{x\}\_\{t\}\)\}\-\\log\\frac\{\\pi\(y\_\{t\}^\{\\prime\}\\mid\\mathbf\{x\}\_\{t\}\)\}\{\\pi\_\{\\rm ref\}\(y\_\{t\}^\{\\prime\}\\mid\\mathbf\{x\}\_\{t\}\)\}\\right\]\\right\)\\right\]\\right\|\_\{\\pi=\\pi\_\{t\}\},\(1\)whereηt\>0\\eta\_\{t\}\>0is the learning rate\.

### 2\.2Preference Cone

In this part, we introduce the notations of user preference cone\.

###### Definition 1\(Preference Matrix and Preference Cone\)\.

LetKKdenote the number of preference directions andddthe number of sub\-objectives\. Given a preference matrix𝐖∈ℝK×d\\mathbf\{W\}\\in\\mathbb\{R\}^\{K\\times d\}, the associated preference cone is defined as

C=\{𝐱∈ℝd∣𝐖𝐱≥𝟎\}\.C=\\\{\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}\\mid\\mathbf\{W\}\\mathbf\{x\}\\geq\\bm\{0\}\\\}\.

###### Definition 2\(ParetoCC\-Optimality\)\.

For any𝐱,𝐲∈ℝd\\mathbf\{x\},\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\}, we say that𝐱\\mathbf\{x\}strictlyCC\-dominates𝐲\\mathbf\{y\}, denoted by𝐱≻C𝐲\\mathbf\{x\}\\succ\_\{C\}\\mathbf\{y\}, if𝐱−𝐲∈C\\mathbf\{x\}\-\\mathbf\{y\}\\in Cand𝐖⁡\(𝐱−𝐲\)≠𝟎\\mathbf\{W\}\(\\mathbf\{x\}\-\\mathbf\{y\}\)\\neq\\bm\{0\}\. An action is said to be*ParetoCC\-optimal*if no other action strictlyCC\-dominates it\. We define the set of ParetoCC\-optimal actions as

PC∗=\{i∈\{1,…,N\}∣∄j∈\{1,…,N\}∖\{i\}:j≻Ci\}\.P^\{\*\}\_\{C\}=\\\{i\\in\\\{1,\\ldots,N\\\}\\mid\\nexists\\,j\\in\\\{1,\\ldots,N\\\}\\setminus\\\{i\\\}:\\ j\\succ\_\{C\}i\\\}\.Here,NNis the number of actions\. For notational convenience, for any two actionsi,j∈\{1,…,N\}i,j\\in\\\{1,\\ldots,N\\\}, we writei≻Cji\\succ\_\{C\}jas shorthand for𝐱i≻C𝐱j\\mathbf\{x\}\_\{i\}\\succ\_\{C\}\\mathbf\{x\}\_\{j\}\.

We assume that, at every round, each non\-optimal response has an optimal alternative with a uniformly positive advantage across all preference directions available before the query, while allowing ties in other comparisons\. This condition excludes non\-optimal responses with only boundary dominators\.

###### Assumption 1\(Uniform Positive Preference Gap\)\.

Let𝐰t−1,k⊤\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}denote thekk\-th row of𝐖t−1\\mathbf\{W\}\_\{t\-1\}available before querytt, and let𝐱t,i\\mathbf\{x\}\_\{t,i\}denote the latent score vector of actioniiin the current candidate pool\. All optimal sets in this assumption are evaluated on this pool usingCt−1C\_\{t\-1\}\. For eachi∉PCt−1∗i\\notin P\_\{C\_\{t\-1\}\}^\{\*\}, let𝒟t\(i\)=\{j∈PCt−1∗:j≻Ct−1i\}\\mathcal\{D\}\_\{t\}\(i\)=\\\{j\\in P\_\{C\_\{t\-1\}\}^\{\*\}:j\\succ\_\{C\_\{t\-1\}\}i\\\}be nonempty and define its*preference gap*as

Δt,i=maxj∈𝒟t​\(i\)⁡mink∈\{1,…,Kt−1\}​𝐰t−1,k⊤​\(𝐱t,j−𝐱t,i\)Ut−1,k−Lt−1,k,\\Delta\_\{t,i\}=\\max\_\{j\\in\\mathcal\{D\}\_\{t\}\(i\)\}\\min\_\{k\\in\\\{1,\\ldots,K\_\{t\-1\}\\\}\}\\frac\{\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}\(\\mathbf\{x\}\_\{t,j\}\-\\mathbf\{x\}\_\{t,i\}\)\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\},whereLt−1,k=∑q=1dmin⁡\(0,𝐰t−1,k​\[q\]\)L\_\{t\-1,k\}=\\sum\_\{q=1\}^\{d\}\\min\(0,\\mathbf\{w\}\_\{t\-1,k\}\[q\]\),Ut−1,k=∑q=1dmax⁡\(0,𝐰t−1,k​\[q\]\)U\_\{t\-1,k\}=\\sum\_\{q=1\}^\{d\}\\max\(0,\\mathbf\{w\}\_\{t\-1,k\}\[q\]\), and\[q\]\[q\]indexes coordinates\. We assumeΔt,i≥Δ\>0\\Delta\_\{t,i\}\\geq\\Delta\>0uniformly over all roundsttand actionsi∉PCt−1∗i\\notin P\_\{C\_\{t\-1\}\}^\{\*\}\.

## 3Our Approach

In this section, we present*multidimensional preference\-guided online direct preference optimization*\(MPG\-DPO\)\. We first formulate the learning problem and then present our algorithm, which constructs a user preference cone from historical data to guide response pair selection for sample efficiency\. Finally, we analyze the theoretical property of the proposed algorithm\.

### 3\.1Problem Formulation

In this part, we formulate the learning problem\.

Multidimensional score\.We assume that each prompt–response pair\(𝐱,y\)\(\\mathbf\{x\},y\)has an underlying score vectorμ⁡\(𝐱,y\)=θ∗⁣⊤​ϕ​\(𝐱,y\)∈\[0,1\]d\\mu\(\\mathbf\{x\},y\)=\\theta^\{\*\\top\}\\phi\(\\mathbf\{x\},y\)\\in\[0,1\]^\{d\}\. The observed multidimensional score is

𝐳t​\(𝐱,y\)=θ∗⁣⊤​ϕ​\(𝐱,y\)\+εt​\(𝐱,y\),∥ϕ⁡\(𝐱,y\)∥2≤1,∥θ∗∥F≤S,\\mathbf\{z\}\_\{t\}\(\\mathbf\{x\},y\)=\\theta^\{\*\\top\}\\phi\(\\mathbf\{x\},y\)\+\\varepsilon\_\{t\}\(\\mathbf\{x\},y\),\\qquad\\lVert\\phi\(\\mathbf\{x\},y\)\\rVert\_\{2\}\\leq 1,\\qquad\\lVert\\theta^\{\*\}\\rVert\_\{F\}\\leq S,whereϕ⁡\(𝐱,y\)∈ℝp\\phi\(\\mathbf\{x\},y\)\\in\\mathbb\{R\}^\{p\}is the prompt–response feature vector,θ∗∈ℝp×d\\theta^\{\*\}\\in\\mathbb\{R\}^\{p\\times d\}is unknown and fixed across rounds, andεt​\(𝐱,y\)∈ℝd\\varepsilon\_\{t\}\(\\mathbf\{x\},y\)\\in\\mathbb\{R\}^\{d\}is a conditionally zero\-mean,σz\\sigma\_\{z\}\-sub\-Gaussian noise vector, whereσz≥0\\sigma\_\{z\}\\geq 0\. We omit the prompt argument inϕ\\phi,μ\\mu,𝐳t\\mathbf\{z\}\_\{t\}, andεt\\varepsilon\_\{t\}when it is clear from the context\.

Learning protocol and performance measure\.In the online preference alignment setting, the learner observes multidimensional scores for the queried responses and receives pairwise preference feedback at each round\. At each roundtt, the learner proceeds as follows:

- 1\.SampleNNcandidate responses fromπt\(⋅∣𝐱t\)\\pi\_\{t\}\(\\cdot\\mid\\mathbf\{x\}\_\{t\}\)and select a pair of indicesBt=\{it,jt\}B\_\{t\}=\\\{i\_\{t\},j\_\{t\}\\\}\.
- 2\.Observe the multidimensional score vectors of the selected responses and their preference label\.
- 3\.Update the estimated preference cone and the LLM policy\.

The selection procedure is evaluated by whether a designated response from the selected pair belongs to the optimal set induced by the preference cone\. Specifically, for any indexat∈Bta\_\{t\}\\in B\_\{t\}, we define the cumulative indicator regret as

Regret01⁡\(T\)\\displaystyle\\operatorname\{Regret\}^\{01\}\(T\)=∑t=1T𝕀\{at∉PCt−1∗\}\.\\displaystyle=\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\\{a\_\{t\}\\notin P\_\{C\_\{t\-1\}\}^\{\*\}\\\}\.

### 3\.2Preference Cone Construction

In this part, we construct the preference cone from historical comparisons\. Each comparison reveals a user preference through the multidimensional score difference from the less preferred response to the preferred one\. This difference defines a*preference direction*, and non\-negative linear combinations of these directions form the personalized preference cone\. Since the feedback may be noisy, we screen new directions against existing preferences and retain only those that pass the consistency check\.

Each comparison\(yt,yt′\)\(y\_\{t\},y\_\{t\}^\{\\prime\}\)with labeloto\_\{t\}yields a preference direction by orienting the multidimensional score difference toward the preference label\. Since scores and preference labels may be noisy, we use the previous preference matrix to discard a direction only when every row supports the opposite preference by a margin exceeding a thresholdτ≥0\\tau\\geq 0\. The retained directions generate the cone

Ct=cone\{os𝐝s‖𝐝s‖2:1≤s≤t,min1≤k≤Ks−1𝐰s−1,k⊤\(−os𝐝s‖𝐝s‖2\)≤τ\},\\vskip\-2\.84526ptC\_\{t\}=\\operatorname\{cone\}\\left\\\{o\_\{s\}\\frac\{\\mathbf\{d\}\_\{s\}\}\{\\\|\\mathbf\{d\}\_\{s\}\\\|\_\{2\}\}:1\\leq s\\leq t,\\ \\min\_\{1\\leq k\\leq K\_\{s\-1\}\}\\mathbf\{w\}\_\{s\-1,k\}^\{\\top\}\\left\(\-o\_\{s\}\\frac\{\\mathbf\{d\}\_\{s\}\}\{\\\|\\mathbf\{d\}\_\{s\}\\\|\_\{2\}\}\\right\)\\leq\\tau\\right\\\},\(2\)where𝐝s=𝐳s​\(ys\)−𝐳s​\(ys′\)\\mathbf\{d\}\_\{s\}=\\mathbf\{z\}\_\{s\}\(y\_\{s\}\)\-\\mathbf\{z\}\_\{s\}\(y\_\{s\}^\{\\prime\}\)is the observed multidimensional score difference, andcone\\operatorname\{cone\}denotes the set of all nonnegative linear combinations of the retained preference directions\.

GivenCtC\_\{t\}, we obtain the preference matrix𝐖t\\mathbf\{W\}\_\{t\}using the double\-description method\([Motzkin et al\., 1953](https://arxiv.org/html/2611.22210#bib.bib14);[Fukuda and Prodon, 1995](https://arxiv.org/html/2611.22210#bib.bib15)\)111A well\-established implementation of the method is available through the Python packagepycddlib\.\. Specifically, we compute unit\-norm generators of the vectors having nonnegative inner products with every point inCtC\_\{t\}and use their transposes as the rows of𝐖t\\mathbf\{W\}\_\{t\}:

\{∑k=1Ktλk𝐰t,k:λk≥0\}=\{𝐰∈ℝd:𝐰⊤𝐱≥0,∀𝐱∈Ct\}\.\\vskip\-2\.84526pt\\left\\\{\\sum\_\{k=1\}^\{K\_\{t\}\}\\lambda\_\{k\}\\mathbf\{w\}\_\{t,k\}:\\lambda\_\{k\}\\geq 0\\right\\\}=\\left\\\{\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\}:\\mathbf\{w\}^\{\\top\}\\mathbf\{x\}\\geq 0,\\ \\forall\\mathbf\{x\}\\in C\_\{t\}\\right\\\}\.\(3\)The resulting matrix givesCt=\{𝐱∈ℝd:𝐖t​𝐱≥𝟎\}C\_\{t\}=\\\{\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}:\\mathbf\{W\}\_\{t\}\\mathbf\{x\}\\geq\\bm\{0\}\\\}\. For computational efficiency, our implementation limits the number of retained rules, while the theoretical analysis uses the full cone\.

### 3\.3Response Pair Selection

In this part, we use the current preference cone to guide response pair selection\. The core idea is to*select a pair of responses such that neither is𝒞\\mathcal\{C\}\-dominated by any other candidate, while their comparison reveals a trade\-off unresolved by the current preference rules*\. Each new query thus contributes a candidate preference direction in the multidimensional score space\.

Estimating multidimensional score confidence bounds\.We estimate multidimensional scores by fitting a ridge regression model to previously observed noisy response scores:

Vt=λ​Ip\+∑s<t∑i∈Bsϕ⁡\(ysi\)​ϕ​\(ysi\)⊤,θ^t=Vt−1​∑s<t∑i∈Bsϕ⁡\(ysi\)​𝐳s​\(ysi\)⊤\.V\_\{t\}=\\lambda I\_\{p\}\+\\sum\_\{s<t\}\\sum\_\{i\\in B\_\{s\}\}\\phi\(y\_\{s\}^\{i\}\)\\phi\(y\_\{s\}^\{i\}\)^\{\\top\},\\qquad\\widehat\{\\theta\}\_\{t\}=V\_\{t\}^\{\-1\}\\sum\_\{s<t\}\\sum\_\{i\\in B\_\{s\}\}\\phi\(y\_\{s\}^\{i\}\)\\mathbf\{z\}\_\{s\}\(y\_\{s\}^\{i\}\)^\{\\top\}\.Here,λ\>0\\lambda\>0is the regularization parameter\.

For any two responsesyy,y′y^\{\\prime\}to𝐱t\\mathbf\{x\}\_\{t\}, the predicted normalized improvement along a unit\-norm row𝐰t−1,k⊤\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}of𝐖t−1\\mathbf\{W\}\_\{t\-1\}and the associated confidence width are

m^t,k​\(y∣y′\)=𝐰t−1,k⊤​θ^t⊤​\[ϕ⁡\(y\)−ϕ⁡\(y′\)\]Ut−1,k−Lt−1,k,ct,k​\(y,y′\)=αt​\(δ\)​‖ϕ⁡\(y\)−ϕ⁡\(y′\)‖Vt−1Ut−1,k−Lt−1,k,\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)=\\frac\{\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}\\widehat\{\\theta\}\_\{t\}^\{\\top\}\[\\phi\(y\)\-\\phi\(y^\{\\prime\}\)\]\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\},\\qquad c\_\{t,k\}\(y,y^\{\\prime\}\)=\\frac\{\\alpha\_\{t\}\(\\delta\)\\\|\\phi\(y\)\-\\phi\(y^\{\\prime\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\},whereαt​\(δ\)=λ​S\+σz​d⁡\(2​log⁡dδ\+log⁡detVtdet\(λ​Ip\)\)\\alpha\_\{t\}\(\\delta\)=\\sqrt\{\\lambda\}S\+\\sigma\_\{z\}\\sqrt\{d\\left\(2\\log\\frac\{d\}\{\\delta\}\+\\log\\frac\{\\det V\_\{t\}\}\{\\det\(\\lambda I\_\{p\}\)\}\\right\)\}\.

Letmt,k​\(y∣y′\)m\_\{t,k\}\(y\\mid y^\{\\prime\}\)denote the corresponding latent margin, obtained by replacingθ^t\\widehat\{\\theta\}\_\{t\}withθ∗\\theta^\{\*\}\. With probability at least1−δ1\-\\delta,\|mt,k​\(y∣y′\)−m^t,k​\(y∣y′\)\|≤ct,k​\(y,y′\)\|m\_\{t,k\}\(y\\mid y^\{\\prime\}\)\-\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)\|\\leq c\_\{t,k\}\(y,y^\{\\prime\}\)simultaneously over rounds \(See Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1)in Appendix[B](https://arxiv.org/html/2611.22210#A2)\)\. The resulting confidence bounds are

UCBt,k⁡\(y∣y′\)=m^t,k​\(y∣y′\)\+ct,k​\(y,y′\),LCBt,k⁡\(y∣y′\)=m^t,k​\(y∣y′\)−ct,k​\(y,y′\)\.\\operatorname\{UCB\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)=\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)\+c\_\{t,k\}\(y,y^\{\\prime\}\),\\qquad\\operatorname\{LCB\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)=\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)\-c\_\{t,k\}\(y,y^\{\\prime\}\)\.\(4\)On this event, a positive lower bound certifies an improvement fromy′y^\{\\prime\}toyyalong direction𝐰t−1,k\\mathbf\{w\}\_\{t\-1,k\}\.

RemovingCC\-dominated responses\.In the following, we show how to select the two responses\. Firstly, a response is removed if another response in the pool confidentlyCt−1C\_\{t\-1\}\-dominates it, so that we can prevent the human from comparing a response that is already known to be inferior to another available response\. The retained indices are

𝒩t=\{i∈\[N\]:∄j∈\[N\]∖\{i\}such thatmink∈\[Kt−1\]LCBt,k\(ytj∣yti\)\>0\}\.\\mathcal\{N\}\_\{t\}=\\left\\\{i\\in\[N\]:\\nexists\\,j\\in\[N\]\\setminus\\\{i\\\}\\text\{ such that \}\\min\_\{k\\in\[K\_\{t\-1\}\]\}\\operatorname\{LCB\}\_\{t,k\}\(y\_\{t\}^\{j\}\\mid y\_\{t\}^\{i\}\)\>0\\right\\\}\.\(5\)For a fixed pair\(ytj,yti\)\(y\_\{t\}^\{j\},y\_\{t\}^\{i\}\), Eq\. \([5](https://arxiv.org/html/2611.22210#S3.E5)\) evaluates the least favorable rule forytjy\_\{t\}^\{j\}improving uponytiy\_\{t\}^\{i\}\. A positive value means that all rows of𝐖t−1\\mathbf\{W\}\_\{t\-1\}support this ordering with positive lower confidence bounds, allowingytiy\_\{t\}^\{i\}to be discarded without a human comparison\.

Algorithm 1Multidimensional Preference\-Guided Online Direct Preference Optimization1:Receive

𝐱t\\mathbf\{x\}\_\{t\}and sample

yti∼πt\(⋅∣𝐱t\)y\_\{t\}^\{i\}\\sim\\pi\_\{t\}\(\\cdot\\mid\\mathbf\{x\}\_\{t\}\)for

i∈\[N\]i\\in\[N\]\.

2:Fit ridge regression to past noisy scores and compute confidence bounds using

𝐖t−1\\mathbf\{W\}\_\{t\-1\}via \([4](https://arxiv.org/html/2611.22210#S3.E4)\)\.

3:Form

𝒩t\\mathcal\{N\}\_\{t\}using \([5](https://arxiv.org/html/2611.22210#S3.E5)\) and select

\(it,jt\)\(i\_\{t\},j\_\{t\}\)using \([6](https://arxiv.org/html/2611.22210#S3.E6)\)\.

4:Set

yt=ytity\_\{t\}=y\_\{t\}^\{i\_\{t\}\}and

yt′=ytjty\_\{t\}^\{\\prime\}=y\_\{t\}^\{j\_\{t\}\}, observe

𝐳t​\(yt\),𝐳t​\(yt′\)\\mathbf\{z\}\_\{t\}\(y\_\{t\}\),\\mathbf\{z\}\_\{t\}\(y\_\{t\}^\{\\prime\}\)and query the preference label

oto\_\{t\}\.

5:Compute

𝐝t=𝐳t​\(yt\)−𝐳t​\(yt′\)\\mathbf\{d\}\_\{t\}=\\mathbf\{z\}\_\{t\}\(y\_\{t\}\)\-\\mathbf\{z\}\_\{t\}\(y\_\{t\}^\{\\prime\}\)and construct

CtC\_\{t\}using \([2](https://arxiv.org/html/2611.22210#S3.E2)\)\.

6:Obtain

𝐖t\\mathbf\{W\}\_\{t\}using \([3](https://arxiv.org/html/2611.22210#S3.E3)\) and update the score predictor with the observed scores\.

7:Update

πt\+1\\pi\_\{t\+1\}using

\(𝐱t,yt,yt′,ot\)\(\\mathbf\{x\}\_\{t\},y\_\{t\},y\_\{t\}^\{\\prime\},o\_\{t\}\)via \([1](https://arxiv.org/html/2611.22210#S2.E1)\)\.

Response pair selection\.After filtering out confidently dominated responses under the preference cone, we select a pair from𝒩t\\mathcal\{N\}\_\{t\}that maximizes the weaker of the two opposing preference margins:

\(it,jt\)∈arg​maxi,j∈𝒩t,i≠j​min​\{maxk∈\[Kt−1\]⁡LCBt,k⁡\(yti∣ytj\),−mink∈\[Kt−1\]⁡UCBt,k⁡\(yti∣ytj\)\}\.\(i\_\{t\},j\_\{t\}\)\\in\\underset\{i,j\\in\\mathcal\{N\}\_\{t\},\\ i\\neq j\}\{\\argmax\}\\min\\\!\\left\\\{\\max\_\{k\\in\[K\_\{t\-1\}\]\}\\operatorname\{LCB\}\_\{t,k\}\(y\_\{t\}^\{i\}\\mid y\_\{t\}^\{j\}\),\-\\min\_\{k\\in\[K\_\{t\-1\}\]\}\\operatorname\{UCB\}\_\{t,k\}\(y\_\{t\}^\{i\}\\mid y\_\{t\}^\{j\}\)\\right\\\}\.\(6\)The two terms are positive when some rule confidently favorsytiy\_\{t\}^\{i\}andytjy\_\{t\}^\{j\}, respectively\. A positive minimum indicates conflicting preferences across rules and measures the weaker side of the disagreement\. Maximizing it favors pairs with strong evidence on both sides of the trade\-off\. The algorithm is summarized in Alg\.[1](https://arxiv.org/html/2611.22210#alg1)\. In implementation, we resample when fewer than two candidates survive, skip cone updates for zero score differences, and retain the previous cone if its update yields a trivial dual\.

### 3\.4Theoretical Analysis

In this part, we analyze the theoretical property of the proposed algorithm\. We note that our theoretical analysis provides a supporting explanation of the filtering mechanism by bounding the frequency of cone\-dominated selections under simplifying assumptions\. The empirical evaluation further examines the benefits of adaptive cone construction and trade\-off selection for downstream performance\.

We introduce the following assumption that requires the queried pair to provide sufficient information about a comparison that establishes the non\-optimality of each selected response\.

###### Assumption 2\(Coverage of a dominating response\)\.

Letqt=∑i∈Bt‖ϕ⁡\(yti\)‖Vt−12q\_\{t\}=\\sum\_\{i\\in B\_\{t\}\}\\\|\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}^\{2\}denote the aggregate feature uncertainty of the queried pair\. There exists a constantκ\>0\\kappa\>0, independent oftt, such that for every selectedi∈Bt∖PCt−1∗i\\in B\_\{t\}\\setminus P\_\{C\_\{t\-1\}\}^\{\*\}, somej∈𝒟t​\(i\)j\\in\\mathcal\{D\}\_\{t\}\(i\)attaining the maximum definingΔt,i\\Delta\_\{t,i\}satisfies

‖ϕ⁡\(ytj\)−ϕ⁡\(yti\)‖Vt−12≤κ​qt\.\\\|\\phi\(y\_\{t\}^\{j\}\)\-\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}^\{2\}\\leq\\kappa q\_\{t\}\.\(7\)

We bound regret by controlling the score\-estimation uncertainty of responses dominated underCt−1C\_\{t\-1\}\.

###### Theorem 1\.

Under Assumptions[1](https://arxiv.org/html/2611.22210#ThmmyAssum1)and[2](https://arxiv.org/html/2611.22210#ThmmyAssum2), letλ≥1\\lambda\\geq 1andδ∈\(0,1\)\\delta\\in\(0,1\)\. With probability at least1−δ1\-\\delta,MPG\-DPOsatisfies, simultaneously for all generated query prefixes of lengthT≥1T\\geq 1and any choicesat∈Bta\_\{t\}\\in B\_\{t\},

Regret01⁡\(T\)≤12​α¯T2​\(δ\)​κ​pΔ2​log⁡\(1\+2​Tλ​p\),\\operatorname\{Regret\}^\{01\}\(T\)\\leq\\frac\{12\\bar\{\\alpha\}\_\{T\}^\{2\}\(\\delta\)\\kappa p\}\{\\Delta^\{2\}\}\\log\\left\(1\+\\frac\{2T\}\{\\lambda p\}\\right\),\(8\)whereα¯T​\(δ\)=λ​S\+σz​d⁡\(2​log⁡dδ\+p​log⁡\(1\+2​Tλ​p\)\)\\bar\{\\alpha\}\_\{T\}\(\\delta\)=\\sqrt\{\\lambda\}S\+\\sigma\_\{z\}\\sqrt\{d\\left\(2\\log\\frac\{d\}\{\\delta\}\+p\\log\\left\(1\+\\frac\{2T\}\{\\lambda p\}\\right\)\\right\)\}\.

Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1)shows that, if queries continue indefinitely, the average regret vanishes asT→∞T\\to\\inftywith probability at least1−δ1\-\\delta\. The full proof is in Appendix[B](https://arxiv.org/html/2611.22210#A2), which also shows that the round\-level regret, counting a round once whenever at least one selected response is non\-optimal underCt−1C\_\{t\-1\}, satisfies the same bound; summing the indicator over both responses instead yields twice this bound\.

###### Proposition 1\(Avoidance of cone\-dominated queries\)\.

Under the conditions of Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1), with fixed problem parameters andδ\\delta, for every generated prefix ofTTexpensive queries, at leastT−O⁡\(log2⁡T\)T\-O\(\\log^\{2\}T\)compare two Pareto\-optimal responses in their respective candidate pools under the cones available before the queries, with probability at least1−δ1\-\\delta\. If queries continue indefinitely, the fraction allocated to such non\-dominated pairs approaches one\.

## 4Experiments

In this section, we first introduce the experimental setup and then present the experimental results\. Our experiments aim to answer the following three questions:Q1: does the proposed method enhance the policy across different LLM families;Q2: does the proposed preference\-guided sampling make better use of the multidimensional information than other methods with additional access to it; andQ3: how sensitive is the proposed method to its key parameters\.

### 4\.1Experimental Setup

Prompts\.We use the prompt\-only split ofultrafeedback\-promptfor online training, following the generation and training pipeline of the official XPO implementation intrl222[https://huggingface\.co/docs/trl/xpo\_trainer](https://huggingface.co/docs/trl/xpo_trainer)\. All methods share this pipeline\. For MPG\-DPO, we initialize the preference matrix using comparisons collected from the preference oracle onultrafeedback\_binarized\.

Models\.We evaluate three model families:Qwen2\.5\-7B\-Instruct,Mistral\-7B\-Instruct\-v0\.3, andOLMo\-2\-1124\-7B\-Instruct\. Each reference policy is initialized from the corresponding policy checkpoint and remains frozen throughout training\. Unless otherwise specified, MPG\-DPO samplesN=8N=8candidate responses per prompt from the current policy\. All methods use the same prompt stream, maximum response length, decoding parameters, optimizer, and preference\-query budget\. We train with LoRA adapters\([Hu et al\., 2022](https://arxiv.org/html/2611.22210#bib.bib31)\)using rankr=32r=32and scaling parameterα=64\\alpha=64, and report results over three random seeds\.ϕ\\phiis meanwhile implemented as an independent feature extractor, which we fix to use an 86M parameter frozenmicrosoft/deberta\-v3\-base333[https://huggingface\.co/microsoft/deberta\-v3\-base](https://huggingface.co/microsoft/deberta-v3-base)model on the prompt\-response pair\.

Datasets and evaluation metrics\.We report four metrics: \(i\) average reward from a held\-out reward model on theIMDbsentiment generation task\([Maas et al\., 2011](https://arxiv.org/html/2611.22210#bib.bib25)\), \(ii\) summarization win rate against human\-written summaries onTL;DR\([Völske et al\., 2017](https://arxiv.org/html/2611.22210#bib.bib24)\), \(iii\) length\-controlled win rate onAlpacaEval2\([Dubois et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib29)\), and \(iv\) win rate onArena\-Hard\([Li et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib30)\)\.

Preference oracle\.We useKimi\-K3\([Team et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib28)\)to simulate human preference feedback\. Each query uses a fixed judging template containing the input prompt and two candidate responses, whose order is randomized independently for each query\. The oracle identifies the preferred response, yielding a binary preference label\. Each oracle call counts as one preference query\. Preference labeling and sub\-objective scoring are performed by separate models\.

Sub\-objective oracle\.We use five fixed evaluation dimensions adapted fromFLASK\([Ye et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib26)\): correctness, completeness, clarity, conciseness, and reasoning quality\. The corresponding scoring rubrics are generated by prompting the preference oracle\. A frozenPrometheus2evaluator \(prometheus\-7b\-v2\.0\)\([Kim et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib27)\)applies these rubrics to assign an integer score from 1 to 5 for each dimension\. The scores are normalized to\[0,1\]\[0,1\]to obtain𝐳t​\(y\)\\mathbf\{z\}\_\{t\}\(y\)\.

Contenders\.We compare all methods under the same preference\-query budget\. We compared our approach against state\-of\-the\-art contenders with official code implementations\. Online DPO samples an on\-policy response pair and applies the standard DPO update\. XPO incorporates the exploratory objective of[Xie et al\. \(2025\)](https://arxiv.org/html/2611.22210#bib.bib4), while APL performs active preference selection following[Oh et al\. \(2026\)](https://arxiv.org/html/2611.22210#bib.bib10)\. APL\+MD modifies the APL selection algorithm by using a weighted average of the reward margin and the sum of squared ridge\-prediction confidence widths between pairs of responses\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2611.22210#S4.T1)compares MPG\-DPO with state\-of\-the\-art baselines across three LLM families under the same preference\-query budget\. The consistent improvements across model families support the effectiveness of preference\-guided pair selection\.

Table 1:Main results across three 7B policies under the same preference\-query budget\. Higher is better for every metric\. Each entry reports mean±\\pmstandard deviation over three seeds\.We also assess the contribution of each component in MPG\-DPO through the following ablations:

- 1\.Without cone filtering: We omit candidate removal and apply the trade\-off selection criterion to all pairs in the full candidate pool\. This variant assesses the benefit of the filtering mechanism\.
- 2\.Without trade\-off selection: We retain cone filtering but select a pair uniformly at random from𝒩t\\mathcal\{N\}\_\{t\}\. This variant assesses the benefit of trade\-off selection beyond filtering alone\.
- 3\.Fixed Pareto cone: We replace the learned cone with the positive orthant by setting𝐖t=I5\\mathbf\{W\}\_\{t\}=I\_\{5\}throughout training, while retaining the filtering and trade\-off selection procedures\. The score predictor and policy continue to update\. This variant compares learned preference trade\-offs with a fixed coordinate\-wise preference relation\.

The results in Table[2](https://arxiv.org/html/2611.22210#S4.T2)show that the full method outperforms each ablated variant, supporting the contribution of each component of the proposed algorithm\. With Table[1](https://arxiv.org/html/2611.22210#S4.T1), these findings answerQ1\.

Table 2:Component ablations onQwen2\.5\-7B\-Instructunder a fixed preference\-query budget\. Each entry reports mean±\\pmstandard deviation over three independent training seeds\.
### 4\.3Comparison with Alternative Uses of Multidimensional Information

We compare MPG\-DPO with alternative pair\-selection strategies that use the same multidimensional feedback\. Feedback budgets and training protocol are fixed, with only the selection criterion varied\.

We construct three information\-enhanced contenders:

- 1\.Scalar\-Margin selects the pair with the largest predicted scalar reward difference from the pool\.
- 2\.APL\+MD\-Score selects the pair with the largest sum of squared ridge\-prediction confidence widths across its two responses, prioritizing uncertainty in the estimated multidimensional scores\.
- 3\.APL\+MD\-Scalar fits a Bradley–Terry model\([Bradley and Terry, 1952](https://arxiv.org/html/2611.22210#bib.bib6)\)to historical five\-dimensional score differences and preference labels\. Treating the predicted scores as fixed, it selects the pair with the largest approximate information gain about the scalarization weights\.

These strategies provide comparisons with preference\-cone\-guided selection based on scalar reward differences, score\-estimation uncertainty, and uncertainty in preference weights\.

All methods share the same prompt stream, predictor architecture, warm\-up data, scoring and preference\-query budgets, and DPO hyperparameters\. Only queried responses receive oracle scores\. We evaluateQwen2\.5\-7B\-InstructandMistral\-7B\-Instruct\-v0\.3over three independent training seeds\. The results in Table[3](https://arxiv.org/html/2611.22210#S4.T3)support the effectiveness of preference\-cone\-guided selection relative to these alternative uses of multidimensional feedback, answeringQ2\.

Table 3:Comparison of multidimensional feedback utilization across two policy models under matched scoring and preference\-query budgets, withN=8N=8\.
### 4\.4Ablation Studies

We examine the sensitivity of MPG\-DPO to the evaluation models, the candidate\-pool size, the number of retained preference rules, and the number of sub\-objectives\.

Effectiveness of personalization\.We evaluate adaptation to different oracle\-defined preferences usingQwen2\.5\-7Bas the backbone\. In each setting, one ofKimi\-K2\.6,DeepSeek\-V4\.1\-Flash, orQwen3\.8\-27Bserves as the oracle, providing preference feedback during training and computing win rates onAlpacaEval2andArena\-Hardduring evaluation\. As shown in Table[4](https://arxiv.org/html/2611.22210#S4.T4), MPG\-DPO achieves higher mean win rates than the evaluated baselines in all three settings, supporting its effectiveness in adapting to different oracle\-defined preferences\.

Candidate\-pool size\.We varyN∈\{2,4,8,16\}N\\in\\\{2,4,8,16\\\}while holding the preference\-query budget and DPO hyperparameters fixed\. Larger pools provide more candidates for pair selection but increase response\-generation cost\. Figure[2\(a\)](https://arxiv.org/html/2611.22210#S4.F2.sf1)reports win rates onAlpacaEval2andArena\-Hard\. Performance initially improves with pool size, with diminishing gains at larger values ofNN\.

Number of retained preference\.We vary the maximum number of rows retained in𝐖t\\mathbf\{W\}\_\{t\}to examine how the number of preferences affects pair selection\. As shown in Figure[2\(b\)](https://arxiv.org/html/2611.22210#S4.F2.sf2), performance improves as the number of retained rules increases from small values, then declines slightly at larger settings\. Intermediate settings achieve the highest win rates within the evaluated range\.

Number of sub\-objectives\.We vary the number of sub\-objectives by randomly selecting subsets of theFLASKevaluation dimensions\. Figure[2\(c\)](https://arxiv.org/html/2611.22210#S4.F2.sf3)shows that including more dimensions improves downstream performance within the evaluated range\. Since the scoring rubrics are generated by the preference oracle, this trend may partly reflect the relevance of these dimensions to the target preferences\. The interaction between the number of sub\-objectivesddand the number of retained rulesKKwarrants further investigation\.

Table 4:Ablation study on the effectiveness of personalization\.These results answerQ3\. Additional ablation results are provided in Appendix[A](https://arxiv.org/html/2611.22210#A1)\.

\(\(a\)\)Pool sizeNN\.\(\(b\)\)Number of preference rulesKK\.\(\(c\)\)Number of sub\-objectivesdd\.
Figure 2:Sensitivity studies with reported win rates of tunedQwen2\.5\-7Bmodels versus the base checkpoint\.

## 5Related Work

This section reviews relevant prior work in detail\.

Exploration in online preference alignment\.Online preference alignment repeatedly generates responses from the current policy, obtains preference feedback, and updates the policy on the newly collected data\([Calandriello et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib13)\)\. A fruitful line of research aims to enhance policy\-level coverage and generate high\-value responses using a scalar implicit reward\. XPO adds a principled exploration bonus to online DPO so that the policy can move beyond the support of the initial model and discover responses with higher latent reward\([Xie et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib4)\)\. VPO similarly introduces optimism through value regularization, while recent theoretical work studies when preference\-based exploration avoids poor dependence on the reward range\([Cen et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib5);[Chen et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib9)\)\. These methods primarily modify the DPO loss and can be seamlessly integrated into our algorithm\.

Active preference selection\.A second line of work assumes a collection of candidate comparisons and actively decides which examples should receive costly preference labels\([Das et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib11);[Lin et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib12);[Oh et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib10)\)\. APO formulates RLHF as a contextual preference bandit and allocates queries to statistically important prompt–response comparisons\([Das et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib11)\)\. ActiveDPO derives a selection score from uncertainty in the LLM’s implicit reward difference and uses gradient projection and batch selection to make this criterion practical for nonlinear language models\([Lin et al\., 2026](https://arxiv.org/html/2611.22210#bib.bib12)\)\. Other work designs risk\-adjusted active\-learning criteria for preference optimization\([Belakaria et al\., 2025](https://arxiv.org/html/2611.22210#bib.bib20)\)\. These approaches rank candidates using scalar reward uncertainty while not exploiting the sub\-objective trade\-offs considered in our setting\. Beyond preference learning,[Wang et al\. \(2023\)](https://arxiv.org/html/2611.22210#bib.bib21)resolved a fundamental identifiability question through sound and complete orientation rules for local background knowledge in their latent\-confounding setting\. Their principled active causal discovery framework used accumulated knowledge to guide intervention selection, offering a complementary perspective on our preference\-guided response\-pair selection\.

Constitutional AI and rule\-guided alignment\.Constitutional AI replaces repeated human supervision with a written set of high\-level principles that guides self\-critique, response revision, and AI\-generated preference labels\([Bai et al\., 2022](https://arxiv.org/html/2611.22210#bib.bib17)\)\.[Lee et al\. \(2024\)](https://arxiv.org/html/2611.22210#bib.bib18)further study empirically whether an external language model can provide preference supervision at a quality comparable to human feedback, while collective constitutional AI obtains the constitution through a public deliberation process rather than solely from model developers\([Huang et al\., 2024](https://arxiv.org/html/2611.22210#bib.bib19)\)\. These methods demonstrate that explicit rules can make alignment supervision more scalable and interpretable\. However, their constitutions are specified as natural\-language principles and are typically shared across interactions\.

Pareto learning\.The preference Pareto cone originates from[Censor \(1977\)](https://arxiv.org/html/2611.22210#bib.bib22)and has recently been used by[Rame et al\. \(2023\)](https://arxiv.org/html/2611.22210#bib.bib2)for reward interpolation and by[Ma et al\. \(2026\)](https://arxiv.org/html/2611.22210#bib.bib23)in multi\-objective bandits\. We adopt the same geometric abstraction but differ in both construction and use: our cone is estimated online from noisy pairwise comparisons via a consistency check, and is adaptively updated as new comparisons arrive to actively guide response selection\.

## 6Conclusion

In this paper, we proposed MPG\-DPO, an online preference alignment method that incorporates historical user preferences into response selection alongside model uncertainty for sample efficiency\. Using low\-cost multidimensional evaluations, the method constructs a personalized preference cone to filter confidently dominated responses and select pairs exhibiting unresolved trade\-offs\. This selection mechanism accounts for preference information already implied by past user comparisons\. Under the stated assumptions, our analysis suggests that the number of selected cone\-dominated responses grows slowly, so such queries make up a vanishing fraction as learning progresses\. Experiments across multiple LLM families show improvements over strong baselines under matched preference\-query budgets, supporting the effectiveness of preference\-cone\-guided selection\.

## References

- Abbasi\-Yadkoriet al\.\(2011\)Y\. Abbasi\-Yadkori, D\. Pál, and C\. SzepesváriImproved algorithms for linear stochastic bandits\.Advances in Neural Information Processing Systems \(NeurIPS\)24,pp\. 2312–2320\.Cited by:[§B\.1](https://arxiv.org/html/2611.22210#A2.SS1.p2.1)\.
- Baiet al\.\(2022\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.Constitutional AI: harmlessness from AI feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p3.1),[§5](https://arxiv.org/html/2611.22210#S5.p4.1)\.
- Belakariaet al\.\(2025\)S\. Belakaria, J\. Kazdan, C\. Marx, C\. Cundy, W\. Neiswanger, S\. Koyejo, B\. E\. Engelhardt, and S\. ErmonSharpe ratio\-guided active learning for preference optimization in RLHF\.InSecond Conference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§5](https://arxiv.org/html/2611.22210#S5.p3.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[item 3](https://arxiv.org/html/2611.22210#S4.I2.i3.p1.1)\.
- Calandrielloet al\.\(2024\)D\. Calandriello, Z\. D\. Guo, R\. Munos, M\. Rowland, Y\. Tang, B\. A\. Pires, P\. H\. Richemond, C\. Le Lan, M\. Valko, T\. Liu, R\. Joshi, Z\. Zheng, and B\. PiotHuman alignment of large language models through online preference optimisation\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 211–238\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.22210#S2.SS1.p3.1),[§5](https://arxiv.org/html/2611.22210#S5.p2.1)\.
- Cenet al\.\(2025\)S\. C\. Cen, J\. C\. Mei, K\. Goshvadi, H\. J\. Dai, T\. Yang, S\. Yang, D\. Schuurmans, Y\. J\. Chi, and B\. DaiValue\-incentivized preference optimization: a unified approach to online and offline rlhf\.InThe 13th International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§5](https://arxiv.org/html/2611.22210#S5.p2.1)\.
- Censor \(1977\)Y\. CensorPareto optimality in multiobjective problems\.Applied Mathematics and Optimization4\(1\),pp\. 41–59\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p4.1),[§5](https://arxiv.org/html/2611.22210#S5.p5.1)\.
- Chenet al\.\(2025\)M\. Chen, Y\. Chen, W\. Sun, and X\. ZhangAvoiding exp \(r\) scaling in rlhf through preference\-based exploration\.Advances in Neural Information Processing Systems \(NeurIPS\)38,pp\. 164474–164510\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§5](https://arxiv.org/html/2611.22210#S5.p2.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.Advances in Neural Information Processing Systems \(NeurIPS\)30,pp\. 4299–4307\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p1.1)\.
- Daset al\.\(2025\)N\. Das, S\. Chakraborty, A\. Pacchiano, and S\. R\. ChowdhuryActive preference optimization for sample efficient RLHF\.InJoint European Conference on Machine Learning and Knowledge Discovery in Databases,pp\. 96–112\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§5](https://arxiv.org/html/2611.22210#S5.p3.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p3.1)\.
- Fukuda and Prodon \(1995\)K\. Fukuda and A\. ProdonDouble description method revisited\.InFranco\-Japanese and Franco\-Chinese conference on combinatorics and computer science,pp\. 91–111\.Cited by:[§3\.2](https://arxiv.org/html/2611.22210#S3.SS2.p3.1)\.
- Guoet al\.\(2024\)S\. Guo, B\. Zhang, T\. Liu, T\. Liu, M\. Khalman, F\. Llinares, A\. Rame, T\. Mesnard, Y\. Zhao, B\. Piot,et al\.Direct language model alignment from online ai feedback\.arXiv preprint arXiv:2402\.04792\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.22210#S2.SS1.p3.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InThe 10th International Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p2.1)\.
- Huanget al\.\(2024\)S\. Huang, D\. Siddarth, L\. Lovitt, T\. I\. Liao, E\. Durmus, A\. Tamkin, and D\. GanguliCollective constitutional AI: aligning a language model with public input\.InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency,pp\. 1395–1417\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p3.1),[§5](https://arxiv.org/html/2611.22210#S5.p4.1)\.
- Kimet al\.\(2024\)S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. SeoPrometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 4334–4353\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p5.1)\.
- Leeet al\.\(2024\)H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. PrakashRLAIF vs\. RLHF: scaling reinforcement learning from human feedback with AI feedback\.Proceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 26874–26901\.Cited by:[§5](https://arxiv.org/html/2611.22210#S5.p4.1)\.
- Liet al\.\(2025\)T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. StoicaFrom crowdsourced data to high\-quality benchmarks: arena\-hard and benchbuilder pipeline\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),pp\. 1340–1363\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p3.1)\.
- Linet al\.\(2026\)X\. Lin, A\. Verma, Z\. Dai, D\. Rus, S\. Ng, and B\. K\. H\. LowActiveDPO: active direct preference optimization for sample\-efficient alignment\.InThe 14th International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§5](https://arxiv.org/html/2611.22210#S5.p3.1)\.
- Maet al\.\(2026\)L\. Ma, Z\. Zhang, M\. Sugiyama, and Z\. ZhouSelective ensemble based on preference\-directed multi\-objective bandits\.arXiv preprint arXiv:2606\.21929\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p4.1),[§5](https://arxiv.org/html/2611.22210#S5.p5.1),[Remark 1](https://arxiv.org/html/2611.22210#ThmmyRemark1.p1.1.1)\.
- Maaset al\.\(2011\)A\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. PottsLearning word vectors for sentiment analysis\.InProceedings of the 49th annual meeting of the Association for Computational Linguistics \(ACL\),pp\. 142–150\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p3.1)\.
- Motzkinet al\.\(1953\)T\. Motzkin, G\. Thompson, and R\. ThrallThe double description method\.Contributions to the Theory of Games\(24\),pp\. 51\.Cited by:[§3\.2](https://arxiv.org/html/2611.22210#S3.SS2.p3.1)\.
- Ohet al\.\(2026\)G\. Oh, J\. Lee, J\. Park, Y\. Yu, W\. Bae, and J\. NohRandom is hard to beat: active selection in online dpo with modern llms\.arXiv preprint arXiv:2604\.02766\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p6.1),[§5](https://arxiv.org/html/2611.22210#S5.p3.1)\.
- Ohet al\.\(2025\)M\. Oh, S\. Lee, and J\. OkComparison\-based active preference learning for multi\-dimensional personalization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 33145–33166\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p4.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems \(NeurIPS\)35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems \(NeurIPS\)36,pp\. 53728–53741\.Cited by:[§2\.1](https://arxiv.org/html/2611.22210#S2.SS1.p3.1)\.
- Rameet al\.\(2023\)A\. Rame, G\. Couairon, C\. Dancette, J\. Gaya, M\. Shukor, L\. Soulier, and M\. CordRewarded soups: towards pareto\-optimal alignment by interpolating weights fine\-tuned on diverse rewards\.Advances in Neural Information Processing Systems \(NeurIPS\)36,pp\. 71095–71134\.Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p4.1),[§5](https://arxiv.org/html/2611.22210#S5.p5.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, J\. Cai, X\. Cai, P\. Cao, Y\. Cao, Z\. Chai, Y\. Charles,et al\.Kimi k3: open frontier intelligence\.arXiv preprint arXiv:2607\.24653\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p4.1)\.
- Völskeet al\.\(2017\)M\. Völske, M\. Potthast, S\. Syed, and B\. SteinTl; dr: mining reddit to learn automatic summarization\.InProceedings of the workshop on new frontiers in summarization,pp\. 59–63\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p3.1)\.
- Wanget al\.\(2023\)T\. Wang, T\. Qin, and Z\. ZhouSound and complete causal identification with latent variables given local background knowledge\.Artificial Intelligence322,pp\. 103964\.Cited by:[§5](https://arxiv.org/html/2611.22210#S5.p3.1)\.
- Xieet al\.\(2025\)T\. Y\. Xie, D\. J\. Foster, A\. Krishnamurthy, C\. Rosset, A\. H\. Awadallah, and A\. RakhlinExploratory preference optimization: harnessing implicit q\*\-approximation for sample\-efficient rlhf\.InThe 13th International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2611.22210#S1.p1.1),[§1](https://arxiv.org/html/2611.22210#S1.p2.1),[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p6.1),[§5](https://arxiv.org/html/2611.22210#S5.p2.1)\.
- Yeet al\.\(2024\)S\. Ye, D\. Kim, S\. Kim, H\. Hwang, S\. Kim, Y\. Jo, J\. Thorne, J\. Kim, and M\. SeoFlask: fine\-grained language model evaluation based on alignment skill sets\.InThe 12th International Conference on Learning Representations \(ICLR\),Vol\.2024,pp\. 55361–55414\.Cited by:[§4\.1](https://arxiv.org/html/2611.22210#S4.SS1.p5.1)\.

## Appendix AAdditional Experimental Details

### A\.1Setup Details

#### Experimental Setup\.

We run all online DPO variants for a fixed number of steps with a matched global batch size across all settings\. Specific hyperparameters are listed in Table[5](https://arxiv.org/html/2611.22210#A1.T5)\.

Table 5:Online DPO hyperparametersFor MPG\-DPO, we use additional hyper\-parameters of pool sizeN=8N=8,d=5d=5sub\-objectives andK=8K=8\. We further useτ=0\.1\\tau=0\.1andλ=1\.0\\lambda=1\.0by default\. We useδ=0\.1\\delta=0\.1,S=1\.0S=1\.0, withαt​\(δ\)\\alpha\_\{t\}\(\\delta\)computed as detailed in Section[3\.3](https://arxiv.org/html/2611.22210#S3.SS3)\.

The human preference oracle is queried only once per update to determine the winning/losing response in the chosen pair\. ForWtW\_\{t\}initialization, a fixed amount of preference responses can be provided for initial computation, which can be conducted as an offline pre\-processing step\.

#### Computational Environment\.

All experiments were conducted on a single machine with 4 NVIDIA H100 80GB GPUs\. Experiments were run in an environment using CUDA version 12\.8 and PyTorch 2\.7\.1\.

### A\.2Additional Results

Figure 3:Number of queries forWtW\_\{t\}warm\-up\.Figure 4:Filtering rate of responses during alignment\.Query efficiency\.We next verify the importance of the warm\-up step to constructWtW\_\{t\}\. Here, we useQwen2\.5\-7B\-InstructandMistral\-7B\-Instruct\-v0\.3and vary the amount of initial data necessary forWtW\_\{t\}construction\. We re\-use the same alignment procedure as described before\. In[Figure 4](https://arxiv.org/html/2611.22210#A1.F4), we can see that a moderate amount of pre\-labeled samples is important for properly initializingWtW\_\{t\}\. While the complete lack of any initial queries causes training to progress more slowly, which is sensible due to the need to constructWtW\_\{t\}completely from online pair\-wise comparisons, using a moderate amount of warm\-up labels can ensure high performance\.

Cone construction\.As a final verification, we also audit the cone construction process in Eq\.[2](https://arxiv.org/html/2611.22210#S3.E2)\. We take the tuned model at different stages of tuning \(25%, 50%, 75%, and 100%\) and measure the filtering rate \(removal of responses in the pool\)\. We observe in[Figure 4](https://arxiv.org/html/2611.22210#A1.F4)that as the tuning progresses and both the cone as well asWtW\_\{t\}can encode increasing information about the model preferences, the model is able to filter out an increasing number of potentially less\-informative responses within the pool\.

Varying Sub\-Objective Oracle\.We also run a comparison by changing the use of the sub\-objective oracle\. As a simple replacement, we prompt aKimi\-K2\.6using the sameFlaskobjectives and templates used for Prometheus, obtaining scores from the new model for training and tuning with the setup described for the main experimental results\.

Table 6:Results usingKimi\-K2\.6sub\-objective oracle\. Each entry reports mean±\\pmstandard deviation over three seeds\.

## Appendix BProof of Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1)

We first review the notations\. At roundtt, the learner usesCt−1C\_\{t\-1\}and𝐖t−1\\mathbf\{W\}\_\{t\-1\}constructed from previous feedback, together withVtV\_\{t\}andθ^t\\widehat\{\\theta\}\_\{t\}fitted to noisy scores from roundss<ts<t\. The current queried scores and preference label are observed after selection\. The operational confidence bounds use

m^t,k​\(y∣y′\)=𝐰t−1,k⊤​θ^t⊤​\[ϕ⁡\(y\)−ϕ⁡\(y′\)\]Ut−1,k−Lt−1,k,ct,k​\(y,y′\)=αt​\(δ\)​‖ϕ⁡\(y\)−ϕ⁡\(y′\)‖Vt−1Ut−1,k−Lt−1,k,\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)=\\frac\{\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}\\widehat\{\\theta\}\_\{t\}^\{\\top\}\[\\phi\(y\)\-\\phi\(y^\{\\prime\}\)\]\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\},\\qquad c\_\{t,k\}\(y,y^\{\\prime\}\)=\\frac\{\\alpha\_\{t\}\(\\delta\)\\\|\\phi\(y\)\-\\phi\(y^\{\\prime\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\},withk∈\[Kt−1\]k\\in\[K\_\{t\-1\}\]andmt,km\_\{t,k\}obtained by replacingθ^t\\widehat\{\\theta\}\_\{t\}withθ∗\\theta^\{\*\}\. All optimal sets are evaluated on the latent scores𝐱t,i=μ⁡\(𝐱t,yti\)\\mathbf\{x\}\_\{t,i\}=\\mu\(\\mathbf\{x\}\_\{t\},y\_\{t\}^\{i\}\)in the current candidate pool\. Regret,Δt,i\\Delta\_\{t,i\}, and𝒟t​\(i\)\\mathcal\{D\}\_\{t\}\(i\)in Assumption[1](https://arxiv.org/html/2611.22210#ThmmyAssum1)all use the same coneCt−1C\_\{t\-1\}as selection\. In particular,PCt−1∗P\_\{C\_\{t\-1\}\}^\{\*\}is evaluated on the current pool, before observing the current preference label\. We useqt=∑i∈Bt‖ϕ⁡\(yti\)‖Vt−12q\_\{t\}=\\sum\_\{i\\in B\_\{t\}\}\\\|\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}^\{2\}as defined in Section[3\.4](https://arxiv.org/html/2611.22210#S3.SS4)\.

### B\.1Technical Lemmas

In this part, we introduce necessary technical lemmas\.

###### Lemma 1\(Simultaneous directional confidence bound\)\.

Under the score model in Section[3\.1](https://arxiv.org/html/2611.22210#S3.SS1), for anyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta, simultaneously for every roundtt, every𝐯∈ℝp\\mathbf\{v\}\\in\\mathbb\{R\}^\{p\}, and every nonzero𝐰∈ℝd\\mathbf\{w\}\\in\\mathbb\{R\}^\{d\},

\|𝐰⊤​\(θ^t−θ∗\)⊤​𝐯\|‖𝐰‖1≤αt​\(δ\)​‖𝐰‖2‖𝐰‖1​‖𝐯‖Vt−1\.\\frac\{\|\\mathbf\{w\}^\{\\top\}\(\\widehat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\)^\{\\top\}\\mathbf\{v\}\|\}\{\\\|\\mathbf\{w\}\\\|\_\{1\}\}\\leq\\alpha\_\{t\}\(\\delta\)\\frac\{\\\|\\mathbf\{w\}\\\|\_\{2\}\}\{\\\|\\mathbf\{w\}\\\|\_\{1\}\}\\\|\\mathbf\{v\}\\\|\_\{V\_\{t\}^\{\-1\}\}\.\(9\)Consequently,LCBt,k⁡\(y∣y′\)≤mt,k​\(y∣y′\)≤UCBt,k⁡\(y∣y′\)\\operatorname\{LCB\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)\\leq m\_\{t,k\}\(y\\mid y^\{\\prime\}\)\\leq\\operatorname\{UCB\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)simultaneously for all rounds, response pairs, and rows of𝐖t−1\\mathbf\{W\}\_\{t\-1\}\.

Proof of Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1)\.Order the queried score vectors chronologically, using a fixed order within each pair\. The corresponding feature vectors are known before each score is observed\. By the self\-normalized concentration inequality of[Abbasi\-Yadkori et al\. \(2011, Theorem 1\)](https://arxiv.org/html/2611.22210#bib.bib32)and a union bound over theddcoordinates, with probability at least1−δ1\-\\delta, simultaneously for allttandq∈\[d\]q\\in\[d\],

‖∑s<t∑i∈Bsϕ⁡\(ysi\)​εs​\(ysi\)​\[q\]‖Vt−1≤σz​2​log⁡dδ\+log⁡detVtdet\(λ​Ip\)\.\\left\\\|\\sum\_\{s<t\}\\sum\_\{i\\in B\_\{s\}\}\\phi\(y\_\{s\}^\{i\}\)\\varepsilon\_\{s\}\(y\_\{s\}^\{i\}\)\[q\]\\right\\\|\_\{V\_\{t\}^\{\-1\}\}\\leq\\sigma\_\{z\}\\sqrt\{2\\log\\frac\{d\}\{\\delta\}\+\\log\\frac\{\\det V\_\{t\}\}\{\\det\(\\lambda I\_\{p\}\)\}\}\.Substitution of the noisy observations into the ridge estimator gives

θ^t−θ∗=Vt−1​\(∑s<t∑i∈Bsϕ⁡\(ysi\)​εs​\(ysi\)⊤−λ​θ∗\)\.\\widehat\{\\theta\}\_\{t\}\-\\theta^\{\*\}=V\_\{t\}^\{\-1\}\\left\(\\sum\_\{s<t\}\\sum\_\{i\\in B\_\{s\}\}\\phi\(y\_\{s\}^\{i\}\)\\varepsilon\_\{s\}\(y\_\{s\}^\{i\}\)^\{\\top\}\-\\lambda\\theta^\{\*\}\\right\)\.Combining the coordinate bounds in Frobenius norm and usingVt⪰λ​IpV\_\{t\}\\succeq\\lambda I\_\{p\}yields

∥Vt1/2\(θ^t−θ∗\)∥F≤σzd⁡\(2​log⁡dδ\+log⁡detVtdet\(λ​Ip\)\)\+λ∥Vt−1/2θ∗∥F≤αt\(δ\)\.\\\|V\_\{t\}^\{1/2\}\(\\widehat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\)\\\|\_\{F\}\\leq\\sigma\_\{z\}\\sqrt\{d\\left\(2\\log\\frac\{d\}\{\\delta\}\+\\log\\frac\{\\det V\_\{t\}\}\{\\det\(\\lambda I\_\{p\}\)\}\\right\)\}\+\\lambda\\\|V\_\{t\}^\{\-1/2\}\\theta^\{\*\}\\\|\_\{F\}\\leq\\alpha\_\{t\}\(\\delta\)\.Thus, for every𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}, Cauchy–Schwarz implies

\|𝐰⊤​\(θ^t−θ∗\)⊤​𝐯\|≤‖𝐰‖2​‖Vt1/2​\(θ^t−θ∗\)‖F​‖𝐯‖Vt−1≤αt​\(δ\)​‖𝐰‖2​‖𝐯‖Vt−1\.\|\\mathbf\{w\}^\{\\top\}\(\\widehat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\)^\{\\top\}\\mathbf\{v\}\|\\leq\\\|\\mathbf\{w\}\\\|\_\{2\}\\\|V\_\{t\}^\{1/2\}\(\\widehat\{\\theta\}\_\{t\}\-\\theta^\{\*\}\)\\\|\_\{F\}\\\|\\mathbf\{v\}\\\|\_\{V\_\{t\}^\{\-1\}\}\\leq\\alpha\_\{t\}\(\\delta\)\\\|\\mathbf\{w\}\\\|\_\{2\}\\\|\\mathbf\{v\}\\\|\_\{V\_\{t\}^\{\-1\}\}\.Dividing by‖𝐰‖1\>0\\\|\\mathbf\{w\}\\\|\_\{1\}\>0proves \([9](https://arxiv.org/html/2611.22210#A2.E9)\)\. Each unit\-norm row satisfies

Ut−1,k−Lt−1,k=∑q=1d\|𝐰t−1,k​\[q\]\|=‖𝐰t−1,k‖1≥‖𝐰t−1,k‖2=1\.U\_\{t\-1,k\}\-L\_\{t\-1,k\}=\\sum\_\{q=1\}^\{d\}\|\\mathbf\{w\}\_\{t\-1,k\}\[q\]\|=\\\|\\mathbf\{w\}\_\{t\-1,k\}\\\|\_\{1\}\\geq\\\|\\mathbf\{w\}\_\{t\-1,k\}\\\|\_\{2\}=1\.Taking𝐰=𝐰t−1,k\\mathbf\{w\}=\\mathbf\{w\}\_\{t\-1,k\}and𝐯=ϕ⁡\(y\)−ϕ⁡\(y′\)\\mathbf\{v\}=\\phi\(y\)\-\\phi\(y^\{\\prime\}\)gives\|mt,k​\(y∣y′\)−m^t,k​\(y∣y′\)\|≤ct,k​\(y,y′\)\|m\_\{t,k\}\(y\\mid y^\{\\prime\}\)\-\\widehat\{m\}\_\{t,k\}\(y\\mid y^\{\\prime\}\)\|\\leq c\_\{t,k\}\(y,y^\{\\prime\}\)\. The confidence event is uniform over time and all directions, so adaptive candidate pools and preference matrices require no additional union bound\.

###### Lemma 2\(Filtering with the previous preference matrix\)\.

On the event of Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1), the filter evaluated using𝐖t−1\\mathbf\{W\}\_\{t\-1\}retains every ParetoCt−1C\_\{t\-1\}\-optimal response in the current pool\. Ifi∈𝒩ti\\in\\mathcal\{N\}\_\{t\}and some responsejjsatisfies

mink∈\[Kt−1\]⁡𝐰t−1,k⊤​\[μ⁡\(ytj\)−μ⁡\(yti\)\]Ut−1,k−Lt−1,k≥Δ,\\min\_\{k\\in\[K\_\{t\-1\}\]\}\\frac\{\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}\[\\mu\(y\_\{t\}^\{j\}\)\-\\mu\(y\_\{t\}^\{i\}\)\]\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\}\\geq\\Delta,then

Δ≤2​αt​\(δ\)​‖ϕ⁡\(ytj\)−ϕ⁡\(yti\)‖Vt−1\.\\Delta\\leq 2\\alpha\_\{t\}\(\\delta\)\\\|\\phi\(y\_\{t\}^\{j\}\)\-\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}\.\(10\)If thisjjalso satisfies \([7](https://arxiv.org/html/2611.22210#S3.E7)\), thenΔ2≤4​αt2​\(δ\)​κ​qt\\Delta^\{2\}\\leq 4\\alpha\_\{t\}^\{2\}\(\\delta\)\\kappa q\_\{t\}\.

Proof of Lemma[2](https://arxiv.org/html/2611.22210#ThmmyLemma2)\.If the filter removesii, another responsejjhasLCBt,k⁡\(ytj∣yti\)\>0\\operatorname\{LCB\}\_\{t,k\}\(y\_\{t\}^\{j\}\\mid y\_\{t\}^\{i\}\)\>0for everyk∈\[Kt−1\]k\\in\[K\_\{t\-1\}\]\. Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1)implies𝐖t−1​\[μ⁡\(ytj\)−μ⁡\(yti\)\]\>𝟎\\mathbf\{W\}\_\{t\-1\}\[\\mu\(y\_\{t\}^\{j\}\)\-\\mu\(y\_\{t\}^\{i\}\)\]\>\\bm\{0\}, and hencej≻Ct−1ij\\succ\_\{C\_\{t\-1\}\}i\. Thus no ParetoCt−1C\_\{t\-1\}\-optimal response in the current pool is removed\.

For a retainediiand ajjsatisfying the stated margin condition, somek∈\[Kt−1\]k\\in\[K\_\{t\-1\}\]hasLCBt,k⁡\(ytj∣yti\)≤0\\operatorname\{LCB\}\_\{t,k\}\(y\_\{t\}^\{j\}\\mid y\_\{t\}^\{i\}\)\\leq 0\. The operational true marginmt,km\_\{t,k\}uses𝐰t−1,k\\mathbf\{w\}\_\{t\-1,k\}, so Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1)gives

0≥LCBt,k⁡\(ytj∣yti\)≥mt,k​\(ytj∣yti\)−2​ct,k​\(ytj,yti\)≥Δ−2​ct,k​\(ytj,yti\)\.0\\geq\\operatorname\{LCB\}\_\{t,k\}\(y\_\{t\}^\{j\}\\mid y\_\{t\}^\{i\}\)\\geq m\_\{t,k\}\(y\_\{t\}^\{j\}\\mid y\_\{t\}^\{i\}\)\-2c\_\{t,k\}\(y\_\{t\}^\{j\},y\_\{t\}^\{i\}\)\\geq\\Delta\-2c\_\{t,k\}\(y\_\{t\}^\{j\},y\_\{t\}^\{i\}\)\.SinceUt−1,k−Lt−1,k=‖𝐰t−1,k‖1≥1U\_\{t\-1,k\}\-L\_\{t\-1,k\}=\\\|\\mathbf\{w\}\_\{t\-1,k\}\\\|\_\{1\}\\geq 1,

Δ≤2​αt​\(δ\)​‖ϕ⁡\(ytj\)−ϕ⁡\(yti\)‖Vt−1Ut−1,k−Lt−1,k≤2​αt​\(δ\)​‖ϕ⁡\(ytj\)−ϕ⁡\(yti\)‖Vt−1\.\\Delta\\leq\\frac\{2\\alpha\_\{t\}\(\\delta\)\\\|\\phi\(y\_\{t\}^\{j\}\)\-\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\}\\leq 2\\alpha\_\{t\}\(\\delta\)\\\|\\phi\(y\_\{t\}^\{j\}\)\-\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{V\_\{t\}^\{\-1\}\}\.Squaring and applying \([7](https://arxiv.org/html/2611.22210#S3.E7)\) proves the final claim\. No current\-round score is required to evaluate the filter; true margins are used only in the analysis\.

###### Lemma 3\(Cumulative feature uncertainty\)\.

Forλ≥1\\lambda\\geq 1,\|Bt\|≤2\|B\_\{t\}\|\\leq 2, and‖ϕ⁡\(yti\)‖2≤1\\\|\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{2\}\\leq 1,

∑t=1Tqt≤3​log⁡detVT\+1detV1≤3​p​log⁡\(1\+2​Tλ​p\)\.\\sum\_\{t=1\}^\{T\}q\_\{t\}\\leq 3\\log\\frac\{\\det V\_\{T\+1\}\}\{\\det V\_\{1\}\}\\leq 3p\\log\\left\(1\+\\frac\{2T\}\{\\lambda p\}\\right\)\.\(11\)

Proof of Lemma[3](https://arxiv.org/html/2611.22210#ThmmyLemma3)\.LetHt=Vt−1/2\(∑i∈Btϕ\(yti\)ϕ\(yti\)⊤\)Vt−1/2⪰0H\_\{t\}=V\_\{t\}^\{\-1/2\}\(\\sum\_\{i\\in B\_\{t\}\}\\phi\(y\_\{t\}^\{i\}\)\\phi\(y\_\{t\}^\{i\}\)^\{\\top\}\)V\_\{t\}^\{\-1/2\}\\succeq 0\. Its eigenvaluesμt,1,…,μt,p\\mu\_\{t,1\},\\ldots,\\mu\_\{t,p\}are nonnegative, withtr⁡\(Ht\)=∑r=1pμt,r=qt\\operatorname\{tr\}\(H\_\{t\}\)=\\sum\_\{r=1\}^\{p\}\\mu\_\{t,r\}=q\_\{t\}\. Moreover,Vt⪰λ​IpV\_\{t\}\\succeq\\lambda I\_\{p\}gives

0≤qt≤λ−1​∑i∈Bt‖ϕ⁡\(yti\)‖22≤2\.0\\leq q\_\{t\}\\leq\\lambda^\{\-1\}\\sum\_\{i\\in B\_\{t\}\}\\\|\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{2\}^\{2\}\\leq 2\.SinceVt\+1=Vt1/2​\(Ip\+Ht\)​Vt1/2V\_\{t\+1\}=V\_\{t\}^\{1/2\}\(I\_\{p\}\+H\_\{t\}\)V\_\{t\}^\{1/2\}, we have

logdetVt\+1detVt=log∏r=1p\(1\+μt,r\)≥log\(1\+qt\)=∫0qtd​u1\+u≥qt3\.\\log\\frac\{\\det V\_\{t\+1\}\}\{\\det V\_\{t\}\}=\\log\\prod\_\{r=1\}^\{p\}\(1\+\\mu\_\{t,r\}\)\\geq\\log\(1\+q\_\{t\}\)=\\int\_\{0\}^\{q\_\{t\}\}\\frac\{du\}\{1\+u\}\\geq\\frac\{q\_\{t\}\}\{3\}\.The product inequality follows because all higher\-order terms in its expansion are nonnegative; the last inequality usesqt≤2q\_\{t\}\\leq 2\. Summing overtttelescopes the log determinants and proves the first bound\. For the second,V1=λ​IpV\_\{1\}=\\lambda I\_\{p\}and

tr⁡\(VT\+1\)=λ​p\+∑t=1T∑i∈Bt‖ϕ⁡\(yti\)‖22≤λ​p\+2​T\.\\operatorname\{tr\}\(V\_\{T\+1\}\)=\\lambda p\+\\sum\_\{t=1\}^\{T\}\\sum\_\{i\\in B\_\{t\}\}\\\|\\phi\(y\_\{t\}^\{i\}\)\\\|\_\{2\}^\{2\}\\leq\\lambda p\+2T\.Applying the arithmetic–geometric mean inequality to the positive eigenvalues ofVT\+1V\_\{T\+1\}yields

detVT\+1≤\(tr⁡\(VT\+1\)p\)p≤\(λ\+2​Tp\)p\.\\det V\_\{T\+1\}\\leq\\left\(\\frac\{\\operatorname\{tr\}\(V\_\{T\+1\}\)\}\{p\}\\right\)^\{p\}\\leq\\left\(\\lambda\+\\frac\{2T\}\{p\}\\right\)^\{p\}\.Dividing bydetV1=λp\\det V\_\{1\}=\\lambda^\{p\}and taking logarithms completes the proof\.

### B\.2Proof of Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1)

In this part, we provide a detailed proof of Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1)\. We work on the event of Lemma[1](https://arxiv.org/html/2611.22210#ThmmyLemma1)and fix arbitrary choicesat∈Bta\_\{t\}\\in B\_\{t\}\. At every query withat∉PCt−1∗a\_\{t\}\\notin P\_\{C\_\{t\-1\}\}^\{\*\}, choose a gap\-achievingj∈𝒟t​\(at\)j\\in\\mathcal\{D\}\_\{t\}\(a\_\{t\}\)satisfying Assumption[2](https://arxiv.org/html/2611.22210#ThmmyAssum2)\. Assumption[1](https://arxiv.org/html/2611.22210#ThmmyAssum1)gives

mink∈\[Kt−1\]⁡𝐰t−1,k⊤​\[μ⁡\(ytj\)−μ⁡\(ytat\)\]Ut−1,k−Lt−1,k=Δt,at≥Δ\.\\min\_\{k\\in\[K\_\{t\-1\}\]\}\\frac\{\\mathbf\{w\}\_\{t\-1,k\}^\{\\top\}\[\\mu\(y\_\{t\}^\{j\}\)\-\\mu\(y\_\{t\}^\{a\_\{t\}\}\)\]\}\{U\_\{t\-1,k\}\-L\_\{t\-1,k\}\}=\\Delta\_\{t,a\_\{t\}\}\\geq\\Delta\.Sinceata\_\{t\}survived the filter based on𝐖t−1\\mathbf\{W\}\_\{t\-1\}, Lemma[2](https://arxiv.org/html/2611.22210#ThmmyLemma2)and coverage yieldΔ2≤4​αt2​\(δ\)​κ​qt\\Delta^\{2\}\\leq 4\\alpha\_\{t\}^\{2\}\(\\delta\)\\kappa q\_\{t\}\. For any generated query prefix of lengthTT, the determinant bound in Lemma[3](https://arxiv.org/html/2611.22210#ThmmyLemma3)givesαt​\(δ\)≤α¯T​\(δ\)\\alpha\_\{t\}\(\\delta\)\\leq\\bar\{\\alpha\}\_\{T\}\(\\delta\)fort≤Tt\\leq T\. Summing over nonoptimal selections and applying the same lemma yields

Regret01⁡\(T\)≤4​α¯T2​\(δ\)​κΔ2​∑t=1Tqt≤12​α¯T2​\(δ\)​κ​pΔ2​log⁡\(1\+2​Tλ​p\)\.\\operatorname\{Regret\}^\{01\}\(T\)\\leq\\frac\{4\\bar\{\\alpha\}\_\{T\}^\{2\}\(\\delta\)\\kappa\}\{\\Delta^\{2\}\}\\sum\_\{t=1\}^\{T\}q\_\{t\}\\leq\\frac\{12\\bar\{\\alpha\}\_\{T\}^\{2\}\(\\delta\)\\kappa p\}\{\\Delta^\{2\}\}\\log\\left\(1\+\\frac\{2T\}\{\\lambda p\}\\right\)\.This proves \([8](https://arxiv.org/html/2611.22210#S3.E8)\) on the simultaneous confidence event, whose probability is at least1−δ1\-\\delta\.

The choicesata\_\{t\}were arbitrary on the same confidence event\. In particular, choose a nonoptimal member wheneverBtB\_\{t\}contains one\. This gives the same bound for∑t=1T𝕀\{Bt⊈PCt−1∗\}\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\\{B\_\{t\}\\not\\subseteq P\_\{C\_\{t\-1\}\}^\{\*\}\\\}\. Since each pair contains at most two responses,∑t=1T∑i∈Bt𝕀\{i∉PCt−1∗\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i\\in B\_\{t\}\}\\mathbb\{I\}\\\{i\\notin P\_\{C\_\{t\-1\}\}^\{\*\}\\\}is at most twice this count\.

For fixedλ,S,σz,κ,p,d,Δ,δ\\lambda,S,\\sigma\_\{z\},\\kappa,p,d,\\Delta,\\delta, we haveα¯T2​\(δ\)=O⁡\(log⁡T\)\\bar\{\\alpha\}\_\{T\}^\{2\}\(\\delta\)=O\(\\log T\)\. If queries continue indefinitely, the bound holds simultaneously for allTTon the same event, soRegret01⁡\(T\)/T→0\\operatorname\{Regret\}^\{01\}\(T\)/T\\to 0with probability at least1−δ1\-\\delta\.

### B\.3Full Statement and Proof of Proposition[1](https://arxiv.org/html/2611.22210#ThmmyProp1)

In this part, we provide the full statement of Proposition[1](https://arxiv.org/html/2611.22210#ThmmyProp1)and corresponding proof\. Under the conditions of Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1), letℛT​\(δ\)\\mathcal\{R\}\_\{T\}\(\\delta\)denote the right\-hand side of \([8](https://arxiv.org/html/2611.22210#S3.E8)\), and letGT=∑t=1T𝕀\{Bt⊆PCt−1∗\}G\_\{T\}=\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\\{B\_\{t\}\\subseteq P\_\{C\_\{t\-1\}\}^\{\*\}\\\}count queries whose two responses are Pareto\-optimal in their respective pools under the cones available before querying\. For each integerM≥1M\\geq 1, letQM=inf\{T∈ℕ:GT≥M\}Q\_\{M\}=\\inf\\\{T\\in\\mathbb\{N\}:G\_\{T\}\\geq M\\\}be the number of expensive queries needed to collectMMsuch comparisons, with the infimum taken over generated query prefixes andinf∅=\+∞\\inf\\varnothing=\+\\infty\. With probability at least1−δ1\-\\delta, simultaneously for all generated query prefixesT≥1T\\geq 1, and for all integersM≥1M\\geq 1such that at least2​M2Mqueries are generated,

GTT≥1−ℛT​\(δ\)T,M≤QM≤M\+⌈ℛ2​M​\(δ\)⌉if​ℛ2​M​\(δ\)≤M\.\\frac\{G\_\{T\}\}\{T\}\\geq 1\-\\frac\{\\mathcal\{R\}\_\{T\}\(\\delta\)\}\{T\},\\qquad M\\leq Q\_\{M\}\\leq M\+\\left\\lceil\\mathcal\{R\}\_\{2M\}\(\\delta\)\\right\\rceil\\quad\\text\{if \}\\mathcal\{R\}\_\{2M\}\(\\delta\)\\leq M\.\(12\)If queries continue indefinitely, then for fixed problem parameters andδ\\delta,QM=M\+O⁡\(log2⁡M\)Q\_\{M\}=M\+O\(\\log^\{2\}M\)asM→∞M\\to\\infty\.

Proof of Proposition[1](https://arxiv.org/html/2611.22210#ThmmyProp1)\.We work on the same simultaneous confidence event as in Theorem[1](https://arxiv.org/html/2611.22210#ThmmyThm1)\. For each round, chooseat∈Bt∖PCt−1∗a\_\{t\}\\in B\_\{t\}\\setminus P\_\{C\_\{t\-1\}\}^\{\*\}when this set is nonempty, and choose anyat∈Bta\_\{t\}\\in B\_\{t\}otherwise\. Since the theorem holds simultaneously for all choices ofata\_\{t\}, including these choices made for evaluation, we have

T−GT=∑t=1T𝕀\{Bt⊈PCt−1∗\}=∑t=1T𝕀\{at∉PCt−1∗\}≤ℛT\(δ\)\.T\-G\_\{T\}=\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\\{B\_\{t\}\\not\\subseteq P\_\{C\_\{t\-1\}\}^\{\*\}\\\}=\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\\{a\_\{t\}\\notin P\_\{C\_\{t\-1\}\}^\{\*\}\\\}\\leq\\mathcal\{R\}\_\{T\}\(\\delta\)\.Dividing byTTproves the first inequality in \([12](https://arxiv.org/html/2611.22210#A2.E12)\)\. Each round produces at most one counted comparison, soQM≥MQ\_\{M\}\\geq M\.

The boundℛT​\(δ\)\\mathcal\{R\}\_\{T\}\(\\delta\)is nondecreasing inTT\. Ifℛ2​M​\(δ\)≤M\\mathcal\{R\}\_\{2M\}\(\\delta\)\\leq M, then the integerT=M\+⌈ℛ2​M​\(δ\)⌉T=M\+\\lceil\\mathcal\{R\}\_\{2M\}\(\\delta\)\\rceilsatisfiesT≤2​MT\\leq 2M, and hence

GT≥T−ℛT​\(δ\)≥M\+⌈ℛ2​M​\(δ\)⌉−ℛ2​M​\(δ\)≥M\.G\_\{T\}\\geq T\-\\mathcal\{R\}\_\{T\}\(\\delta\)\\geq M\+\\lceil\\mathcal\{R\}\_\{2M\}\(\\delta\)\\rceil\-\\mathcal\{R\}\_\{2M\}\(\\delta\)\\geq M\.It follows thatQM≤M\+⌈ℛ2​M​\(δ\)⌉Q\_\{M\}\\leq M\+\\lceil\\mathcal\{R\}\_\{2M\}\(\\delta\)\\rceil\. For fixed problem parameters andδ\\delta,ℛ2​M​\(δ\)=O⁡\(log2⁡M\)=o⁡\(M\)\\mathcal\{R\}\_\{2M\}\(\\delta\)=O\(\\log^\{2\}M\)=o\(M\), so, if queries continue indefinitely, the condition holds for all sufficiently largeMM, givingQM=M\+O⁡\(log2⁡M\)Q\_\{M\}=M\+O\(\\log^\{2\}M\)on the same confidence event\. No additional union bound is needed\.

The countGTG\_\{T\}evaluates each comparison using its own round’s candidate pool and the cone available before querying; previously collected comparisons are not reevaluated under later cones\. The proposition quantifies query allocation to nondominated pairs, without assuming that all such comparisons have equal information value or establishing a label complexity for a target policy reward\.

相似文章

数据稀缺下大语言模型的协同个性化偏好对齐

Hugging Face Daily Papers

论文提出了一种近似帕累托最优(APO)方法,用于协同学习轻量级大语言模型对齐器的初始化参数,以支持在数据稀缺条件下的少样本个性化。该方法通过将更新方向兼容的用户分组,并协调相互竞争的目标以趋近帕累托最优点,同时给出了可证明的次优性界限。

联合建模大语言模型中的偏好与不完美决策

arXiv cs.AI

论文提出偏好—行为模型(PBM),通过逆博弈问题联合推断大语言模型(LLM)的偏好与不完美决策行为参数,并在公共物品博弈等三种社会困境中进行评估。研究发现,传统行为经济学偏好测量工具无法一致预测 LLM 在战略环境中的行为,而跨博弈拟合的模型则具有预测价值。

大型语言模型应学习个性化而非聚合的人类偏好

arXiv cs.LG

这篇立场论文主张,大型语言模型应从个性化而非聚合的人类偏好中学习,指出社会选择理论中的理论局限性以及人口多样性带来的实际问题。它提出了有边界的个性化框架,在尊重个体自主性的同时维护普遍的安全约束。