Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

# Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
Source: [https://arxiv.org/html/2605.20834](https://arxiv.org/html/2605.20834)
###### Abstract

Direct Preference Optimization \(DPO\) has emerged as a popular alternative to Reinforcement Learning from Human Feedback \(RLHF\), offering theoretical equivalence with simpler implementation\. We prove this equivalence is*conditional*rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF\-optimal policy must prefer human\-preferred responses\. When this assumption fails, DPO optimizes*relative advantage*over the reference policy rather than*absolute alignment*with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses\. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases\. To address this, we introduce Constrained Preference Optimization \(CPO\), augmenting RLHF with constraints for provable alignment\. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets\. Our theoretical analysis establishes when DPOs’ guarantees hold and provides solutions preserving simplicity with provable alignment\. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state\-of\-the\-art performance\. Code is available at:[https://github\.com/visitworld123/CPO](https://github.com/visitworld123/CPO)\.

Machine Learning, ICML

## 1Introduction

Aligning large language models \(LLMs\) with human preferences has emerged as a central challenge\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.20834#bib.bib16); Baiet al\.,[2022](https://arxiv.org/html/2605.20834#bib.bib4)\)\. A prominent approach is Reinforcement Learning from Human Feedback \(RLHF\)\(Christianoet al\.,[2017](https://arxiv.org/html/2605.20834#bib.bib9); Stiennonet al\.,[2020](https://arxiv.org/html/2605.20834#bib.bib20)\), which optimizes the policy model to generate human\-preferred responses by leveraging reward model feedback\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.20834#bib.bib16); Schulmanet al\.,[2017](https://arxiv.org/html/2605.20834#bib.bib19)\)\. However, its computationally expensive and unstable nature\(Casperet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib25)\)has motivated the development of Direct Preference Optimization \(DPO\) as an elegant alternative, offering theoretical equivalence to RLHF with significantly simpler implementation\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib18)\)\. DPO is derived from a mathematical reparameterization\(Tunstallet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib21); Ivisonet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib13); Dubeyet al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib24)\): under the Bradley\-Terry \(BT\) model\(Bradley and Terry,[1952](https://arxiv.org/html/2605.20834#bib.bib5)\), the optimal RLHF policy can be expressed analytically in terms of the reward function, enabling direct policy optimization without explicit reward modeling or RL training, which has led to its widespread adoption\.

Recent theoretical analyses have revealed critical distinctions between DPO and RLHF\.Fischet al\.\([2024](https://arxiv.org/html/2605.20834#bib.bib27)\)show that DPO’s implicit rewards overfit and trend toward infinite magnitude, often yielding degenerate policies where even preferred responses receive near\-zero probability\.Linet al\.\([2024](https://arxiv.org/html/2605.20834#bib.bib26)\)demonstrate that DPO’s implicit reward model generalizes significantly worse than explicit reward models under distribution shift\.Im and Li \([2024](https://arxiv.org/html/2605.20834#bib.bib28)\)examine how performance gaps emerge when reward and policy models have different representational capacities\.Shiet al\.\([2025](https://arxiv.org/html/2605.20834#bib.bib29)\)reveal that DPO prioritizes statistically distinguishable behaviors over value\-aligned ones, potentially causing misalignment despite decreasing loss\. These findings raise a fundamental open problem:

Under what conditions can DPO be derived through RLHF?

In this work, we revisit the derivation of DPO and identify a critical but previously overlooked assumption:*the RLHF\-optimal policy must prefer human\-preferred responses over dispreferred ones*\. Specifically, DPO’s derivation relies on substituting the RLHF\-optimal policyπ∗\\pi^\{\*\}into the BT model to eliminate the reward function\. This substitution, however, is only valid whenπ∗\\pi^\{\*\}*respects*the preference structure encoded in the BT model that is, when it assigns higher probability to the preferred response\. We show that this critical assumption is*not*guaranteed by the RLHF framework \(Sec\.LABEL:sec:assumption\)\. This violation arises because RLHF balances reward maximization against KL divergence from the reference policy\. When the reference policy is sufficiently misaligned, the KL penalty dominates, causingπ∗\\pi^\{\*\}to inherit incorrect preferences fromπref\\pi\_\{\\text\{ref\}\}, thereby violating the implicit assumption underlying DPO\.

We prove that when this implicit assumption is violated, DPO optimizes a fundamentally different objective than RLHF, creating a risk of misalignment with human preferences\. Specifically, DPO optimizes for*relative advantage*over the reference policy rather than*absolute alignment*with human preferences, causing a fundamental shift in the optimization objective\. This violation leads to pathological convergence: policies can decrease DPO loss while systematically preferring dispreferred responses\. We characterize an*undesirable solution space*\(DefinitionLABEL:def:undesirable\) where policies simultaneously satisfy DPO’s optimization objective yet contradict human preferences\. This reveals that DPO inherits RLHF’s algebraic structure through reward reparameterization but does not inherit its alignment guarantees\. The equivalence is thus conditional on reference policy quality\.

To address this fundamental limitation, we introduce*Constrained Preference Optimization \(CPO\)*, which augments the RLHF objective with explicit constraints\. The constraint term aligns the optimal solution of RLHF with the requirements of BT theory, thereby guaranteeing alignment with human preferences\. We further provide a geometric interpretation of DPO and CPO through the lens of soft margin ranking loss\(Burgeset al\.,[2005](https://arxiv.org/html/2605.20834#bib.bib6); Schroffet al\.,[2015](https://arxiv.org/html/2605.20834#bib.bib31)\)\. DPO approximates margin ranking loss with a target margin that can be negative, providing an intuitive geometric explanation for why DPO can converge to preference\-violating policies\. CPO corrects this by ensuring non\-negative effective margins through its constraint terms\. This perspective provides geometric intuition for understanding when and why DPO fails and how CPO addresses these failures\. To further eliminate the need for explicit reward modeling, we develop a conservative variant, E\-CPOC, which achieves formal equivalence to explicitly constrained RLHF under standard statistical assumptions\. Central to the equivalence analysis is a*Loss\-to\-Delta bridge*\(PropositionLABEL:prop:loss\_to\_delta\) that converts the observable training loss gap into a guarantee on policy\-level proximity inδ\\delta\-space, with a bound whose constant is*independent*of the number of preference pairsNN—making the equivalence guarantee*verifiable*from training diagnostics alone, without assuming global optimality\. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state\-of\-the\-art performance\.

We summarize our main contributions as follows:

- •We prove that DPO and RLHF are conditionally equivalent \(Sec\.LABEL:sec:assumall\), depending on an implicit assumption: the RLHF\-optimal policy must prefer human\-preferred responses over dispreferred ones\. Whether this assumption holds depends on the quality of the reference policy\. This reveals that DPO does not inherit RLHF’s alignment guarantees, making the equivalence conditional on reference policy quality\.
- •We establish that when the assumption is violated, DPO and RLHF optimize fundamentally different objectives: RLHF optimizes for absolute alignment with human preferences, while DPO optimizes for relative advantage over the reference policy\. Consequently, DPO’s gradient descent can converge to a pathological space where policies simultaneously satisfy DPO’s optimization objective yet violate human preferences \(Sec\.LABEL:sec:violation\)\.
- •We propose Constrained Preference Optimization \(CPO\), augmenting RLHF with explicit constraints to enforce preference alignment with provable absolute advantage guarantees \(Sec\.[3\.2](https://arxiv.org/html/2605.20834#S3.SS2)\)\. We further propose Conservative Explicitly Constrained Preference Optimization \(E\-CPOC\), which explicitly enforces preference alignment without requiring a reward model \(Sec\.[3\.5](https://arxiv.org/html/2605.20834#S3.SS5)\)\. E\-CPOC achieves formal equivalence to explicitly constrained RLHF under standard statistical learning assumptions \(TheoremLABEL:thm:ecpoc\_equivalencein AppendixLABEL:app:aee\), requiring only the Bradley\-Terry model, approximate realizability, finite\-sample data, and a mildℓ2\\ell^\{2\}\-δ\\delta\-proximity condition \(Assumptions[3\.1](https://arxiv.org/html/2605.20834#S3.Thmtheorem1)–[3\.4](https://arxiv.org/html/2605.20834#S3.Thmtheorem4)in Sec\.[3\.1](https://arxiv.org/html/2605.20834#S3.SS1)\)\. Theℓ2\\ell^\{2\}\-δ\\delta\-proximity condition uses the natural mean\-square norm that the loss function directly controls and can be*derived*from loss suboptimality via a verifiable bridge with an*NN\-independent*bound \(PropositionLABEL:prop:loss\_to\_delta, CorollaryLABEL:cor:verifiable\_equiv\) under a mild non\-degeneracy condition on preference probabilities, without assuming global optimality directly\.
- •Comprehensive experiments on standard benchmarks demonstrate the efficacy of our method \(Sec\.[5](https://arxiv.org/html/2605.20834#S5)\)\. We also provide a geometric understanding by proving that DPO is equivalent to soft margin ranking loss with a potentially negative margin\. Our method corrects this by ensuring non\-negative effective margins \(Sec\.[4](https://arxiv.org/html/2605.20834#S4)\), connecting preference learning to the learning\-to\-rank literature with intuitive geometric interpretations\.

## 2Preliminaries

### 2\.1Notation

Let𝒳\\mathcal\{X\}denote the space of prompts and𝒴\\mathcal\{Y\}denote the space of responses\. A policyπ:𝒳×𝒴→\[0,1\]\\pi:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\[0,1\]is a conditional probability distribution over responses given prompts\. We useπref\\pi\_\{\\text\{ref\}\}to denote a fixed reference policy \(typically a supervised fine\-tuned model\) andπθ\\pi\_\{\\theta\}to denote a learnable policy parameterized byθ\\theta\.

For a given promptxxand response pair\(yw,yl\)\(y\_\{w\},y\_\{l\}\)whereywy\_\{w\}is preferred overyly\_\{l\}, the log\-probability ratio is defined as:

δπ​\(x,yw,yl\):=log⁡π​\(yw\|x\)−log⁡π​\(yl\|x\)\.\\delta\_\{\\pi\}\(x,y\_\{w\},y\_\{l\}\):=\\log\\pi\(y\_\{w\}\|x\)\-\\log\\pi\(y\_\{l\}\|x\)\.\(1\)When the context is clear, we abbreviate this asδπ\\delta\_\{\\pi\}\. This quantity measures the policy’s preference strength forywy\_\{w\}overyly\_\{l\}in log\-space\.

### 2\.2RLHF Framework

###### Definition 2\.1\(RLHF Objective\)\.

Given a reward functionr:𝒳×𝒴→ℝr:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}, a reference policyπref\\pi\_\{\\text\{ref\}\}, and a temperature parameterβ\>0\\beta\>0, the RLHF optimization objective is:

maxπ𝔼x∼𝒟,y∼π\(⋅\|x\)\[r\(x,y\)\]−β⋅KL\(π\(⋅\|x\)∥πref\(⋅\|x\)\),\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\cdot\\operatorname\{KL\}\(\\pi\(\\cdot\|x\)\\\|\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\),\(2\)where𝒟\\mathcal\{D\}is the prompt distribution andKL\\operatorname\{KL\}denotes the Kullback\-Leibler divergence\.

The KL regularization term prevents the learned policy from deviating too far fromπref\\pi\_\{\\text\{ref\}\}, ensuring stable training and preventing reward over\-optimization\(Gaoet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib11)\)\.

The optimal solution to the RLHF objective has the closed form\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib18)\):

π∗​\(y\|x\)=1Z​\(x\)​πref​\(y\|x\)​exp⁡\(r​\(x,y\)β\),\\pi^\{\*\}\(y\|x\)=\\frac\{1\}\{Z\(x\)\}\\pi\_\{\\text\{ref\}\}\(y\|x\)\\exp\\left\(\\frac\{r\(x,y\)\}\{\\beta\}\\right\),\(3\)whereZ​\(x\)=∑y′πref​\(y′\|x\)​exp⁡\(r​\(x,y′\)/β\)Z\(x\)=\\sum\_\{y^\{\\prime\}\}\\pi\_\{\\text\{ref\}\}\(y^\{\\prime\}\|x\)\\exp\(r\(x,y^\{\\prime\}\)/\\beta\)is the partition function\. Then, for any response pair\(yw,yl\)\(y\_\{w\},y\_\{l\}\), the reward difference can be expressed as:

r​\(x,yw\)−r​\(x,yl\)=β​\[log⁡π∗​\(yw\|x\)πref​\(yw\|x\)−log⁡π∗​\(yl\|x\)πref​\(yl\|x\)\]\.r\(x,y\_\{w\}\)\-r\(x,y\_\{l\}\)=\\beta\\left\[\\log\\frac\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\-\\log\\frac\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\]\.\(4\)
This reward difference can be presented using the log\-probability ratio Eq\. \([1](https://arxiv.org/html/2605.20834#S2.E1)\):

δπ∗=δπref\+r​\(x,yw\)−r​\(x,yl\)β\.\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{r\(x,y\_\{w\}\)\-r\(x,y\_\{l\}\)\}\{\\beta\}\.\(5\)

### 2\.3Bradley\-Terry Preference Model

###### Definition 2\.2\(Bradley\-Terry Model\(Bradley and Terry,[1952](https://arxiv.org/html/2605.20834#bib.bib5)\)\)\.

Human preference forywy\_\{w\}overyly\_\{l\}given promptxxis modeled as:

p∗​\(yw≻yl\|x\)=σ​\(r∗​\(yw\)−r∗​\(yl\)\)p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)=\\sigma\(r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\)\(6\)whereσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function andr∗​\(⋅\)r^\{\*\}\(\\cdot\)is the latent true reward function representing human preferences\.

Ifyw≻yly\_\{w\}\\succ y\_\{l\}\(i\.e\.,p∗​\(yw≻yl\|x\)\>0\.5p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)\>0\.5\), then necessarilyr∗​\(yw\)−r∗​\(yl\)\>0r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\>0\.

### 2\.4Direct Preference Optimization

Substituting the reward reparameterization Eq\. \([4](https://arxiv.org/html/2605.20834#S2.E4)\) into the Bradley\-Terry model Eq\. \([6](https://arxiv.org/html/2605.20834#S2.E6)\):

p∗​\(yw≻yl\)=σ​\(r∗​\(x,yw\)−r∗​\(x,yl\)\)=σ​\(β​\(δπ∗−δπref\)\)\.p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\)=\\sigma\(r^\{\*\}\(x,y\_\{w\}\)\-r^\{\*\}\(x,y\_\{l\}\)\)=\\sigma\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\.\(7\)
DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib18)\)approximatesπ∗\\pi^\{\*\}with a parameterized policyπθ\\pi\_\{\\theta\}and maximizes the log\-likelihood:

ℒDPO​\(πθ\)=−𝔼\(x,yw,yl\)∼𝒟​\[log⁡σ​\(β​\(δπθ−δπref\)\)\]\.\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\left\[\\log\\sigma\(\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\right\]\.\(8\)

## 3Constrained Preference Optimization

To relax the identified implicit assumption, we propose*Constrained Preference Optimization \(CPO\)*, which enhances the vanilla RLHF to a constrained RLHF\. The optimal solution of the constrained RLHF can be safely integrated into the BT model, as the proposed constraint explicitly encourages or ensures the preference alignment\. Before presenting the framework, we state the assumptions underlying our theoretical results\.

### 3\.1Assumptions

A distinguishing feature of our analysis is that*all assumptions are either standard or provably mild*\. We require only the Bradley\-Terry preference model, standard statistical learning conditions, and a natural optimization quality measure that admits a verifiable sufficient condition from training diagnostics\. No global optimality, exact realizability, or pointwise \(ℓ∞\\ell^\{\\infty\}\) optimization assumptions are needed\.

###### Assumption 3\.1\(Bradley\-Terry Model\)\.

The true preference distribution follows the Bradley\-Terry model:p∗​\(yw≻yl\|x\)=σ​\(r∗​\(x,yw\)−r∗​\(x,yl\)\)p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)=\\sigma\(r^\{\*\}\(x,y\_\{w\}\)\-r^\{\*\}\(x,y\_\{l\}\)\)\.

This is the standard preference model adopted throughout the RLHF literature\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib18); Christianoet al\.,[2017](https://arxiv.org/html/2605.20834#bib.bib9)\), positing a latent reward functionr∗r^\{\*\}that generates human preferences via a logistic link\.

###### Assumption 3\.2\(ϵapprox\\epsilon\_\{\\mathrm\{approx\}\}\-Approximate Realizability\)\.

The population\-level constrained MLEπMLE∗:=arg⁡maxθ∈Θ⁡𝔼p∗​\[log⁡pπθ\]\\pi^\{\*\}\_\{\\mathrm\{MLE\}\}:=\\arg\\max\_\{\\theta\\in\\Theta\}\\mathbb\{E\}\_\{p^\{\*\}\}\[\\log p\_\{\\pi\_\{\\theta\}\}\]satisfies:

max\(x,yw,yl\)∈𝒟⁡\|δπMLE∗​\(x,yw,yl\)−δtarget​\(x,yw,yl\)\|≤ϵapprox,\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\left\|\\delta\_\{\\pi^\{\*\}\_\{\\mathrm\{MLE\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\delta\_\{\\mathrm\{target\}\}\(x,y\_\{w\},y\_\{l\}\)\\right\|\\leq\\epsilon\_\{\\mathrm\{approx\}\},whereδtarget\\delta\_\{\\mathrm\{target\}\}denotes the target log\-probability ratio achieving exact equivalence in the population limit, andϵapprox≥0\\epsilon\_\{\\mathrm\{approx\}\}\\geq 0quantifies the expressiveness gap of the policy class\{πθ\}\\\{\\pi\_\{\\theta\}\\\}\.

Whenϵapprox=0\\epsilon\_\{\\mathrm\{approx\}\}=0, the policy class is exactly realizable\. For overparameterized neural networks, smallϵapprox\\epsilon\_\{\\mathrm\{approx\}\}is expected; the properness of the cross\-entropy scoring rule ensures the MLE is at least as good as any fixedθ\\thetain aggregate loss\.

###### Assumption 3\.3\(Finite\-Sample Data\)\.

The dataset𝒟\\mathcal\{D\}containsNNi\.i\.d\. samples from the true preference distribution, with statistical estimation errorϵstat\(N\):=sup\(x,yw,yl\)\|p^N\(yw≻yl\|x\)−p∗\(yw≻yl\|x\)\|\\epsilon\_\{\\mathrm\{stat\}\}\(N\):=\\sup\_\{\(x,y\_\{w\},y\_\{l\}\)\}\|\\hat\{p\}\_\{N\}\(y\_\{w\}\\succ y\_\{l\}\|x\)\-p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)\|\. By Hoeffding’s inequality,ϵstat​\(N\)=O​\(1/N\)\\epsilon\_\{\\mathrm\{stat\}\}\(N\)=O\(1/\\sqrt\{N\}\)\.

This is the standard finite\-sample condition in statistical learning\. In the population limit \(N→∞N\\to\\infty,ϵstat=0\\epsilon\_\{\\mathrm\{stat\}\}=0\), it reduces to exact distributional convergence\.

###### Assumption 3\.4\(ℓ2\\ell^\{2\}\-δ\\delta\-Proximity\)\.

The returned policyπ^\\hat\{\\pi\}satisfies:

1N​∑i=1N\(δπ^,i−δπMLE∗,i\)2≤ϵopt,22,\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(\\delta\_\{\\hat\{\\pi\},i\}\-\\delta\_\{\\pi^\{\*\}\_\{\\mathrm\{MLE\}\},i\}\\right\)^\{2\}\\leq\\epsilon\_\{\\mathrm\{opt,2\}\}^\{2\},whereπMLE∗\\pi^\{\*\}\_\{\\mathrm\{MLE\}\}is the class\-optimal MLE policy \(Assumption[3\.2](https://arxiv.org/html/2605.20834#S3.Thmtheorem2)\) andϵopt,2≥0\\epsilon\_\{\\mathrm\{opt,2\}\}\\geq 0quantifies the mean\-square optimization error inδ\\delta\-space\.

This is the*core optimization requirement*for the equivalence result\. It uses the naturalℓ2\\ell^\{2\}\(mean\-square\) norm that the loss function directly controls, and is*strictly weaker*than the pointwise \(ℓ∞\\ell^\{\\infty\}\) conditionmaxi⁡\|δπ^,i−δi∗\|≤ϵopt\\max\_\{i\}\|\\delta\_\{\\hat\{\\pi\},i\}\-\\delta^\{\*\}\_\{i\}\|\\leq\\epsilon\_\{\\mathrm\{opt\}\}:ℓ2\\ell^\{2\}\-proximity permits larger deviations on a few difficult data points as long as the average error remains controlled\. Crucially, it admits a*verifiable sufficient condition*: under Assumption[3\.5](https://arxiv.org/html/2605.20834#S3.Thmtheorem5), small training loss gap impliesℓ2\\ell^\{2\}\-δ\\delta\-proximity with anNN\-independent bound \(PropositionLABEL:prop:loss\_to\_delta\)\.

###### Assumption 3\.5\(Non\-degenerate Preferences\)\.

Define the*logistic curvature*at the class\-optimal policy:

κ0:=min1≤i≤N⁡σ​\(gi​\(δi∗\)\)​\(1−σ​\(gi​\(δi∗\)\)\),\\kappa\_\{0\}:=\\min\_\{1\\leq i\\leq N\}\\sigma\(g\_\{i\}\(\\delta^\{\*\}\_\{i\}\)\)\(1\-\\sigma\(g\_\{i\}\(\\delta^\{\*\}\_\{i\}\)\)\),\(9\)wheregi​\(δi\):=β​\(δi−δref,i\)−Ψcons,ig\_\{i\}\(\\delta\_\{i\}\):=\\beta\(\\delta\_\{i\}\-\\delta\_\{\\mathrm\{ref\},i\}\)\-\\Psi\_\{\\mathrm\{cons\},i\}is the margin function of the preference optimization loss, withΨcons,i\\Psi\_\{\\mathrm\{cons\},i\}denoting the adaptive constraint margin \(formally defined in Sec\.[3\.5](https://arxiv.org/html/2605.20834#S3.SS5)\), andδ∗=δπMLE∗\\delta^\{\*\}=\\delta\_\{\\pi^\{\*\}\_\{\\mathrm\{MLE\}\}\}denotes the class\-optimalδ\\delta\-values\. We assumeκ0\>0\\kappa\_\{0\}\>0\.

This requires that no preference pair has deterministic \(probability0or11\) preference under the class\-optimal policy—a mild regularity condition automatically satisfied for any smooth parameterization with bounded parameters \(AssumptionLABEL:assump:smooth\_boundedin AppendixLABEL:app:converge\)\. Importantly, this assumption is*not*required for the core equivalence result \(TheoremLABEL:thm:ecpoc\_equivalence\); it is needed only for the Loss\-to\-Delta bridge \(PropositionLABEL:prop:loss\_to\_delta\) that converts the verifiable loss gap into theℓ2\\ell^\{2\}\-δ\\delta\-proximity guarantee\.

###### Condition 3\.6\(Connected Comparison Graph\)\.

For each promptx∈𝒳x\\in\\mathcal\{X\}, the preference pairs in𝒟\\mathcal\{D\}involvingxxform a connected comparison graph with finite diameterdxd\_\{x\}\. Letd:=maxx∈𝒳⁡dxd:=\\max\_\{x\\in\\mathcal\{X\}\}d\_\{x\}\.

This structural condition is required*only*for extending pairwiseδ\\delta\-equivalence to full policy equivalence—it is not needed for the core pairwise results\. In practice, preference datasets with reasonable response coverage naturally satisfy this condition with moderate diameter\.

Table 1:Assumption dependency map for the E\-CPOC equivalence \(TheoremLABEL:thm:ecpoc\_equivalence\)\.Core: required for the pairwiseδ\\delta\-equivalence bound\.Bridge: provides a verifiable sufficient condition for the coreℓ2\\ell^\{2\}\-δ\\delta\-proximity\.Ext: required only for extension to full policy equivalence\.
### 3\.2Constrained RLHF Framework

The RLHF\-optimal policy may satisfyδπ∗<0\\delta\_\{\\pi^\{\*\}\}<0\. Thus, we augment the RLHF objective with an explicit constraint term that directly encouragesδπ\>0\\delta\_\{\\pi\}\>0for preferred responses\.

###### Definition 3\.7\(Constrained RLHF\)\.

Given a reward functionr:𝒳×𝒴→ℝr:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}, a reference policyπref\\pi\_\{\\text\{ref\}\}, a temperature parameterβ\>0\\beta\>0, and the strength of preference alignmentγ\\gamma, the constrained RLHF optimization objective is:

maxπ⁡𝔼x∼𝒟​𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]\\displaystyle\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]−β​KL⁡\(π∥πref\)\\displaystyle\-\\beta\\operatorname\{KL\}\(\\pi\\\|\\pi\_\{\\text\{ref\}\}\)\(10\)\+γ​𝔼\(x,yw,yl\)∼𝒟​\[δπ\],\\displaystyle\+\\gamma\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[\\delta\_\{\\pi\}\],whereδπ\\delta\_\{\\pi\}is the log\-probability ratio\.

The constraint term directly*encourages*the policy to preferywy\_\{w\}overyly\_\{l\}in log\-probability space\. Whenγ=0\\gamma=0, it recovers vanilla RLHF\. The parameterγ\\gammaprovides explicit control over the strength of preference alignment\.

A closed\-form solution for the optimal policy of Constrained RLHF is difficult to derive; we therefore characterize it via the first\-order optimality condition, with the proof given in Appendix[D\.4](https://arxiv.org/html/2605.20834#A4.SS4)\.

###### Theorem 3\.8\(Optimal Policy for Constrained RLHF\)\.

The optimal policyπ∗\\pi^\{\*\}for the Constrained RLHF objective satisfies the first\-order optimality condition:

β​log⁡π∗​\(y\|x\)πref​\(y\|x\)=r​\(x,y\)\+c​\(x,y\)π∗​\(y\|x\)−β−λ​\(x\),\\beta\\log\\frac\{\\pi^\{\*\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}=r\(x,y\)\+\\frac\{c\(x,y\)\}\{\\pi^\{\*\}\(y\|x\)\}\-\\beta\-\\lambda\(x\),\(11\)wherec​\(x,y\)=γ​∑\(yw,yl\)∈𝒫​\(x\)p​\(yw,yl\|x\)​\(𝕀​\(y=yw\)−𝕀​\(y=yl\)\)c\(x,y\)=\\gamma\\sum\_\{\(y\_\{w\},y\_\{l\}\)\\in\\mathcal\{P\}\(x\)\}p\(y\_\{w\},y\_\{l\}\|x\)\(\\mathbb\{I\}\(y=y\_\{w\}\)\-\\mathbb\{I\}\(y=y\_\{l\}\)\)and𝒫​\(x\)\\mathcal\{P\}\(x\)denotes preference pairs forxx\.

For a preference pair\(yw,yl\)\(y\_\{w\},y\_\{l\}\), this implies:

β​\(δπ∗−δπref\)=r​\(yw\)−r​\(yl\)\+c​\(x,yw\)π∗​\(yw\|x\)−c​\(x,yl\)π∗​\(yl\|x\)\.\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=r\(y\_\{w\}\)\-r\(y\_\{l\}\)\+\\frac\{c\(x,y\_\{w\}\)\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\-\\frac\{c\(x,y\_\{l\}\)\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\.\(12\)

The theoretical results derived under the notational simplification \(Appendix[D\.4](https://arxiv.org/html/2605.20834#A4.SS4)\) extend naturally to the general case where responses appear in multiple preference pairs, as shown in Appendix[F](https://arxiv.org/html/2605.20834#A6)\(Proposition[F\.1](https://arxiv.org/html/2605.20834#A6.Thmtheorem1)\)\.

### 3\.3Preference Optimization with Constrained RLHF

We now derive a constrained preference optimization analogous to DPO but based on constrained RLHF\. For a single preference pair, Theorem[3\.8](https://arxiv.org/html/2605.20834#S3.Thmtheorem8)simplifies to:

r​\(yw\)−r​\(yl\)=β​\(δπ∗−δπref\)−γ​\(1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)\)\.r\(y\_\{w\}\)\-r\(y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\gamma\\left\(\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\right\)\.\(13\)This implies that:

p∗​\(yw≻yl\)=σ​\(β​\(δπ∗−δπref\)−γ~∗​\(x,yw,yl\)\),p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\)=\\sigma\\left\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\\right\),\(14\)whereγ~∗​\(x,yw,yl\)\\tilde\{\\gamma\}^\{\*\}\(x,y\_\{w\},y\_\{l\}\)is:

γ~∗​\(x,yw,yl\)=γ​\(1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)\)\.\\tilde\{\\gamma\}^\{\*\}\(x,y\_\{w\},y\_\{l\}\)=\\gamma\\left\(\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\right\)\.\(15\)The termγ~∗​\(x,yw,yl\)=γ​\(1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)\)\\tilde\{\\gamma\}^\{\*\}\(x,y\_\{w\},y\_\{l\}\)=\\gamma\\left\(\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\right\)acts as an*adaptive margin*that depends on the optimal policy probabilities\. When the optimal policy assigns low probability to both responses \(hard pairs\), the margin is large; when it assigns high probability \(easy pairs\), the margin is small\.

From Eq\. \([13](https://arxiv.org/html/2605.20834#S3.E13)\), the optimal policy for Constrained RLHF satisfies:

r​\(yw\)−r​\(yl\)=β​\(δπ∗−δπref\)−γ~∗​\(x,yw,yl\)r\(y\_\{w\}\)\-r\(y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\(16\)
DPO approximatesπ∗\\pi^\{\*\}withπθ\\pi\_\{\\theta\}, while usingπθ\\pi\_\{\\theta\}in the margin term will create a*non\-stationary optimization objective*, as the loss itself depends on the parameters being optimized\. To obtain a stationary objective suitable for gradient descent, we approximate the optimal policy probabilities in the margin term with the reference policy probabilities:

1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)≈1πref​\(yw\|x\)\+1πref​\(yl\|x\)\.\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\approx\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\.\(17\)
This yields the constrained preference optimization loss:

ℒCPO​\(πθ\)=−𝔼𝒟​\[log⁡σ​\(β​\(δπθ−δπref\)−γ~ref​\(x,yw,yl\)\)\],\\mathcal\{L\}\_\{\\text\{CPO\}\}\(\\pi\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\log\\sigma\\left\(\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\\right\)\\right\],\(18\)where the*reference\-based adaptive margin*is:

γ~ref​\(x,yw,yl\)=γ​\(1πref​\(yw\|x\)\+1πref​\(yl\|x\)\)\.\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)=\\gamma\\left\(\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)\.\(19\)
PropositionLABEL:prop:stationary\_cposhows that the approximation error\|γ~∗−γ~ref\|\|\\tilde\{\\gamma\}^\{\*\}\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\|isO​\(γ​R~max/β/q02\)O\(\\gamma\\sqrt\{\\tilde\{R\}\_\{\\max\}/\\beta\}/q\_\{0\}^\{2\}\)under mild regularity conditions \(AssumptionLABEL:assump:cpo\_regularityin AppendixLABEL:app:nottheta\), whereq0=pmin​e−2​Rmax/βq\_\{0\}=p\_\{\\min\}\\,e^\{\-2R\_\{\\max\}/\\beta\}andR~max=Rmax\+γ/q0\\tilde\{R\}\_\{\\max\}=R\_\{\\max\}\+\\gamma/q\_\{0\}is an effective reward bound that accounts for the constraint contribution\. The bound vanishes asβ→∞\\beta\\to\\inftyand reduces to the unconstrained case \(R~max=Rmax\\tilde\{R\}\_\{\\max\}=R\_\{\\max\}\) whenγ=0\\gamma=0\. Crucially, usingγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}instead ofγ~θ\\tilde\{\\gamma\}\_\{\\theta\}makes the loss function*stationary*with respect toθ\\theta, enabling standard gradient descent with convergence guarantees\. Further discussion is provided in AppendixLABEL:app:nottheta\.

The CPO loss with reference\-based margin \(Eq\. \([18](https://arxiv.org/html/2605.20834#S3.E18)\)\) defines a stationary optimization problem\. Under standard smoothness and boundedness assumptions \(AssumptionLABEL:assump:smooth\_boundedin AppendixLABEL:app:converge\), gradient descent onℒCPO​\(πθ\)\\mathcal\{L\}\_\{\\text\{CPO\}\}\(\\pi\_\{\\theta\}\)converges to a stationary point\.

Whenγ=0\\gamma=0, CPO reduces exactly to standard DPO, making it a strict generalization\. CPO can be viewed as a principled way to add a margin to preference learning, similar to margin\-based ranking losses in information retrieval, but derived from RLHF\. The margin term is related to but distinct from the IPO\(Azaret al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib1)\)regularization, which modifies the loss function rather than the underlying RLHF objective\. CPO’s margin emerges naturally from augmenting the RLHF objective\.

### 3\.4Theoretical Guarantees

Thanks to the introduced constraint term, CPO can guarantee the absolute advantage, thereby ensuring the implicit assumption is satisfied\.

###### Theorem 3\.9\(Absolute Advantage Guarantee\)\.

For a preference dataset𝒟\\mathcal\{D\}, choosingγ≥γ∗\\gamma\\geq\\gamma^\{\*\}guarantees the absolute advantage of CPO’s optimal policyδπCPO∗\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\>0for all preference pairs in𝒟\\mathcal\{D\}, withγ∗\\gamma^\{\*\}defined as:

max\(x,yw,yl\)∈𝒟⁡β⋅max⁡\{0,−δπref​\(x,yw,yl\)−r∗​\(yw\)−r∗​\(yl\)β\}1πref​\(yw\|x\)\+1πref​\(yl\|x\)\.\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\beta\\cdot\\frac\{\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\}\{\\beta\}\\right\\\}\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\}\.\(20\)

Besides the absolute advantage guarantee, Theorem[3\.10](https://arxiv.org/html/2605.20834#S3.Thmtheorem10)shows that CPO avoids pathological convergence to the undesirable solution space with proof in Appendix[D\.6](https://arxiv.org/html/2605.20834#A4.SS6)\.

###### Theorem 3\.10\(CPO Avoids Pathological Convergence\)\.

Whenγ≥γ∗\\gamma\\geq\\gamma^\{\*\}, CPO does not converge to the undesirable solution space𝒰\\mathcal\{U\}defined in DefinitionLABEL:def:undesirable\.

Algorithm 1Constrained Preference Optimization \(CPO\)0:Preference dataset

𝒟=\{\(x\(i\),yw\(i\),yl\(i\)\)\}i=1N\\mathcal\{D\}=\\\{\(x^\{\(i\)\},y\_\{w\}^\{\(i\)\},y\_\{l\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{N\}
0:Reference policy

πref\\pi\_\{\\text\{ref\}\}
0:Hyperparameters:

β\>0\\beta\>0\(temperature\),

γ\>0\\gamma\>0\(margin weight\)

0:Learning rate

η\\eta
1:Initialize policy parameters

θ\\theta\(e\.g\., from

πref\\pi\_\{\\text\{ref\}\}\)

2:Precompute:For each

\(x,yw,yl\)∈𝒟\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}:

3:

δref\(i\)←log⁡πref​\(yw\|x\)−log⁡πref​\(yl\|x\)\\delta\_\{\\text\{ref\}\}^\{\(i\)\}\\leftarrow\\log\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\-\\log\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)
4:

γ~ref\(i\)←γ⋅\(1/πref​\(yw\|x\)\+1/πref​\(yl\|x\)\)\\tilde\{\\gamma\}\_\{\\text\{ref\}\}^\{\(i\)\}\\leftarrow\\gamma\\cdot\(1/\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+1/\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\)
5:foreach training iterationdo

6:Sample batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}
7:foreach

\(x,yw,yl\)∈ℬ\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{B\}do

8:Compute log\-ratios:

9:

δθ←log⁡πθ​\(yw\|x\)−log⁡πθ​\(yl\|x\)\\delta\_\{\\theta\}\\leftarrow\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)
10:Compute CPO loss:

11:

logits←β​\(δθ−δref\(i\)\)−γ~ref\(i\)\\text\{logits\}\\leftarrow\\beta\(\\delta\_\{\\theta\}\-\\delta\_\{\\text\{ref\}\}^\{\(i\)\}\)\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}^\{\(i\)\}
12:

ℓ←−log⁡σ​\(logits\)\\ell\\leftarrow\-\\log\\sigma\(\\text\{logits\}\)
13:endfor

14:Compute gradient:

g←∇θ1\|ℬ\|​∑\(x,yw,yl\)∈ℬℓg\\leftarrow\\nabla\_\{\\theta\}\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{B\}\}\\ell
15:Update parameters:

θ←θ−η​g\\theta\\leftarrow\\theta\-\\eta g
16:endfor

17:Return: Optimized policy

πθ\\pi\_\{\\theta\}

Algorithm[1](https://arxiv.org/html/2605.20834#alg1)presents the complete CPO training procedure\. The key differences from standard DPO are: \(1\) precomputation of reference\-based adaptive marginsγ~ref\(i\)\\tilde\{\\gamma\}\_\{\\text\{ref\}\}^\{\(i\)\}for each sample \(lines 2\-4\), which can be done once before training, and \(2\) subtracting this margin from the logits \(line 12\)\. The precomputation step ensures the optimization objective is stationary, enabling standard gradient descent with convergence guarantees\. The adaptive margin naturally adjusts based on the reference policy’s confidence for each preference pair, as discussed in Theorem[D\.3](https://arxiv.org/html/2605.20834#A4.Thmtheorem3)\.

Using the reference\-based marginγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}, the CPO loss becomes a stationary objective, and its gradient is:

∇θℒCPO=−β​𝔼​\[σ​\(−β​\(δπθ−δπref\)\+γ~ref\)​g\],\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=\-\\beta\\mathbb\{E\}\\left\[\\sigma\\left\(\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\+\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\\right\)g\\right\],\(21\)whereg=\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)g=\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\); details are provided in Appendix[D\.7](https://arxiv.org/html/2605.20834#A4.SS7)\. Crucially, sinceγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}does not depend onθ\\theta, we have∇θγ~ref=0\\nabla\_\{\\theta\}\\tilde\{\\gamma\}\_\{\\text\{ref\}\}=0, making this a standard first\-order gradient suitable for gradient descent\.

The gradient weightw=σ​\(β​\(δπref−δπθ\)\+γ~ref\)w=\\sigma\(\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\+\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\)has an intuitive interpretation: 1\) Whenδπθ\\delta\_\{\\pi\_\{\\theta\}\}is small \(policy not yet preferringywy\_\{w\}\), the weight is large, providing strong gradient signal, 2\) Whenδπθ\\delta\_\{\\pi\_\{\\theta\}\}is large \(policy already strongly prefersywy\_\{w\}\), the weight is small, reducing unnecessary updates, and 3\) The margin termγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}shifts the weighting function, ensuring that even whenδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}is negative \(reference policy misaligned\), the gradient remains strong enough to pushδπθ\\delta\_\{\\pi\_\{\\theta\}\}toward positive values\. Building on these observations, CPO connects preference optimization to constrained RLHF through the adaptive marginγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}, providing a principled framework for margin\-based preference learning\.

### 3\.5Explicitly Constrained Preference Optimization

While CPO provides a principled framework, it relies on the selection of the hyper\-parameterγ\\gammaand uses a soft penalty that*encourages*πθ\\pi\_\{\\theta\}to preferywy\_\{w\}overyly\_\{l\}instead of*ensuring*the preference\. We now introduce*Conservative Explicitly Constrained Preference Optimization*\(E\-CPOC\), which explicitly enforces preference alignment through hard constraints without requiring a reward model\.

###### Definition 3\.11\(Explicitly Constrained RLHF\)\.

We formulate the preference\-aligned RLHF objective as:

maxπ⁡𝔼x∼𝒟​𝔼y∼π\(⋅\|x\)​\[r​\(x,y\)\]−β​KL⁡\(π∥πref\)\\max\_\{\\pi\}\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta\\operatorname\{KL\}\(\\pi\\\|\\pi\_\{\\text\{ref\}\}\)\(22\)s\.t\.δπ​\(x,yw,yl\)≥γ​\(x,yw,yl\),∀\(x,yw,yl\)∈𝒟s\.t\.\\ \\delta\_\{\\pi\}\(x,y\_\{w\},y\_\{l\}\)\\geq\\gamma\(x,y\_\{w\},y\_\{l\}\),\\quad\\forall\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\(23\)whereγ​\(x,yw,yl\)\>0\\gamma\(x,y\_\{w\},y\_\{l\}\)\>0is a minimum required preference margin for each pair\.

The constraint directly ensures that the learned policy must preferywy\_\{w\}overyly\_\{l\}with at least marginγ\\gammain log\-probability space\. This is the condition needed to guarantee absolute preference alignment \(AssumptionLABEL:assump:alignment\)\. Similar to CPO, we first give the log\-probability ratio of the optimal policyδπ∗\\delta\_\{\\pi^\{\*\}\}in Theorem[3\.12](https://arxiv.org/html/2605.20834#S3.Thmtheorem12)with details and proofs in AppendixLABEL:app:oerlhf\.

###### Theorem 3\.12\(Log\-probability Ratio of Optimal Policy\)\.

The optimal policy for constrained RLHF satisfies:

δπ∗=δπref\+Δ​rβ\+Φ​\(δπref,Δ​r;γ,τ\),\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r\}\{\\beta\}\+\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r;\\gamma,\\tau\),\(24\)whereΦ​\(δπref,Δ​r;γ,τ\)\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r;\\gamma,\\tau\)is defined as:

1τ​log⁡\(1\+exp⁡\(τ​\(γ−δπref−Δ​rβ\)\)\),\\frac\{1\}\{\\tau\}\\log\\left\(1\+\\exp\\left\(\\tau\\left\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\frac\{\\Delta r\}\{\\beta\}\\right\)\\right\)\\right\),\(25\)withτ\>0\\tau\>0controlling smoothness\.

We now derive the E\-CPOC loss from the explicitly constrained RLHF\. From Theorem[3\.12](https://arxiv.org/html/2605.20834#S3.Thmtheorem12), the optimal policy satisfiesδπ∗=δπref\+Δ​rβ\+Φ​\(δπref,Δ​r\)\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r\}\{\\beta\}\+\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r\)\. The first\-order optimality condition \(AppendixLABEL:app:oerlhf\) reveals that the effective margin contribution of the Lagrange multipliers equalsβ​Φ\\beta\\Phiexactly \(PropositionLABEL:prop:optimal\_multiplier\)\. A key structural insight is that while the individual multiplierμ∗\\mu^\{\*\}depends onπ∗\\pi^\{\*\}, the effective marginμ∗​\(1/π∗​\(yw\|x\)\+1/π∗​\(yl\|x\)\)=β​Φ\\mu^\{\*\}\(1/\\pi^\{\*\}\(y\_\{w\}\|x\)\+1/\\pi^\{\*\}\(y\_\{l\}\|x\)\)=\\beta\\Phiadmits a closed form independent ofπ∗\\pi^\{\*\}\. Unlike CPO’s marginγ~ref=γ​\(1/πref​\(yw\|x\)\+1/πref​\(yl\|x\)\)\\tilde\{\\gamma\}\_\{\\text\{ref\}\}=\\gamma\(1/\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+1/\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\)which arises from approximating1/π∗1/\\pi^\{\*\}with1/πref1/\\pi\_\{\\text\{ref\}\}, the marginβ​Φ\\beta\\Phiabsorbs the1/π∗1/\\pi^\{\*\}factors through the endogenous Lagrange multipliers of the KKT conditions, eliminating the approximation error entirely\.

The general adaptive marginΦ​\(δπref,Δ​r\)\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r\)depends on the true reward differenceΔ​r\\Delta r, which is typically unknown\. Rather than introducing a separate reward model to estimateΔ​r\\Delta r, we derive a reward\-model\-free formulation with provable guarantees by exploiting a key monotonicity property:Φ​\(δπref,Δ​r\)\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r\)is monotone non\-increasing inΔ​r\\Delta r\(Proposition[E\.1](https://arxiv.org/html/2605.20834#A5.Thmtheorem1)\)\. Since preference data satisfiesΔ​r\>0\\Delta r\>0by the Bradley\-Terry model, the maximum value ofΦ\\Phiis achieved atΔ​r→0\+\\Delta r\\to 0^\{\+\}, yielding the conservative upper boundΦcons​\(δπref\):=Φ​\(δπref,0\)≥Φ​\(δπref,Δ​r∗\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\):=\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\)\\geq\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r^\{\*\}\)for allΔ​r∗\>0\\Delta r^\{\*\}\>0\. Applying the Bradley\-Terry model, we obtain the E\-CPOC loss \(detailed derivation in AppendixLABEL:app:cpoed, Algorithm[2](https://arxiv.org/html/2605.20834#alg2)in Appendix[E](https://arxiv.org/html/2605.20834#A5)\):

ℒE\-CPOC​\(πθ\)=−𝔼𝒟​\[log⁡σ​\(β​\(δπθ−δπref\)−β​Φcons​\(δπref\)\)\],\\mathcal\{L\}\_\{\\text\{E\-CPOC\}\}\(\\pi\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\log\\sigma\\left\(\\beta\\left\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\right\)\-\\beta\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)\\right\],\(26\)whereΦcons​\(δπref\)=Φ​\(δπref,0\)=1τ​log⁡\(1\+exp⁡\(τ​\(γ−δπref\)\)\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\)=\\frac\{1\}\{\\tau\}\\log\\left\(1\+\\exp\\left\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)\\right\)\.

E\-CPOC requires no reward model while providing provable alignment guarantees\. The adaptive margin functionΦcons​\(δπref\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)provides stronger correction for difficult samples \(whereδπref≪γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll\\gamma\) and minimal correction for easy samples \(whereδπref≫γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg\\gamma\), implementing sample\-adaptive weighting that emerges naturally from the constrained optimization framework\. Key properties include: \(1\) monotonicity inδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, ensuring consistent behavior; \(2\) automatic gradient weighting that focuses optimization on difficult pairs; and \(3\) interpretability as measuring constraint violation degree \(Propositions[E\.4](https://arxiv.org/html/2605.20834#A5.Thmtheorem4),[E\.5](https://arxiv.org/html/2605.20834#A5.Thmtheorem5),[E\.6](https://arxiv.org/html/2605.20834#A5.Thmtheorem6)\)\. The complete derivation, algorithm, theoretical analysis, and detailed property proofs are provided in Appendix[E](https://arxiv.org/html/2605.20834#A5)\. E\-CPOC is provably equivalent to explicitly constrained RLHF in the sense thatδπE\-CPOC∗≥δπEC\-RLHF∗​\(Δ​r∗\)\\delta\_\{\\pi^\{\*\}\_\{\\text\{E\-CPOC\}\}\}\\geq\\delta\_\{\\pi^\{\*\}\_\{\\text\{EC\-RLHF\}\}\}\(\\Delta r^\{\*\}\)for any true reward differenceΔ​r∗\>0\\Delta r^\{\*\}\>0\(TheoremLABEL:thm:ecpoc\_equivalencein AppendixLABEL:app:aee\)\. This equivalence requires only standard statistical learning assumptions \(Assumptions[3\.1](https://arxiv.org/html/2605.20834#S3.Thmtheorem1)–[3\.4](https://arxiv.org/html/2605.20834#S3.Thmtheorem4)\), without requiring a reward model\. Theℓ2\\ell^\{2\}\-δ\\delta\-proximity condition can be verified in practice through the training loss gap via a bridge lemma with anNN\-independent bound \(PropositionLABEL:prop:loss\_to\_delta; CorollaryLABEL:cor:verifiable\_equiv\)\.

## 4Preference Learning as Reranking

We provide an intuitive understanding of what will happen to DPO when the implicit AssumptionLABEL:assump:alignmentis violated, and how CPO and E\-CPOC correct this failure\.

The standard margin ranking loss for preference learning is:

ℒhinge​\(sw,sl;m\)=max⁡\(0,m−\(sw−sl\)\)\\mathcal\{L\}\_\{\\text\{hinge\}\}\(s\_\{w\},s\_\{l\};m\)=\\max\(0,m\-\(s\_\{w\}\-s\_\{l\}\)\)\(27\)wheresw,sls\_\{w\},s\_\{l\}are scores for the preferred and rejected responses, andm≥0m\\geq 0is the target margin\.

We can see that DPO is the hinge loss with target marginm=δπrefm=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\. Proof can be found in Appendix[G\.1](https://arxiv.org/html/2605.20834#A7.SS1)\.

###### Proposition 4\.1\(DPO as Soft Margin Ranking\)\.

DPO is a smooth approximation to margin ranking loss\. In the high\-temperature limitβ→∞\\beta\\to\\infty, for any fixed\(δπθ,δπref\)∈ℝ2\(\\delta\_\{\\pi\_\{\\theta\}\},\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\in\\mathbb\{R\}^\{2\}:

limβ→∞1β​ℒDPO​\(πθ\)=max⁡\(0,δπref−δπθ\)\.\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\}\)=\\max\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\.\(28\)

Through the geometric view, hinge loss leads to sharp corner atδπθ=δπref\\delta\_\{\\pi\_\{\\theta\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}and DPO loss results in smooth transition aroundδπθ=δπref\\delta\_\{\\pi\_\{\\theta\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}whereβ\\betacontrols sharpness of transition \(largerβ\\beta⇒\\Rightarrowsharper corner, closer to hard hinge\)\. Whenδπref<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<0\(reference policy disfavorsywy\_\{w\}\), DPO implements a*negative target margin*\. The loss becomes zero whenδπθ\>δπref\\delta\_\{\\pi\_\{\\theta\}\}\>\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, which may still correspond toδπθ<0\\delta\_\{\\pi\_\{\\theta\}\}<0\. In contrast, CPO provides guaranteed positive margin, as shown in Theorem[4\.2](https://arxiv.org/html/2605.20834#S4.Thmtheorem2), with the proof in Appendix[G\.2](https://arxiv.org/html/2605.20834#A7.SS2)\.

###### Theorem 4\.2\(CPO as Corrected Soft Margin Ranking\)\.

The CPO loss function is a smooth approximation to margin ranking loss:

limβ→∞1β​ℒCPO​\(πθ\)=max⁡\(0,δπref\+2​γβ−δπθ\),\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}\(\\pi\_\{\\theta\}\)=\\max\\left\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{2\\gamma\}\{\\beta\}\-\\delta\_\{\\pi\_\{\\theta\}\}\\right\),\(29\)with guaranteed non\-negative margin:

meff∗=δπref\+2​γ∗β≥0,m^\{\*\}\_\{\\text\{eff\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{2\\gamma^\{\*\}\}\{\\beta\}\\geq 0,\(30\)Whereγ\\gammais chosen according to Corollary[3\.9](https://arxiv.org/html/2605.20834#S3.Thmtheorem9)\.

Meanwhile, E\-CPOC also provides sample\-adaptive margin, as shown in Theorem[4\.3](https://arxiv.org/html/2605.20834#S4.Thmtheorem3), with the proof in Appendix[G\.3](https://arxiv.org/html/2605.20834#A7.SS3)\.

###### Theorem 4\.3\(E\-CPOC as Adaptive Margin Ranking\)\.

The E\-CPOC loss implements an adaptive margin ranking loss:

limβ→∞1β​ℒE\-CPOC​\(πθ\)=max⁡\(0,δπref\+Φcons​\(δπref\)−δπθ\),\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{E\-CPOC\}\}\(\\pi\_\{\\theta\}\)=\\max\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\delta\_\{\\pi\_\{\\theta\}\}\),\(31\)with guaranteed non\-negative margin:

m∗​\(δπref\)=δπref\+Φcons​\(δπref\)\.m^\{\*\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\.\(32\)

The margin ranking perspective provides three points: 1\) DPO’s equivalence to margin ranking loss with target marginδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}reveals that whenδπref<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<0, DPO optimizes toward a*negative target*, allowing policies that preferyly\_\{l\}overywy\_\{w\}to achieve low loss\. This provides an intuitive explanation for the pathological behavior in Sec\.LABEL:sec:assumption: 1\) CPO and E\-CPOC can be understood as implementing soft margin ranking loss with guaranteed non\-negative margins; 2\) The softplus functionlog⁡\(1\+ex\)\\log\(1\+e^\{x\}\)provides a smooth, differentiable approximation to the hard max operation in hinge loss\. This enables gradient\-based optimization while preserving the essential margin\-based structure; 3\) The parameterβ\\betacontrols how closely the soft margin loss approximates the hard hinge loss\. Largerβ\\betayields sharper transitions and behavior closer to hard margin ranking\.

## 5Experiments

Experimental Setup\.Following previous work\(Menget al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib2)\), we use Llama\-3\-8B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib24)\), and the princeton\-nlp/llama3\-ultrafeedback\-armorm to conduct preference alignment\. We then compare our methods with many baselines listed in TableLABEL:tab:llama3\-8b\-alignment\-resultsand the base model without any alignment process\. Following previous work, we select AlpacaEval 2\(Liet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib37)\)and Arena\-Hard\(Liet al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib38)\)to evaluate our method, which both evaluate conversational skills based on real\-life queries\.

Main Results\.As shown in TableLABEL:tab:llama3\-8b\-alignment\-results, all considered alignment methods yield consistent improvements over the SFT\-Base on both AlpacaEval 2 and Arena\-Hard, confirming the effectiveness of post\-training preference optimization\. Our proposed CPO establishes new SOTA performance among the reported methods\. On AlpacaEval 2, CPO achieves the highest win rate of 25\.15% \(outperforming DPO at 24\.60% by \+0\.55%\) and the strongest length\-controlled win rate of 26\.57% \(surpassing SimPO’s 25\.91% by \+0\.66%\), while maintaining a competitive average response length of 1879 tokens similar to strong baselines like DPO and RDPO, without exhibiting excessive verbosity\. The advantage is particularly pronounced on Arena\-Hard, where CPO reaches 32\.6% WR with a 90% confidence interval\. This represents a \+2\.6% gain over SimPO \(the runner\-up at 30\.0%\) and an even larger \+3\.7% over DPO, highlighting CPO’s superior ability to handle difficult, discriminative prompts where length bias and subtle preference distinctions matter most\.

## 6Conclusion

We prove DPO and RLHF are conditionally equal\. By augmenting RLHF with explicit constraints, we propose CPO and E\-CPOC which address DPO’s failure modes by explicitly ensuring the preference\. E\-CPOC achieves provable equivalence to explicitly constrained RLHF under only standard statistical learning assumptions, requiring no reward model while providing guaranteed alignment\. Our experiments demonstrate that CPO achieves SOTA performance\.

Limitations:This paper needs to be verified on a larger scale and with a larger model to demonstrate the effectiveness of our method\. In addition, the performance of E\-CPOC should also be validated in experiments beyond the theoretical aspects, and training dynamics visualizations \(e\.g\., loss curves, preference accuracy, and fraction of pairs in𝒰\\mathcal\{U\}over training steps\) would further complement the theoretical characterization\.

## Acknowledgement

Zhiqin Yang, Yonggang Zhang, Wei Xue, and Yike Guo were supported by Hong Kong Generative AI Research & Development Center\. Bo Han was supported by NSFC Major Research Plan No\. 92570109 and NSFC General Program No\. 62376235\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. By enforcing stronger adherence to preference data, CPO may amplify biases present in the training data\. This concern is shared across all preference learning methods \(DPO, SimPO, IPO, RLHF\) whose impact depends on data quality\. Notably, CPO’sγ\\gammaprovides a controllable lever: smallerγ\\gammareduces enforcement \(recovering DPO atγ=0\\gamma=0\), allowing practitioners to calibrate adherence based on data confidence\.

## References

- M\. G\. Azar, Z\. D\. Guo, B\. Piot, R\. Munos, M\. Rowland, M\. Valko, and D\. Calandriello \(2024\)A general theoretical paradigm to understand learning from human preferences\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 4447–4455\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p2.1),[Appendix B](https://arxiv.org/html/2605.20834#A2.p3.1),[§3\.3](https://arxiv.org/html/2605.20834#S3.SS3.p7.1)\.
- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez, J\. Kerr, J\. Mueller, J\. Ladish, J\. Landau, K\. Ndousse, K\. Lukosuite, L\. Lovitt, M\. Sellitto, N\. Elhage, N\. Schiefer, N\. Mercado, N\. DasSarma, R\. Lasenby, R\. Larson, S\. Ringer, S\. Johnston, S\. Kravec, S\. E\. Showk, S\. Fort, T\. Lanham, T\. Telleen\-Lawton, T\. Conerly, T\. Henighan, T\. Hume, S\. R\. Bowman, Z\. Hatfield\-Dodds, B\. Mann, D\. Amodei, N\. Joseph, S\. McCandlish, T\. Brown, and J\. Kaplan \(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.External Links:[Link](https://arxiv.org/abs/2212.08073)Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.External Links:[Document](https://dx.doi.org/10.2307/2334029)Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1),[Definition 2\.2](https://arxiv.org/html/2605.20834#S2.Thmtheorem2)\.
- C\. Burges, T\. Shaked, E\. Renshaw, A\. Lazier, M\. Deeds, N\. Hamilton, and G\. Hullender \(2005\)Learning to rank using gradient descent\.InProceedings of the 22nd international conference on Machine learning,pp\. 89–96\.Cited by:[Appendix C](https://arxiv.org/html/2605.20834#A3.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p5.2)\.
- Z\. Cao, T\. Qin, T\. Liu, M\. Tsai, and H\. Li \(2007\)Learning to rank: from pairwise approach to listwise approach\.InProceedings of the 24th international conference on Machine learning,pp\. 129–136\.Cited by:[Appendix C](https://arxiv.org/html/2605.20834#A3.p1.1)\.
- S\. Casper, X\. Davies, C\. Shi, T\. K\. Gilbert, J\. Scheurer, J\. Rando, R\. Freedman, T\. Korbak, D\. Lindner, P\. Freire,et al\.\(2023\)Open problems and fundamental limitations of reinforcement learning from human feedback\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.Vol\.30\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1),[§3\.1](https://arxiv.org/html/2605.20834#S3.SS1.p2.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p1.1),[§5](https://arxiv.org/html/2605.20834#S5.p1.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)Model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p2.1)\.
- A\. Fisch, J\. Eisenstein, V\. Zayats, A\. Agarwal, A\. Beirami, C\. Nagpal, P\. Shaw, and J\. Berant \(2024\)Robust preference optimization through reward model distillation\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p2.2)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§2\.2](https://arxiv.org/html/2605.20834#S2.SS2.p1.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)ORPO: monolithic preference optimization without reference model\.In2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p2.1)\.
- S\. Im and Y\. Li \(2024\)Understanding the learning dynamics of alignment with human feedback\.pp\. 20983–21006\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p2.2)\.
- H\. Ivison, Y\. Wang, V\. Pyatkin, N\. Lambert, M\. Peters, P\. Dasigi, J\. Jang, D\. Wadden, N\. A\. Smith, I\. Beltagy,et al\.\(2023\)Camels in a changing climate: enhancing lm adaptation with tulu 2\.arXiv preprint arXiv:2311\.10702\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2024\)From live data to high\-quality benchmarks: the arena\-hard pipeline\.Blog post\.\[Accessed 07\-02\-2025\]\.Cited by:[§5](https://arxiv.org/html/2605.20834#S5.p1.1)\.
- X\. Li, T\. Zhang, Y\. Dubois, R\. Taori, I\. Gulrajani, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2023\)Alpacaeval: an automatic evaluator of instruction\-following models\.Cited by:[§5](https://arxiv.org/html/2605.20834#S5.p1.1)\.
- Y\. Lin, S\. Seto, M\. Ter Hoeve, K\. Metcalf, B\. J\. Theobald, X\. Wang, Y\. Zhang, C\. Huang, and T\. Zhang \(2024\)On the limited generalization capability of the implicit reward model induced by direct preference optimization\.pp\. 16015–16026\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p2.2)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)Simpo: simple preference optimization with a reference\-free reward\.Advances in Neural Information Processing Systems37,pp\. 124198–124235\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.SS0.SSS0.Px1.p1.2),[§5](https://arxiv.org/html/2605.20834#S5.p1.1)\.
- R\. Munos, M\. Valko, D\. Calandriello, M\. G\. Azar, M\. Rowland, Z\. D\. Guo, Y\. Tang, M\. Geist, T\. Mesnard, C\. Fiegel,et al\.\(2024\)Nash learning from human feedback\.InForty\-first International Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Vol\.35,pp\. 27730–27744\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Vol\.36,pp\. 53728–53741\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p2.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.20834#S2.SS2.p2.3),[§2\.4](https://arxiv.org/html/2605.20834#S2.SS4.p2.2),[§3\.1](https://arxiv.org/html/2605.20834#S3.SS1.p2.1)\.
- F\. Schroff, D\. Kalenichenko, and J\. Philbin \(2015\)Facenet: a unified embedding for face recognition and clustering\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 815–823\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p5.2)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- R\. Shi, M\. Song, R\. Zhou, Z\. Zhang, M\. Fazel, and S\. S\. Du \(2025\)Understanding the performance gap in preference learning: a dichotomy of rlhf and dpo\.arXiv preprint arXiv:2505\.19770\.Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p2.2)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.Vol\.33,pp\. 3008–3021\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1),[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- L\. Tunstall, E\. Beeching, N\. Lambert, N\. Rajani, K\. Rasul, Y\. Belkada, S\. Huang, L\. von Werra, C\. Fourrier, N\. Habib,et al\.\(2023\)Zephyr: direct distillation of lm alignment\.arXiv preprint arXiv:2310\.16944\.External Links:[Link](https://arxiv.org/abs/2310.16944)Cited by:[§1](https://arxiv.org/html/2605.20834#S1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§A\.4](https://arxiv.org/html/2605.20834#A1.SS4.p1.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[Appendix B](https://arxiv.org/html/2605.20834#A2.p1.1)\.

## Appendix AMore Experimental Results

### A\.1Measurement of violation frequency

![Refer to caption](https://arxiv.org/html/2605.20834v1/x1.png)Figure 1:Measurement of violation frequency on Llama\-3\-8B\-Instruct under Llama3 ultrafeedback armorn\.We compute the violation statistics\. As shown in Figure[1](https://arxiv.org/html/2605.20834#A1.F1)that AssumptionLABEL:assump:alignmentis violated for 45\.5% of preference pairs\*\* \(Llama\-3\-8B\-Instruct,β=0\.1\\beta=0\.1\)\. The reward correctionΔ​r∗/β\\Delta r^\{\*\}/\\betais small \(mean=0\.20\) relative to the large spread ofδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(std=46\.69\), meaning the reward signal often cannot compensate for the reference policy’s misalignment—placing nearly half of all pairs in the regime where DPO optimizes a fundamentally different objective than RLHF \(TheoremLABEL:thm:conditional\_equivalence\)\. A 45\.5% violation rate on an \*instruction\-tuned\* model confirms the pathology is far from a corner case, strongly motivating CPO’s margin correction \(Theorem\.[3\.9](https://arxiv.org/html/2605.20834#S3.Thmtheorem9)and Theorem[3\.10](https://arxiv.org/html/2605.20834#S3.Thmtheorem10)\)\.

### A\.2Varying Reference Policy Quality

In this section, we systematically vary reference policy quality to directly validation\.

#### Constructing misaligned references\.

From the dataset, we extract a fractionR∈\{0\.2,0\.3,0\.4\}R\\in\\\{0\.2,0\.3,0\.4\\\}of the data and use the rejected responses to SFT Llama\-3\-8B\-Instruct \(1 epoch, lr=2×10−52\\times 10^\{\-5\}\), forcing the model to learn to generate low\-quality responses as a misaligned reference\. The remaining\(1−R\)\(1\-R\)fraction retains original preference ordering for DPO/CPO training\.

#### Verifying misalignment\.

We compute theδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}distribution of each misaligned reference on the original preference data\. As shown in Table[2](https://arxiv.org/html/2605.20834#A1.T2), asRRincreases, the AssumptionLABEL:assump:alignmentviolation rate grows correspondingly, confirming the effectiveness of misalignment\.

Table 2:Misalignment statistics under different corruption ratios\.
#### Results\.

Under each corruption ratio, we train DPO and CPO starting from the misaligned reference on the clean data\. Results on AlpacaEval 2 are shown in Table[3](https://arxiv.org/html/2605.20834#A1.T3)\. Note that the original evaluatorgpt\-4\-1106\-previewhas been deprecated; we re\-evaluate usinggpt\-4\.1as the annotator\.

Table 3:AlpacaEval 2 performance under misaligned reference policies\.
#### Analysis\.

1\. DPO degrades under a misaligned reference\.DPO’s LC WR drops across all three corruption ratios, with stronger corruption leading to worse performance\. This is consistent with PropositionLABEL:prop:undesirable\_space\.

2\. CPO remains robust under the same conditions\.CPO achieves LC of 27\.60% and 27\.35% atR=0\.2R=0\.2andR=0\.3R=0\.3respectively, remaining stable\. This validates the core role of the margin termγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(Eq\.[20](https://arxiv.org/html/2605.20834#S3.E20)\): even whenδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}is negative, the margin preserves gradient strength, enabling the policy to pushδπθ\\delta\_\{\\pi\_\{\\theta\}\}past 0 and escape𝒰\\mathcal\{U\}\.

3\. Fraction in𝒰\\mathcal\{U\}directly validates the theory\.The fraction in𝒰\\mathcal\{U\}trajectory during training exhibits a characteristic three\-phase pattern that directly reflects the theoretical mechanism:

- •Phase 1 \(Initialization\):At step 0, the policy is identical to the reference \(frac in𝒰\\mathcal\{U\}= 0\)\. Sinceδπθ=δπref\\delta\_\{\\pi\_\{\\theta\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}for all samples, the conditionδπθ\>δπref\\delta\_\{\\pi\_\{\\theta\}\}\>\\delta\_\{\\pi\_\{\\text\{ref\}\}\}is not satisfied, so no samples fall in𝒰\\mathcal\{U\}\.
- •Phase 2 \(Entry into𝒰\\mathcal\{U\}\):As training begins, gradients pushδπθ\\delta\_\{\\pi\_\{\\theta\}\}upward\. For misaligned samples whereδπref<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<0, the policy improves relative to the reference but has not yet crossed 0, resulting inδπref<δπθ<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<\\delta\_\{\\pi\_\{\\theta\}\}<0, exactly the𝒰\\mathcal\{U\}region\. This causes frac in𝒰\\mathcal\{U\}to rise\.
- •Phase 3 \(Escape attempt\):Asδπθ\\delta\_\{\\pi\_\{\\theta\}\}continues to increase and some samples cross 0 \(δπθ\>0\\delta\_\{\\pi\_\{\\theta\}\}\>0\), they exit𝒰\\mathcal\{U\}, causing frac in𝒰\\mathcal\{U\}to decrease\. The critical divergence emerges: - –DPO:The gradient weightσ​\(−β​\(δπθ−δπref\)\)\\sigma\(\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)weakens asδπθ\\delta\_\{\\pi\_\{\\theta\}\}approaches 0 \(PropositionLABEL:prop:undesirable\_space\)\. Many samples get stuck atδπθ≈0−\\delta\_\{\\pi\_\{\\theta\}\}\\approx 0^\{\-\}, unable to escape\. - –CPO:The margin termγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}shifts the gradient weighting function, ensuring strong gradient signal even near the𝒰\\mathcal\{U\}boundary\. Most samples successfully push pastδπθ=0\\delta\_\{\\pi\_\{\\theta\}\}=0, and frac in𝒰\\mathcal\{U\}rapidly drops\.

![Refer to caption](https://arxiv.org/html/2605.20834v1/x2.png)Figure 2:Fraction of training samples in the undesirable solution space𝒰\\mathcal\{U\}\(Definition 3\.3\) over training steps under different corruption ratiosR∈\{0\.2,0\.3,0\.4\}R\\in\\\{0\.2,0\.3,0\.4\\\}\.The training dynamics of frac in𝒰\\mathcal\{U\}are visualized in Figure[2](https://arxiv.org/html/2605.20834#A1.F2)\. The difference in frac in𝒰\\mathcal\{U\}directly corresponds to the theoretical contrast between DPO’s weak gradient near the𝒰\\mathcal\{U\}boundary \(PropositionLABEL:prop:undesirable\_space\) and CPO’s margin\-corrected gradients \(Theorem[4\.2](https://arxiv.org/html/2605.20834#S4.Thmtheorem2)\)\.

### A\.3Sensitivity Analysis ofγ\\gamma

We conduct a sensitivity analysis of the hyperparameterγ\\gammaon AlpacaEval 2\. Note that the original evaluatorgpt\-4\-1106\-previewhas been deprecated; we re\-evaluate usinggpt\-4\.1as the annotator\. Results are shown in Table[4](https://arxiv.org/html/2605.20834#A1.T4)\. CPO performs robustly acrossγ∈\[0\.2,0\.4\]\\gamma\\in\[0\.2,0\.4\]\(WR 26–28%, LC 31–34%\), with peak performance atγ=0\.25\\gamma=0\.25\. Performance drops notably below0\.20\.2, where the margin correction becomes insufficient to address the assumption violation\. In all experiments reported in the main paper, we useγ=0\.25\\gamma=0\.25\.

Table 4:Sensitivity analysis ofγ\\gammaon AlpacaEval 2 using Llama\-3\-8B\-Instruct\.
### A\.4Evaluation on IFEval

Following the reviewer’s suggestion, we add IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib43)\)as an additional benchmark to evaluate instruction\-following capability\. Results are shown in Table[5](https://arxiv.org/html/2605.20834#A1.T5)\. CPO achieves the highest performance on both strict and loose accuracy, confirming that the improvement from CPO extends beyond conversational benchmarks to instruction\-following tasks\.

Table 5:IFEval results on Llama\-3\-8B\-Instruct\.
### A\.5Comparison with Clipped\-Reference Baseline

To isolate the effect of CPO’s adaptive margin from generic margin regularization, we compare with a clipped\-reference baseline that clipsδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}to be non\-negative before applying the standard DPO loss\. Results on AlpacaEval 2 are shown in Table[6](https://arxiv.org/html/2605.20834#A1.T6)\. CPO substantially outperforms the clipped\-reference baseline, demonstrating that the adaptive marginγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}provides benefits beyond simply preventing negative margins

Table 6:Comparison with clipped\-reference baseline on AlpacaEval 2\.

## Appendix BRelated Work

RLHF has become the standard approach for aligning LLMs with human preferences\(Christianoet al\.,[2017](https://arxiv.org/html/2605.20834#bib.bib9); Stiennonet al\.,[2020](https://arxiv.org/html/2605.20834#bib.bib20); Ouyanget al\.,[2022](https://arxiv.org/html/2605.20834#bib.bib16)\)\. The framework typically uses the Bradley\-Terry model\(Bradley and Terry,[1952](https://arxiv.org/html/2605.20834#bib.bib5)\)to learn reward functions from pairwise preference data, followed by policy optimization with KL regularization to prevent reward over\-optimization\(Ziegleret al\.,[2019](https://arxiv.org/html/2605.20834#bib.bib22); Gaoet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib11)\)\. While effective, RLHF requires training a separate reward model and applying RL algorithms like PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.20834#bib.bib19)\), making it computationally expensive and potentially unstable\.

DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.20834#bib.bib18)\)proposes to bypass explicit reward modeling by directly optimizing policies on preference data, claiming theoretical equivalence to RLHF\. This simplicity has led to widespread adoption and numerous variants\. IPO\(Azaret al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib1)\)is proposed for better calibration and robustness to noise\. KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib10)\)extends to binary feedback rather than pairwise comparisons\. ORPO\(Honget al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib12)\)combines preference optimization with supervised fine\-tuning\. However, all these variants retain structural similarities to DPO and do not address the conditional nature of the DPO\-RLHF equivalence that we identify\.

Recent works have begun examining theoretical properties of preference\-based methods\.Azaret al\.\([2024](https://arxiv.org/html/2605.20834#bib.bib1)\)analyze the Nash equilibrium properties of preference learning and propose regularization for improved sample efficiency\.Munoset al\.\([2024](https://arxiv.org/html/2605.20834#bib.bib15)\)study the game\-theoretic foundations of RLHF\. While these works provide valuable theoretical insights, none have systematically characterized the conditions under which DPO\-RLHF equivalence holds or fails\. Our work is the first to identify the implicit assumption, prove the equivalence is conditional, characterize precise failure conditions, and provide methods with provable alignment guarantees\.

#### Reference\-free methods\.

SimPO\(Menget al\.,[2024](https://arxiv.org/html/2605.20834#bib.bib2)\)replaces the reward reparameterizationr​\(x,y\)=β​log⁡πθ​\(y\|x\)πref​\(y\|x\)r\(x,y\)=\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}with length\-normalized log\-probability as an implicit reward, a heuristic not derived from any RLHF objective or Bradley\-Terry model\. By removing the reference policy entirely, SimPO sidesteps the assumption violation identified in this work \(Assumption 3\.1\), but at the cost of abandoning the RLHF\-BT framework: it cannot claim equivalence to any reward\-maximizing objective with KL regularization\. In contrast, CPO and E\-CPOC retain the full RLHF\-BT chain with formal guarantees: absolute advantage \(Theorem 4\.9\), avoidance of𝒰\\mathcal\{U\}\(Theorem 4\.10\), and provable equivalence to constrained RLHF \(Theorem L\.17\)\. Our goal is to understand*why*DPO’s RLHF equivalence breaks and*how*to fix it with guarantees, which is fundamentally different from designing reference\-free heuristics\. Nonetheless, investigating how reference\-free methods relate to the margin ranking perspective \(Section 5\) is an interesting direction for future work\.

## Appendix CExplanation about Preference Learning as Reranking

The margin ranking loss is a fundamental tool in learning\-to\-rank\(Burgeset al\.,[2005](https://arxiv.org/html/2605.20834#bib.bib6); Caoet al\.,[2007](https://arxiv.org/html/2605.20834#bib.bib8)\), where it ensures that relevant items are ranked above irrelevant ones with sufficient margin\. Our analysis shows that preference learning can be viewed as a reranking problem in log\-probability space, where the margin ensures that preferred responses are ranked above dispreferred ones\. This unified view through margin ranking loss not only provides an intuitive understanding of DPO’s failure modes and our corrections, but also connects preference learning to a rich body of existing theory and practice in ranking and margin\-based learning\.

## Appendix DProofs\.

#### Clarification on AssumptionLABEL:assump:alignment\.

The algebraic substitution in Eq\.[7](https://arxiv.org/html/2605.20834#S2.E7)holds regardless of the sign ofδπ∗\\delta\_\{\\pi^\{\*\}\}\. AssumptionLABEL:assump:alignmentis not needed for the substitution itself, but for the equivalence of objectives between DPO and RLHF\. The DPO loss−log⁡σ​\(β​\(δπθ−δπref\)\)\-\\log\\sigma\(\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)is monotonically decreasing inδπθ\\delta\_\{\\pi\_\{\\theta\}\}, so it always pushesδπθ\\delta\_\{\\pi\_\{\\theta\}\}upward toward preferringywy\_\{w\}\. Meanwhile, the RLHF optimal policy satisfiesδπ∗=δπref\+Δ​r∗/β\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Delta r^\{\*\}/\\beta, which can be≤0\\leq 0\. When this happens, DPO’s optimum \(δπθ→∞\\delta\_\{\\pi\_\{\\theta\}\}\\to\\infty\) diverges from RLHF’s optimum \(δπ∗<0\\delta\_\{\\pi^\{\*\}\}<0\), and the two methods optimize fundamentally different objectives \(TheoremLABEL:thm:conditional\_equivalence\)\.

### D\.1Proof of Necessary Condition for AssumptionLABEL:assump:alignment

Proof of PropositionLABEL:thm:necessary\_condition\(Necessary Condition for AssumptionLABEL:assump:alignment\)\.

###### Proof\.

Human preferenceyw≻yly\_\{w\}\\succ y\_\{l\}impliesr∗​\(yw\)−r∗​\(yl\)\>0r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\>0\.

By the core relationship \(Equation \([5](https://arxiv.org/html/2605.20834#S2.E5)\)\):

δπ∗=δπref\+r∗​\(yw\)−r∗​\(yl\)β\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\}\{\\beta\}\(33\)
For AssumptionLABEL:assump:alignmentto hold, we requireδπ∗\>0\\delta\_\{\\pi^\{\*\}\}\>0:

δπref\+r∗​\(yw\)−r∗​\(yl\)β\>0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\}\{\\beta\}\>0\(34\)
Rearranging yields the necessary condition\. ∎

### D\.2Proof of Non\-emptiness and Weak Gradient

Proof of PropositionLABEL:prop:undesirable\_space\(Non\-emptiness and Weak gradient\)\.

###### Proof\.

Non\-emptiness:Sinceδπref<−Δ​r∗/β<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<\-\\Delta r^\{\*\}/\\beta<0, any policy withδπref<δπ<0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}<\\delta\_\{\\pi\}<0belongs to𝒰\\mathcal\{U\}\.

Weak gradient:The gradient ofℒDPO\\mathcal\{L\}\_\{\\text\{DPO\}\}with respect to policy parameters is:

∇θℒDPO=−𝔼​\[σ​\(−β​\(δπθ−δπref\)\)⋅β⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\]\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\-\\mathbb\{E\}\\left\[\\sigma\(\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\cdot\\beta\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\\right\]\(35\)
Forπ∈𝒰\\pi\\in\\mathcal\{U\}, we haveδπ\>δπref\\delta\_\{\\pi\}\>\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, soδπ−δπref\>0\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\>0, making−β​\(δπ−δπref\)<0\-\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)<0and thusσ​\(−β​\(δπ−δπref\)\)<0\.5\\sigma\(\-\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)<0\.5\. The gradient direction pushesδπθ\\delta\_\{\\pi\_\{\\theta\}\}to increase \(i\.e\.,∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)increases\)\.

However, asδπ\\delta\_\{\\pi\}increases toward 0 \(the boundary of preference violation\), the differenceδπ−δπref\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}becomes larger, causing−β​\(δπ−δπref\)\-\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)to become more negative andσ​\(−β​\(δπ−δπref\)\)→0\\sigma\(\-\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\to 0\. This causes the gradient magnitude to become progressively weaker, constituting a weak\-gradient problem conditioned on the quality ofπref\\pi\_\{\\text\{ref\}\}\. Sinceδπ<0\\delta\_\{\\pi\}<0throughout𝒰\\mathcal\{U\}, the policy remains trapped in the preference\-violating region while the loss continues to decrease\. ∎

### D\.3Proof of Conditional Equivalence of DPO and RLHF

Proof of TheoremLABEL:thm:conditional\_equivalence\(Conditional Equivalence of DPO and RLHF\)

###### Proof\.

We prove both directions of the if\-and\-only\-if statement\. Throughout, letπ∗\\pi^\{\*\}denote the RLHF\-optimal policy andπDPO\\pi\_\{\\text\{DPO\}\}denote the DPO\-optimal policy\.

\(⇒\\Rightarrow\) Sufficiency:Assume Condition \(LABEL:eq:equivalence\_condition\) holds for all\(x,yw,yl\)∈𝒟\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\.

*Step 1: RLHF\-optimal policy respects human preferences\.*By Eq\.[3](https://arxiv.org/html/2605.20834#S2.E3), the RLHF\-optimal policy satisfies:

δπ∗​\(x,yw,yl\)=δπref​\(x,yw,yl\)\+r∗​\(x,yw\)−r∗​\(x,yl\)β\.\\delta\_\{\\pi^\{\*\}\}\(x,y\_\{w\},y\_\{l\}\)=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\+\\frac\{r^\{\*\}\(x,y\_\{w\}\)\-r^\{\*\}\(x,y\_\{l\}\)\}\{\\beta\}\.\(36\)Under Condition \(LABEL:eq:equivalence\_condition\), we haveδπref\>−Δ​r∗/β\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\>\-\\Delta r^\{\*\}/\\beta, whereΔ​r∗:=r∗​\(x,yw\)−r∗​\(x,yl\)\>0\\Delta r^\{\*\}:=r^\{\*\}\(x,y\_\{w\}\)\-r^\{\*\}\(x,y\_\{l\}\)\>0\. Therefore:

δπ∗=δπref\+Δ​r∗β\>0,\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\>0,\(37\)implyingπ∗​\(yw\|x\)\>π∗​\(yl\|x\)\\pi^\{\*\}\(y\_\{w\}\|x\)\>\\pi^\{\*\}\(y\_\{l\}\|x\), which aligns with the human preferenceyw≻yly\_\{w\}\\succ y\_\{l\}\.

*Step 2: Bradley\-Terry model is well\-defined atπ∗\\pi^\{\*\}\.*Sinceδπ∗\>0\\delta\_\{\\pi^\{\*\}\}\>0, the Bradley\-Terry preference probability atπ∗\\pi^\{\*\}is:

p∗​\(yw≻yl\|x\)=σ​\(r∗​\(x,yw\)−r∗​\(x,yl\)\)=σ​\(β​\(δπ∗−δπref\)\)\>0\.5\.p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)=\\sigma\(r^\{\*\}\(x,y\_\{w\}\)\-r^\{\*\}\(x,y\_\{l\}\)\)=\\sigma\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\>0\.5\.\(38\)This is consistent with the human preference structure, validating the use of the Bradley\-Terry model in DPO’s derivation\.

*Step 3:π∗\\pi^\{\*\}is a stationary point of the DPO objective\.*The DPO objective is:

ℒDPO​\(π\)=−𝔼\(x,yw,yl\)∼𝒟​\[log⁡σ​\(β​\(δπ−δπref\)\)\]\.\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\)=\-\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\left\[\\log\\sigma\(\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\right\]\.\(39\)Taking the gradient with respect to policy parametersθ\\theta:

∇θℒDPO=−𝔼​\[σ​\(−β​\(δπθ−δπref\)\)⋅β⋅∇θδπθ\],\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\-\\mathbb\{E\}\\left\[\\sigma\(\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\cdot\\beta\\cdot\\nabla\_\{\\theta\}\\delta\_\{\\pi\_\{\\theta\}\}\\right\],\(40\)where∇θδπθ=∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\\nabla\_\{\\theta\}\\delta\_\{\\pi\_\{\\theta\}\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\.

By the reward reparameterization \(Lemma[4](https://arxiv.org/html/2605.20834#S2.E4)\), atπ=π∗\\pi=\\pi^\{\*\}:

r∗​\(x,y\)=β​log⁡π∗​\(y\|x\)πref​\(y\|x\)\+β​log⁡Z​\(x\),r^\{\*\}\(x,y\)=\\beta\\log\\frac\{\\pi^\{\*\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\+\\beta\\log Z\(x\),\(41\)whereZ​\(x\)Z\(x\)is the partition function\. Substituting into the Bradley\-Terry model:

p∗​\(yw≻yl\|x\)=σ​\(β​\(δπ∗−δπref\)\)\.p^\{\*\}\(y\_\{w\}\\succ y\_\{l\}\|x\)=\\sigma\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\.\(42\)This shows thatπ∗\\pi^\{\*\}satisfies the maximum likelihood condition for the Bradley\-Terry model parameterized by DPO, making it a stationary point ofℒDPO\\mathcal\{L\}\_\{\\text\{DPO\}\}\.

*Step 4:π∗\\pi^\{\*\}is the global optimum of the DPO objective\.*The DPO lossℒDPO​\(π\)=−𝔼​\[log⁡σ​\(β​\(δπ−δπref\)\)\]\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\)=\-\\mathbb\{E\}\[\\log\\sigma\(\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\]is strictly convex inδπ\\delta\_\{\\pi\}\. To see this, note thatf​\(z\)=−log⁡σ​\(z\)=log⁡\(1\+e−z\)f\(z\)=\-\\log\\sigma\(z\)=\\log\(1\+e^\{\-z\}\)has second derivativef′′​\(z\)=σ​\(z\)​σ​\(−z\)\>0f^\{\\prime\\prime\}\(z\)=\\sigma\(z\)\\sigma\(\-z\)\>0, establishing strict convexity\.

Under Condition \(LABEL:eq:equivalence\_condition\),π∗\\pi^\{\*\}satisfies the first\-order optimality condition \(Step 3\)\. SinceℒDPO\\mathcal\{L\}\_\{\\text\{DPO\}\}depends only onδπ\\delta\_\{\\pi\}and is strictly convex in this quantity, the optimalδπ\\delta\_\{\\pi\}values are uniquely determined\. Specifically, any policyπ~\\tilde\{\\pi\}minimizingℒDPO\\mathcal\{L\}\_\{\\text\{DPO\}\}must satisfy:

δπ~​\(x,yw,yl\)=δπ∗​\(x,yw,yl\)∀\(x,yw,yl\)∈𝒟\.\\delta\_\{\\tilde\{\\pi\}\}\(x,y\_\{w\},y\_\{l\}\)=\\delta\_\{\\pi^\{\*\}\}\(x,y\_\{w\},y\_\{l\}\)\\quad\\forall\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\.\(43\)
This equality of log\-probability ratios implies thatπ~\\tilde\{\\pi\}andπ∗\\pi^\{\*\}have identical preference structures:π~​\(yw\|x\)/π~​\(yl\|x\)=π∗​\(yw\|x\)/π∗​\(yl\|x\)\\tilde\{\\pi\}\(y\_\{w\}\|x\)/\\tilde\{\\pi\}\(y\_\{l\}\|x\)=\\pi^\{\*\}\(y\_\{w\}\|x\)/\\pi^\{\*\}\(y\_\{l\}\|x\)for all preference pairs\. Therefore,πDPO\\pi\_\{\\text\{DPO\}\}andπ∗\\pi^\{\*\}are equivalent in terms of preference ordering, establishing that DPO and RLHF optimize the same objective under the given condition\.

\(⇐\\Leftarrow\) Necessity:We prove the contrapositive: if Condition \(LABEL:eq:equivalence\_condition\) is violated, then DPO and RLHF optimize different objectives\.

Suppose there exists\(x0,yw0,yl0\)∈𝒟\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)\\in\\mathcal\{D\}such that:

δπref​\(x0,yw0,yl0\)≤−r∗​\(x0,yw0\)−r∗​\(x0,yl0\)β\.\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)\\leq\-\\frac\{r^\{\*\}\(x\_\{0\},y\_\{w\}^\{0\}\)\-r^\{\*\}\(x\_\{0\},y\_\{l\}^\{0\}\)\}\{\\beta\}\.\(44\)
*Step 1: RLHF\-optimal policy violates human preference at\(x0,yw0,yl0\)\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)\.*By Eq\.[3](https://arxiv.org/html/2605.20834#S2.E3):

δπ∗​\(x0,yw0,yl0\)=δπref​\(x0,yw0,yl0\)\+r∗​\(x0,yw0\)−r∗​\(x0,yl0\)β≤0,\\delta\_\{\\pi^\{\*\}\}\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)\+\\frac\{r^\{\*\}\(x\_\{0\},y\_\{w\}^\{0\}\)\-r^\{\*\}\(x\_\{0\},y\_\{l\}^\{0\}\)\}\{\\beta\}\\leq 0,\(45\)implyingπ∗​\(yl0\|x0\)≥π∗​\(yw0\|x0\)\\pi^\{\*\}\(y\_\{l\}^\{0\}\|x\_\{0\}\)\\geq\\pi^\{\*\}\(y\_\{w\}^\{0\}\|x\_\{0\}\)\. Thus,π∗\\pi^\{\*\}prefers the dispreferred response at this data point, yet RLHF accepts this as optimal due to the KL regularization constraint\.

*Step 2:π∗\\pi^\{\*\}is not optimal for the DPO objective\.*We construct a policyπϵ\\pi\_\{\\epsilon\}that achieves lower DPO loss thanπ∗\\pi^\{\*\}\. For the promptx0x\_\{0\}where the condition is violated, define:

πϵ​\(y\|x0\)=π∗​\(y\|x0\)​exp⁡\(ϵ​\[𝕀​\(y=yw0\)−𝕀​\(y=yl0\)\]\)∑y′π∗​\(y′\|x0\)​exp⁡\(ϵ​\[𝕀​\(y′=yw0\)−𝕀​\(y′=yl0\)\]\)\\pi\_\{\\epsilon\}\(y\|x\_\{0\}\)=\\frac\{\\pi^\{\*\}\(y\|x\_\{0\}\)\\exp\(\\epsilon\[\\mathbb\{I\}\(y=y\_\{w\}^\{0\}\)\-\\mathbb\{I\}\(y=y\_\{l\}^\{0\}\)\]\)\}\{\\sum\_\{y^\{\\prime\}\}\\pi^\{\*\}\(y^\{\\prime\}\|x\_\{0\}\)\\exp\(\\epsilon\[\\mathbb\{I\}\(y^\{\\prime\}=y\_\{w\}^\{0\}\)\-\\mathbb\{I\}\(y^\{\\prime\}=y\_\{l\}^\{0\}\)\]\)\}\(46\)for someϵ\>0\\epsilon\>0, andπϵ​\(y\|x\)=π∗​\(y\|x\)\\pi\_\{\\epsilon\}\(y\|x\)=\\pi^\{\*\}\(y\|x\)for allx≠x0x\\neq x\_\{0\}\. This construction ensuresπϵ\\pi\_\{\\epsilon\}is a valid probability distribution with:

δπϵ​\(x0,yw0,yl0\)=δπ∗​\(x0,yw0,yl0\)\+2​ϵ\.\\delta\_\{\\pi\_\{\\epsilon\}\}\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)=\\delta\_\{\\pi^\{\*\}\}\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)\+2\\epsilon\.\(47\)
The DPO loss can be decomposed as:

ℒDPO​\(π\)=𝔼\(x,yw,yl\)∼𝒟​\[−log⁡σ​\(β​\(δπ​\(x,yw,yl\)−δπref​\(x,yw,yl\)\)\)\]\.\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\)=\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\left\[\-\\log\\sigma\(\\beta\(\\delta\_\{\\pi\}\(x,y\_\{w\},y\_\{l\}\)\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\)\)\\right\]\.\(48\)
Letp0p\_\{0\}denote the probability mass of\(x0,yw0,yl0\)\(x\_\{0\},y\_\{w\}^\{0\},y\_\{l\}^\{0\}\)in𝒟\\mathcal\{D\}\. Sinceπϵ=π∗\\pi\_\{\\epsilon\}=\\pi^\{\*\}for allx≠x0x\\neq x\_\{0\}, the difference in losses is:

ℒDPO​\(πϵ\)−ℒDPO​\(π∗\)\\displaystyle\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\epsilon\}\)\-\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi^\{\*\}\)=p0​\[−log⁡σ​\(β​\(δπ∗\+2​ϵ−δπref\)\)\+log⁡σ​\(β​\(δπ∗−δπref\)\)\]\\displaystyle=p\_\{0\}\\left\[\-\\log\\sigma\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\+2\\epsilon\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\+\\log\\sigma\(\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\right\]\(49\)=p0​log⁡σ​\(Δ​r∗\)σ​\(Δ​r∗\+2​β​ϵ\),\\displaystyle=p\_\{0\}\\log\\frac\{\\sigma\(\\Delta r^\{\*\}\)\}\{\\sigma\(\\Delta r^\{\*\}\+2\\beta\\epsilon\)\},\(50\)where we usedδπ∗−δπref=Δ​r∗/β\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}=\\Delta r^\{\*\}/\\betafrom Step 1\.

Sinceσ\\sigmais strictly increasing andϵ\>0\\epsilon\>0, we haveσ​\(Δ​r∗\+2​β​ϵ\)\>σ​\(Δ​r∗\)\\sigma\(\\Delta r^\{\*\}\+2\\beta\\epsilon\)\>\\sigma\(\\Delta r^\{\*\}\), which implies:

ℒDPO​\(πϵ\)<ℒDPO​\(π∗\)\.\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\epsilon\}\)<\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi^\{\*\}\)\.\(51\)
Therefore,π∗\\pi^\{\*\}is not a global minimum of the DPO objective\.

*Step 3: DPO optimizes a different objective\.*Sinceπ∗\\pi^\{\*\}is not optimal for DPO, we haveπDPO≠π∗\\pi\_\{\\text\{DPO\}\}\\neq\\pi^\{\*\}\. RLHF optimizes𝔼​\[r\]−β​KL⁡\(π∥πref\)\\mathbb\{E\}\[r\]\-\\beta\\operatorname\{KL\}\(\\pi\\\|\\pi\_\{\\text\{ref\}\}\)and converges toπ∗\\pi^\{\*\}, while DPO optimizes𝔼​\[log⁡σ​\(β​\(δπ−δπref\)\)\]\\mathbb\{E\}\[\\log\\sigma\(\\beta\(\\delta\_\{\\pi\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\]and converges toπDPO≠π∗\\pi\_\{\\text\{DPO\}\}\\neq\\pi^\{\*\}\. Therefore, the two methods optimize fundamentally different objectives when Condition \(LABEL:eq:equivalence\_condition\) is violated\. ∎

### D\.4Proof of Optimal Policy for Constrained RLHF

###### Definition D\.2\(Augmented Reward\)\.

Under the notation above, for a given preference dataset𝒟\\mathcal\{D\}, define the augmented reward function:

r~​\(x,y\)=\{r​\(x,y\)\+γif​∃\(x,yw,yl\)∈𝒟​with​y=ywr​\(x,y\)−γif​∃\(x,yw,yl\)∈𝒟​with​y=ylr​\(x,y\)otherwise\\tilde\{r\}\(x,y\)=\\begin\{cases\}r\(x,y\)\+\\gamma&\\text\{if \}\\exists\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\\text\{ with \}y=y\_\{w\}\\\\ r\(x,y\)\-\\gamma&\\text\{if \}\\exists\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\\text\{ with \}y=y\_\{l\}\\\\ r\(x,y\)&\\text\{otherwise\}\\end\{cases\}\(52\)
This corresponds to the case where each response appears at most once as a winner and once as a loser for each prompt, with uniform weighting over preference pairs\.

Proof of Theorem[3\.8](https://arxiv.org/html/2605.20834#S3.Thmtheorem8)\(Optimal Policy for Constrained RLHF\)\.

###### Proof\.

We first reformulate the margin term in the objective function\. Under the notation in Remark[D\.1](https://arxiv.org/html/2605.20834#A4.Thmtheorem1), the margin term can be written as:

γ​𝔼\(x,yw,yl\)∼𝒟​\[δπ\]=γ​𝔼\(x,yw,yl\)∼𝒟​\[log⁡π​\(yw\|x\)−log⁡π​\(yl\|x\)\]\\gamma\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[\\delta\_\{\\pi\}\]=\\gamma\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[\\log\\pi\(y\_\{w\}\|x\)\-\\log\\pi\(y\_\{l\}\|x\)\]\(53\)
For each promptxx, collecting all preference pairs and responses:

=γ​𝔼x​\[∑\(yw,yl\)∈𝒫​\(x\)p​\(yw,yl\|x\)​\(log⁡π​\(yw\|x\)−log⁡π​\(yl\|x\)\)\]=\\gamma\\mathbb\{E\}\_\{x\}\\left\[\\sum\_\{\(y\_\{w\},y\_\{l\}\)\\in\\mathcal\{P\}\(x\)\}p\(y\_\{w\},y\_\{l\}\|x\)\(\\log\\pi\(y\_\{w\}\|x\)\-\\log\\pi\(y\_\{l\}\|x\)\)\\right\]\(54\)
This can be rewritten as an expectation over responses by defining:

c​\(x,y\)=γ​∑\(yw,yl\)∈𝒫​\(x\)p​\(yw,yl\|x\)​\(𝕀​\(y=yw\)−𝕀​\(y=yl\)\)c\(x,y\)=\\gamma\\sum\_\{\(y\_\{w\},y\_\{l\}\)\\in\\mathcal\{P\}\(x\)\}p\(y\_\{w\},y\_\{l\}\|x\)\(\\mathbb\{I\}\(y=y\_\{w\}\)\-\\mathbb\{I\}\(y=y\_\{l\}\)\)\(55\)
Then:

γ​𝔼\(x,yw,yl\)∼𝒟​\[δπ\]=𝔼x​\[∑yc​\(x,y\)​log⁡π​\(y\|x\)\]\\gamma\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[\\delta\_\{\\pi\}\]=\\mathbb\{E\}\_\{x\}\\left\[\\sum\_\{y\}c\(x,y\)\\log\\pi\(y\|x\)\\right\]\(56\)
For a fixed promptxx, the Lagrangian is:

ℒx=∑yπ​\(y\|x\)​r​\(x,y\)−β​∑yπ​\(y\|x\)​log⁡π​\(y\|x\)πref​\(y\|x\)\+∑yc​\(x,y\)​log⁡π​\(y\|x\)−λ​\(x\)​\(∑yπ​\(y\|x\)−1\)\\mathcal\{L\}\_\{x\}=\\sum\_\{y\}\\pi\(y\|x\)r\(x,y\)\-\\beta\\sum\_\{y\}\\pi\(y\|x\)\\log\\frac\{\\pi\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\+\\sum\_\{y\}c\(x,y\)\\log\\pi\(y\|x\)\-\\lambda\(x\)\\left\(\\sum\_\{y\}\\pi\(y\|x\)\-1\\right\)\(57\)
Taking the derivative with respect toπ​\(y\|x\)\\pi\(y\|x\)and setting to zero:

∂ℒx∂π​\(y\|x\)=r​\(x,y\)−β​log⁡π​\(y\|x\)πref​\(y\|x\)−β\+c​\(x,y\)π​\(y\|x\)−λ​\(x\)=0\\frac\{\\partial\\mathcal\{L\}\_\{x\}\}\{\\partial\\pi\(y\|x\)\}=r\(x,y\)\-\\beta\\log\\frac\{\\pi\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\-\\beta\+\\frac\{c\(x,y\)\}\{\\pi\(y\|x\)\}\-\\lambda\(x\)=0\(58\)
Rearranging gives Equation \([11](https://arxiv.org/html/2605.20834#S3.E11)\)\. For a preference pair, taking the difference between the conditions forywy\_\{w\}andyly\_\{l\}yields Equation \([12](https://arxiv.org/html/2605.20834#S3.E12)\)\. ∎

### D\.5Proof of Absolute Advantage Guarantee

Before providing detailed proof, we first introduce the Lemma\.

###### Lemma D\.3\(Absolute Advantage Guarantee: Sample Level\)\.

For a preference pair\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\), if the hyper\-parameterγ\\gammain CPO satisfies:

γ\>β⋅max⁡\{0,−δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)β\}1πref​\(yw\|x\)\+1πref​\(yl\|x\),\\gamma\>\\beta\\cdot\\frac\{\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\\right\\\}\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\},\(59\)then CPO guarantees absolute advantage:δπCPO∗​\(x,yw,yl\)\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\(x,y\_\{w\},y\_\{l\}\)\>0for this pair\.

Proof of Lemma[D\.3](https://arxiv.org/html/2605.20834#A4.Thmtheorem3)\(Absolute Advantage Guarantee\)

###### Proof\.

LetπCPO∗\\pi^\{\*\}\_\{\\text\{CPO\}\}denote the optimal policy for CPO, satisfying:

r​\(x,yw\)−r​\(x,yl\)=β​\(δπCPO∗−δπref\)−γ~ref​\(x,yw,yl\),r\(x,y\_\{w\}\)\-r\(x,y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\),\(60\)whereγ~ref​\(x,yw,yl\)=γ​\(1πref​\(yw\|x\)\+1πref​\(yl\|x\)\)\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)=\\gamma\\left\(\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)is the reference\-based adaptive margin\. This results in a sample\-adaptive larger margin than DPO:

δπCPO∗=δπDPO∗\+γ~ref​\(x,yw,yl\)β\.\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}=\\delta\_\{\\pi^\{\*\}\_\{\\text\{DPO\}\}\}\+\\frac\{\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\.\(61\)
Part 1 \(Margin relationship\):From Eq\.[13](https://arxiv.org/html/2605.20834#S3.E13)\(derived from Theorem[3\.8](https://arxiv.org/html/2605.20834#S3.Thmtheorem8)withc​\(x,yw\)=γc\(x,y\_\{w\}\)=\\gammaandc​\(x,yl\)=−γc\(x,y\_\{l\}\)=\-\\gamma\), we have:

r​\(yw\)−r​\(yl\)=β​\(δπ∗−δπref\)−γ​\(1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)\)r\(y\_\{w\}\)\-r\(y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\gamma\\left\(\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\right\)\(62\)
In CPO, we approximate the optimal policy probabilities in the margin term with the reference policy probabilities \(as justified in Eq\. \([17](https://arxiv.org/html/2605.20834#S3.E17)\) and PropositionLABEL:prop:stationary\_cpo\):

γ​\(1π∗​\(yw\|x\)\+1π∗​\(yl\|x\)\)≈γ​\(1πref​\(yw\|x\)\+1πref​\(yl\|x\)\)=γ~ref​\(x,yw,yl\)\\gamma\\left\(\\frac\{1\}\{\\pi^\{\*\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi^\{\*\}\(y\_\{l\}\|x\)\}\\right\)\\approx\\gamma\\left\(\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)=\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\(63\)
Therefore:

r​\(yw\)−r​\(yl\)=β​\(δπCPO∗−δπref\)−γ~ref​\(x,yw,yl\)r\(y\_\{w\}\)\-r\(y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\(64\)
For standard DPO \(γ=0\\gamma=0\):

r​\(yw\)−r​\(yl\)=β​\(δπDPO∗−δπref\)r\(y\_\{w\}\)\-r\(y\_\{l\}\)=\\beta\(\\delta\_\{\\pi^\{\*\}\_\{\\text\{DPO\}\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\(65\)
Rearranging the CPO equation:

δπCPO∗=δπref\+r​\(yw\)−r​\(yl\)β\+γ~ref​\(x,yw,yl\)β\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{r\(y\_\{w\}\)\-r\(y\_\{l\}\)\}\{\\beta\}\+\\frac\{\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\(66\)
Comparing with DPO:

δπDPO∗=δπref\+r​\(yw\)−r​\(yl\)β\\delta\_\{\\pi^\{\*\}\_\{\\text\{DPO\}\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{r\(y\_\{w\}\)\-r\(y\_\{l\}\)\}\{\\beta\}\(67\)
Subtracting yields:

δπCPO∗=δπDPO∗\+γ~ref​\(x,yw,yl\)β\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}=\\delta\_\{\\pi^\{\*\}\_\{\\text\{DPO\}\}\}\+\\frac\{\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\(68\)
Part 2 \(Absolute advantage\):To ensureδπCPO∗​\(x,yw,yl\)\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\(x,y\_\{w\},y\_\{l\}\)\>0, we require:

δπref​\(x,yw,yl\)\+Δ​r∗​\(x,yw,yl\)β\+γ~ref​\(x,yw,yl\)β\>0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\+\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\+\\frac\{\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\>0\(69\)
Rearranging:

γ~ref​\(x,yw,yl\)β\>−δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)β\\frac\{\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\>\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\(70\)
γ~ref​\(x,yw,yl\)\>−β​δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\(x,y\_\{w\},y\_\{l\}\)\>\-\\beta\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\(71\)
Substituting the definition ofγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}:

γ​\(1πref​\(yw\|x\)\+1πref​\(yl\|x\)\)\>−β​δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)\\gamma\\left\(\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)\>\-\\beta\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\(72\)
Solving forγ\\gamma:

γ\>−β​δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)1πref​\(yw\|x\)\+1πref​\(yl\|x\)\\gamma\>\\frac\{\-\\beta\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\}\(73\)
When−δπref−Δ​r∗β≤0\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\\leq 0\(i\.e\., standard DPO already satisfies absolute advantage for this pair\), anyγ≥0\\gamma\\geq 0suffices\. Otherwise, we needγ\\gammato compensate for the deficit\. This is captured by:

γ\>β⋅max⁡\{0,−δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)β\}1πref​\(yw\|x\)\+1πref​\(yl\|x\)\\gamma\>\\beta\\cdot\\frac\{\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\\right\\\}\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\}\(74\)∎

Proof of Theorem[3\.9](https://arxiv.org/html/2605.20834#S3.Thmtheorem9)\(Absolute Advantage Guarantee\), given the Lemma above\.

###### Proof\.

For each\(x,yw,yl\)∈𝒟\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}, Theorem[D\.3](https://arxiv.org/html/2605.20834#A4.Thmtheorem3)requires:

γ\>β⋅max⁡\{0,−δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)β\}1πref​\(yw\|x\)\+1πref​\(yl\|x\)\\gamma\>\\beta\\cdot\\frac\{\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\\right\\\}\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\}\(76\)
To satisfy this condition for all preference pairs simultaneously, we take the maximum over the dataset:

γ∗=max\(x,yw,yl\)∈𝒟⁡β⋅max⁡\{0,−δπref​\(x,yw,yl\)−Δ​r∗​\(x,yw,yl\)β\}1πref​\(yw\|x\)\+1πref​\(yl\|x\)\\gamma^\{\*\}=\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\beta\\cdot\\frac\{\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{\\Delta r^\{\*\}\(x,y\_\{w\},y\_\{l\}\)\}\{\\beta\}\\right\\\}\}\{\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\+\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\}\(77\)
Anyγ≥γ∗\\gamma\\geq\\gamma^\{\*\}then satisfies the condition for all pairs, guaranteeing absolute advantage across the entire dataset\. ∎

### D\.6Proof of Avoiding Pathological Convergence in CPO

Proof of Theorem[3\.10](https://arxiv.org/html/2605.20834#S3.Thmtheorem10)\(CPO Avoids Pathological Convergence\)\.

###### Proof\.

Recall𝒰=\{π:δπ​<0​and​δπ\>​δπref\}\\mathcal\{U\}=\\\{\\pi:\\delta\_\{\\pi\}<0\\text\{ and \}\\delta\_\{\\pi\}\>\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\\}\.

By Theorem[D\.3](https://arxiv.org/html/2605.20834#A4.Thmtheorem3), whenγ≥γ∗\\gamma\\geq\\gamma^\{\*\}:

δπCPO∗\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{CPO\}\}\}\>0\(78\)
ThereforeπCPO∗∉𝒰\\pi^\{\*\}\_\{\\text\{CPO\}\}\\notin\\mathcal\{U\}\. Under AssumptionLABEL:assump:smooth\_bounded\(Lipschitz continuous gradients, bounded parameter domain\), the CPO lossℒCPO\\mathcal\{L\}\_\{\\text\{CPO\}\}is convex in the log\-probability space, and gradient descent converges to the global optimumπCPO∗\\pi^\{\*\}\_\{\\text\{CPO\}\}\. SinceπCPO∗∉𝒰\\pi^\{\*\}\_\{\\text\{CPO\}\}\\notin\\mathcal\{U\}, the optimization avoids𝒰\\mathcal\{U\}entirely\. ∎

### D\.7Derivation of CPO gradients

Derivation of CPO gradients, i\.e\., Eq\.[21](https://arxiv.org/html/2605.20834#S3.E21)\.

###### Proof\.

Letz=β​\(δπθ−δπref\)−γ~refz=\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\tilde\{\\gamma\}\_\{\\text\{ref\}\}\. Then:

ℒCPO=−𝔼​\[log⁡σ​\(z\)\]\\mathcal\{L\}\_\{\\text\{CPO\}\}=\-\\mathbb\{E\}\[\\log\\sigma\(z\)\]\(79\)
Taking the gradient with respect toθ\\theta:

∇θℒCPO=−𝔼​\[σ′​\(z\)σ​\(z\)⋅∇θz\]\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=\-\\mathbb\{E\}\\left\[\\frac\{\\sigma^\{\\prime\}\(z\)\}\{\\sigma\(z\)\}\\cdot\\nabla\_\{\\theta\}z\\right\]\(80\)
Sinceγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\}is constant with respect toθ\\theta:

∇θz=β⋅∇θ\(log⁡πθ​\(yw\|x\)−log⁡πθ​\(yl\|x\)\)=β⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\\nabla\_\{\\theta\}z=\\beta\\cdot\\nabla\_\{\\theta\}\(\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)=\\beta\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\(81\)
Using the identityσ′​\(z\)/σ​\(z\)=1−σ​\(z\)=σ​\(−z\)\\sigma^\{\\prime\}\(z\)/\\sigma\(z\)=1\-\\sigma\(z\)=\\sigma\(\-z\):

∇θℒCPO=−𝔼​\[σ​\(−z\)⋅β⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\]\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=\-\\mathbb\{E\}\\left\[\\sigma\(\-z\)\\cdot\\beta\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\\right\]\(82\)
Substituting−z=−β​\(δπθ−δπref\)\+γ~ref=β​\(δπref−δπθ\)\+γ~ref\-z=\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\+\\tilde\{\\gamma\}\_\{\\text\{ref\}\}=\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\+\\tilde\{\\gamma\}\_\{\\text\{ref\}\}, we obtain the stated form\. ∎

## Appendix EE\-CPOC: Conservative Explicitly Constrained Preference Optimization

This section provides a complete treatment of E\-CPOC \(Conservative Explicitly Constrained Preference Optimization\), our primary method that requires no knowledge of reward differences\. We present the theoretical foundation, algorithm, and guarantees\.

### E\.1Motivation and Derivation

The general adaptive margin loss derived in SectionLABEL:app:cpoedrequires reward differencesΔ​r=r​\(yw\)−r​\(yl\)\\Delta r=r\(y\_\{w\}\)\-r\(y\_\{l\}\), which are typically unknown\. To derive a reward\-model\-free method with provable guarantees, we exploit aconservative upper boundbased on worst\-case analysis\.

###### Proposition E\.1\(Monotonicity ofΦ\\PhiinΔ​r\\Delta r\)\.

The adaptive margin functionΦ​\(δπref,Δ​r;γ\)=max⁡\{0,γ−δπref−Δ​r/β\}\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r;\\gamma\)=\\max\\\{0,\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\Delta r/\\beta\\\}is monotone non\-increasing inΔ​r\\Delta r\.

###### Proof\.

For fixedδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}andγ\\gamma, compute the derivative:

∂Φ∂\(Δ​r\)=\{−1/βif​γ−δπref−Δ​r/β\>00if​γ−δπref−Δ​r/β≤0\\frac\{\\partial\\Phi\}\{\\partial\(\\Delta r\)\}=\\begin\{cases\}\-1/\\beta&\\text\{if \}\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\Delta r/\\beta\>0\\\\ 0&\\text\{if \}\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\Delta r/\\beta\\leq 0\\end\{cases\}\(83\)Therefore,Φ\\Phiis monotone non\-increasing inΔ​r\\Delta r\. ∎

###### Corollary E\.2\(Conservative Bound\)\.

Since human preference data satisfiesΔ​r∗\>0\\Delta r^\{\*\}\>0by the Bradley\-Terry model, andΦ\\Phiis monotone decreasing inΔ​r\\Delta r, the maximum value ofΦ\\Phiover all validΔ​r∗\>0\\Delta r^\{\*\}\>0is achieved in the limitΔ​r→0\+\\Delta r\\to 0^\{\+\}:

Φcons​\(δπref\):=Φ​\(δπref,0\)=max⁡\{0,γ−δπref\}≥Φ​\(δπref,Δ​r∗\)∀Δ​r∗\>0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\):=\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\)=\\max\\\{0,\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\\}\\geq\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r^\{\*\}\)\\quad\\forall\\Delta r^\{\*\}\>0\(84\)

This motivates the conservative adaptive margin function:

Φcons​\(δπref;γ,τ\)=1τ​log⁡\(1\+exp⁡\(τ​\(γ−δπref\)\)\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\};\\gamma,\\tau\)=\\frac\{1\}\{\\tau\}\\log\\left\(1\+\\exp\\left\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)\\right\)\(85\)

### E\.2E\-CPOC Algorithm

Algorithm 2E\-CPOC: Conservative Explicitly Constrained Preference Optimization \(No Reward Model\)0:Preference dataset

𝒟=\{\(x\(i\),yw\(i\),yl\(i\)\)\}i=1N\\mathcal\{D\}=\\\{\(x^\{\(i\)\},y\_\{w\}^\{\(i\)\},y\_\{l\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{N\}
0:Reference policy

πref\\pi\_\{\\text\{ref\}\}
0:Hyperparameters:

β\>0\\beta\>0\(temperature\),

γ\>0\\gamma\>0\(target margin\),

τ\>0\\tau\>0\(smoothness\)

0:Learning rate

η\\eta
1:Initialize policy parameters

θ\\theta\(e\.g\., from

πref\\pi\_\{\\text\{ref\}\}\)

2:Precompute:For each

\(x,yw,yl\)∈𝒟\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}:

3:

δref\(i\)←log⁡πref​\(yw\|x\)−log⁡πref​\(yl\|x\)\\delta\_\{\\text\{ref\}\}^\{\(i\)\}\\leftarrow\\log\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\-\\log\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)
4:

Φ\(i\)←1τ​log⁡\(1\+exp⁡\(τ​\(γ−δref\(i\)\)\)\)\\Phi^\{\(i\)\}\\leftarrow\\frac\{1\}\{\\tau\}\\log\(1\+\\exp\(\\tau\(\\gamma\-\\delta\_\{\\text\{ref\}\}^\{\(i\)\}\)\)\)⊳\\trianglerightConservative margin

5:

Ψ\(i\)←β​Φ\(i\)\\Psi^\{\(i\)\}\\leftarrow\\beta\\Phi^\{\(i\)\}⊳\\trianglerightAdaptive margin

6:foreach training iterationdo

7:Sample batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}
8:foreach

\(x,yw,yl\)∈ℬ\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{B\}do

9:

δθ←log⁡πθ​\(yw\|x\)−log⁡πθ​\(yl\|x\)\\delta\_\{\\theta\}\\leftarrow\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)
10:

logits←β​\(δθ−δref\(i\)\)−Ψ\(i\)\\text\{logits\}\\leftarrow\\beta\(\\delta\_\{\\theta\}\-\\delta\_\{\\text\{ref\}\}^\{\(i\)\}\)\-\\Psi^\{\(i\)\}
11:

ℓ←−log⁡σ​\(logits\)\\ell\\leftarrow\-\\log\\sigma\(\\text\{logits\}\)
12:endfor

13:Update:

θ←θ−η​∇θ1\|ℬ\|​∑ℓ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\\ell
14:endfor

15:Return:

πθ\\pi\_\{\\theta\}

### E\.3Theoretical Guarantees

###### Theorem E\.3\(E\-CPOC Absolute Advantage Guarantee\)\.

For any preference pair\(x,yw,yl\)\(x,y\_\{w\},y\_\{l\}\)with true reward differenceΔ​r∗\>0\\Delta r^\{\*\}\>0, the E\-CPOC optimal policy satisfies:

δπcons∗=δπref\+Δ​r∗β\+Φcons​\(δπref;γ,τ\)\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\};\\gamma,\\tau\)\(86\)
\(1\) Upper bound property:

δπcons∗≥δπ∗​\(Δ​r∗\)\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}\\geq\\delta\_\{\\pi^\{\*\}\}\(\\Delta r^\{\*\}\)\(87\)
\(2\) Absolute advantage:Choosing

γ≥γcons∗:=max\(x,yw,yl\)∈𝒟⁡\{−δπref​\(x,yw,yl\)\}\\gamma\\geq\\gamma^\{\*\}\_\{\\text\{cons\}\}:=\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\\{\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\\\}\(88\)guaranteesδπcons∗≥γ\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}\\geq\\gamma\>0for all preference pairs in𝒟\\mathcal\{D\}\.

\(3\) Sample\-adaptive behavior:

- •Difficult samples \(δπref≪0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll 0\):Φcons​\(δπref\)≈γ−δπref\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(large correction\)
- •Easy samples \(δπref≫γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg\\gamma\):Φcons​\(δπref\)≈0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx 0\(minimal correction\)
- •Neutral samples \(δπref≈γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\approx\\gamma\):Φcons​\(δπref\)≈log⁡2τ\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\frac\{\\log 2\}\{\\tau\}\(smooth transition\)

### E\.4Proof of Theorem[E\.3](https://arxiv.org/html/2605.20834#A5.Thmtheorem3)

###### Proof\.

From Theorem[3\.12](https://arxiv.org/html/2605.20834#S3.Thmtheorem12), the optimal policy for constrained RLHF satisfies:

δπ∗=δπref\+Δ​rβ\+Φ​\(δπref,Δ​r;γ,τ\)\\delta\_\{\\pi^\{\*\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r\}\{\\beta\}\+\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r;\\gamma,\\tau\)\(89\)
Step 1: Upper bound property

By Proposition[E\.1](https://arxiv.org/html/2605.20834#A5.Thmtheorem1)and Corollary[E\.2](https://arxiv.org/html/2605.20834#A5.Thmtheorem2), using the conservative marginΦcons​\(δπref\):=Φ​\(δπref,0\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\):=\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\):

δπcons∗\\displaystyle\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}=δπref\+Δ​r∗β\+Φcons​\(δπref\)\\displaystyle=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\(90\)=δπref\+Δ​r∗β\+Φ​\(δπref,0\)\\displaystyle=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\+\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\)\(91\)≥δπref\+Δ​r∗β\+Φ​\(δπref,Δ​r∗\)\\displaystyle\\geq\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\+\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},\\Delta r^\{\*\}\)\(92\)=δπ∗​\(Δ​r∗\)\\displaystyle=\\delta\_\{\\pi^\{\*\}\}\(\\Delta r^\{\*\}\)\(93\)
This proves property \(1\)\.

Step 2: Absolute advantage guarantee

To ensureδπcons∗\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}\>0, we need:

δπref\+Δ​r∗β\+Φcons​\(δπref\)\>0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\>0\(94\)
In the worst case whereΔ​r∗→0\+\\Delta r^\{\*\}\\to 0^\{\+\}, this reduces to:

δπref\+Φcons​\(δπref\)\>0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\>0\(95\)
Case 1: Ifγ−δπref\>0\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\>0\(constraint is active\), then:

Φcons​\(δπref\)=γ−δπref\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(96\)
Therefore:

δπcons∗≈δπref\+\(γ−δπref\)=γ\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}\\approx\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\gamma\(97\)
Case 2: Ifγ−δπref≤0\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\leq 0\(constraint is not active\), then:

Φcons​\(δπref\)=0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=0\(98\)
and

δπcons∗=δπref\+Δ​r∗β\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{\\Delta r^\{\*\}\}\{\\beta\}\(99\)
For this case, we needδπref≥0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\geq 0for absolute advantage\. This is guaranteed whenγ≥−δπref\\gamma\\geq\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}for all pairs\.

Combining both cases, choosingγ≥γcons∗:=max\(x,yw,yl\)∈𝒟⁡\{−δπref​\(x,yw,yl\)\}\\gamma\\geq\\gamma^\{\*\}\_\{\\text\{cons\}\}:=\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\\{\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\\\}ensures:

δπcons∗≥min⁡\{γ,δπref\+Δ​r∗/β\}≥min⁡\{γ,γ\}=γ\>0\\delta\_\{\\pi^\{\*\}\_\{\\text\{cons\}\}\}\\geq\\min\\\{\\gamma,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Delta r^\{\*\}/\\beta\\\}\\geq\\min\\\{\\gamma,\\gamma\\\}=\\gamma\>0\(100\)
This proves property \(2\)\.

Step 3: Sample\-adaptive behavior

From the softplus formulationΦcons​\(δπref\)=1τ​log⁡\(1\+exp⁡\(τ​\(γ−δπref\)\)\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\frac\{1\}\{\\tau\}\\log\(1\+\\exp\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\):

- •Whenδπref≪0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll 0\(difficult samples\):γ−δπref≫0\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg 0, so: Φcons​\(δπref\)≈1τ⋅τ​\(γ−δπref\)=γ−δπref\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\frac\{1\}\{\\tau\}\\cdot\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(101\)
- •Whenδπref≫γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg\\gamma\(easy samples\):γ−δπref≪0\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll 0, so: Φcons​\(δπref\)≈1τ​log⁡\(1\+exp⁡\(τ​\(γ−δπref\)\)\)≈1τ​exp⁡\(τ​\(γ−δπref\)\)→0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\frac\{1\}\{\\tau\}\\log\(1\+\\exp\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\)\\approx\\frac\{1\}\{\\tau\}\\exp\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\\to 0\(102\)
- •Whenδπref≈γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\approx\\gamma\(neutral samples\):γ−δπref≈0\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\approx 0, so: Φcons​\(δπref\)≈1τ​log⁡\(1\+1\)=log⁡2τ\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\frac\{1\}\{\\tau\}\\log\(1\+1\)=\\frac\{\\log 2\}\{\\tau\}\(103\)

This proves property \(3\)\. ∎

### E\.5Properties of E\-CPOC

###### Proposition E\.4\(Properties ofΦcons\\Phi\_\{\\text\{cons\}\}\)\.

The conservative adaptive margin functionΦcons​\(δπref;γ,τ\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\};\\gamma,\\tau\)satisfies:

1. 1\.Non\-negativity:Φcons​\(δπref\)≥0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\geq 0for allδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}
2. 2\.Monotonicity inδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}:∂Φcons∂δπref=−σ​\(τ​\(γ−δπref\)\)<0\\frac\{\\partial\\Phi\_\{\\text\{cons\}\}\}\{\\partial\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\}=\-\\sigma\\left\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)<0
3. 3\.Boundary behavior: limδπref→−∞Φcons\\displaystyle\\lim\_\{\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\to\-\\infty\}\\Phi\_\{\\text\{cons\}\}=γ−δπref\(strong compensation\)\\displaystyle=\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\quad\\text\{\(strong compensation\)\}\(104\)limδπref→\+∞Φcons\\displaystyle\\lim\_\{\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\to\+\\infty\}\\Phi\_\{\\text\{cons\}\}=0\(no compensation needed\)\\displaystyle=0\\quad\\text\{\(no compensation needed\)\}\(105\)
4. 4\.Interpretability:Φcons\\Phi\_\{\\text\{cons\}\}measures the*degree of constraint violation*, providing exactly the margin needed to satisfy the constraint in the worst\-case scenario\.

###### Proof\.

Properties \(1\) and \(3\) follow from the properties of softplus\. For \(2\), compute:

∂Φcons∂δπref=−1τ⋅τ​exp⁡\(τ​\(γ−δπref\)\)1\+exp⁡\(τ​\(γ−δπref\)\)=−σ​\(τ​\(γ−δπref\)\)\\frac\{\\partial\\Phi\_\{\\text\{cons\}\}\}\{\\partial\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\}=\-\\frac\{1\}\{\\tau\}\\cdot\\frac\{\\tau\\exp\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\}\{1\+\\exp\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\}=\-\\sigma\(\\tau\(\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\)\(106\)Sinceσ​\(z\)∈\(0,1\)\\sigma\(z\)\\in\(0,1\)for allzz, the derivative is strictly negative\. ∎

### E\.6Gradient Analysis and Sample Weighting

###### Proposition E\.5\(E\-CPOC Gradient\)\.

The gradient of the E\-CPOC loss is:

∇θℒE\-CPOC=−𝔼​\[β⋅w​\(δπθ,δπref\)⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\]\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{E\-CPOC\}\}=\-\\mathbb\{E\}\\left\[\\beta\\cdot w\(\\delta\_\{\\pi\_\{\\theta\}\},\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\\right\]\(107\)where the weight function is:

w​\(δπθ,δπref\)=σ​\(β​\(δπref−δπθ\)\+Ψcons​\(δπref\)\)w\(\\delta\_\{\\pi\_\{\\theta\}\},\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\sigma\\left\(\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\+\\Psi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)\(108\)

###### Proof\.

Definez=β​\(δπθ−δπref\)−Ψcons​\(δπref\)z=\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\Psi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\. Then:

∇θℒE\-CPOC=−𝔼​\[σ′​\(z\)σ​\(z\)⋅β⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\]\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{E\-CPOC\}\}=\-\\mathbb\{E\}\\left\[\\frac\{\\sigma^\{\\prime\}\(z\)\}\{\\sigma\(z\)\}\\cdot\\beta\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\\right\]\(109\)
Usingσ′​\(z\)/σ​\(z\)=1−σ​\(z\)=σ​\(−z\)\\sigma^\{\\prime\}\(z\)/\\sigma\(z\)=1\-\\sigma\(z\)=\\sigma\(\-z\):

=−𝔼​\[β⋅σ​\(−z\)⋅\(∇θlog⁡πθ​\(yw\|x\)−∇θlog⁡πθ​\(yl\|x\)\)\]=\-\\mathbb\{E\}\\left\[\\beta\\cdot\\sigma\(\-z\)\\cdot\(\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\-\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\)\\right\]\(110\)
Since−z=−β​\(δπθ−δπref\)\+Ψcons​\(δπref\)=β​\(δπref−δπθ\)\+Ψcons​\(δπref\)\-z=\-\\beta\(\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\+\\Psi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\+\\Psi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\), we obtain the stated form\. ∎

###### Proposition E\.6\(Adaptive Gradient Weighting viaΦcons\\Phi\_\{\\text\{cons\}\}\)\.

The E\-CPOC weight functionw​\(δπθ,δπref\)w\(\\delta\_\{\\pi\_\{\\theta\}\},\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)implements automatic sample difficulty weighting through the adaptive margin functionΦcons\\Phi\_\{\\text\{cons\}\}:

1. 1\.Difficult samples\(δπref≪γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll\\gamma\): Φcons​\(δπref\)≈γ−δπref≫0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg 0\(111\)The large positiveΦcons\\Phi\_\{\\text\{cons\}\}significantly increases the weight, providing stronger gradient signal to pushδπθ\\delta\_\{\\pi\_\{\\theta\}\}upward\.
2. 2\.Easy samples\(δπref≫γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg\\gamma\): Φcons​\(δπref\)≈0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx 0\(112\)The weight reduces to standard DPO, avoiding unnecessary emphasis on already well\-aligned samples\.
3. 3\.Neutral samples\(δπref≈γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\approx\\gamma\): Φcons​\(δπref\)≈log⁡2τ\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\frac\{\\log 2\}\{\\tau\}\(113\)Provides smooth transition between difficult and easy regimes\.

###### Proof\.

The weight function is:

w​\(δπθ,δπref\)=σ​\(β​\(δπref−δπθ\)\+β​Φcons​\(δπref\)\)w\(\\delta\_\{\\pi\_\{\\theta\}\},\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\sigma\\left\(\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\+\\beta\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\right\)\(114\)
The termΨcons​\(δπref\)=β​Φcons​\(δπref\)\\Psi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\beta\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)acts as an additive boost to the logit\. Whenδπref≪γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll\\gamma,Φcons\\Phi\_\{\\text\{cons\}\}is large, increasing the sigmoid input and thus the weight\. Whenδπref≫γ\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg\\gamma,Φcons≈0\\Phi\_\{\\text\{cons\}\}\\approx 0, and the weight behaves like standard DPO\. The smooth transition follows from the softplus formulation ofΦcons\\Phi\_\{\\text\{cons\}\}\. ∎

### E\.7Comparison with CPO

## Appendix FGeneralization to Multiple Appearances

###### Proposition F\.1\(Generalization to Multiple Appearances\)\.

For a responseyyappearing in multiple preference pairs for promptxx, define the aggregated margin coefficient:

c​\(x,y\)=γ​∑\(yw,yl\)∈𝒫​\(x\)p​\(yw,yl\|x\)​\(𝕀​\(y=yw\)−𝕀​\(y=yl\)\)c\(x,y\)=\\gamma\\sum\_\{\(y\_\{w\},y\_\{l\}\)\\in\\mathcal\{P\}\(x\)\}p\(y\_\{w\},y\_\{l\}\|x\)\(\\mathbb\{I\}\(y=y\_\{w\}\)\-\\mathbb\{I\}\(y=y\_\{l\}\)\)\(115\)where𝒫​\(x\)\\mathcal\{P\}\(x\)denotes all preference pairs for promptxx\.

Then the optimality condition \(e\.g\., Theorems[3\.8](https://arxiv.org/html/2605.20834#S3.Thmtheorem8)\) remain valid withc​\(x,y\)c\(x,y\)replacing the simplified coefficients±γ\\pm\\gamma\. Furthermore, all approximation error bounds \(PropositionLABEL:prop:approximation\_error\) and convergence guarantees \(PropositionLABEL:prop:cpo\_convergence\) remain unchanged, as they depend only on the stationarity of the objective and the KL regularization strengthβ\\beta\.

###### Proof\.

The margin term in the Constrained RLHF objective is linear in preference pairs:

γ​𝔼\(x,yw,yl\)∼𝒟​\[δπ\]=𝔼x​\[∑yc​\(x,y\)​log⁡π​\(y\|x\)\]\\gamma\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\[\\delta\_\{\\pi\}\]=\\mathbb\{E\}\_\{x\}\\left\[\\sum\_\{y\}c\(x,y\)\\log\\pi\(y\|x\)\\right\]\(116\)
The first\-order optimality condition remains:

β​log⁡π∗​\(y\|x\)πref​\(y\|x\)=r​\(x,y\)\+c​\(x,y\)π∗​\(y\|x\)−β−λ​\(x\)\\beta\\log\\frac\{\\pi^\{\*\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}=r\(x,y\)\+\\frac\{c\(x,y\)\}\{\\pi^\{\*\}\(y\|x\)\}\-\\beta\-\\lambda\(x\)\(117\)
For CPO, E\-CPOC, and A\-CPO, the key insight is that the margin termsγ~ref\\tilde\{\\gamma\}\_\{\\text\{ref\}\},Ψcons\\Psi\_\{\\text\{cons\}\}, andw~ref\\tilde\{w\}\_\{\\text\{ref\}\}are precomputed based onπref\\pi\_\{\\text\{ref\}\}and remain stationary during optimization\. When a response appears in multiple pairs, the algorithm naturally aggregates the gradients through mini\-batch computation:

∇θℒ=∑\(x,yw,yl\)∈𝒟∇θℓ​\(x,yw,yl\)\\nabla\_\{\\theta\}\\mathcal\{L\}=\\sum\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\nabla\_\{\\theta\}\\ell\(x,y\_\{w\},y\_\{l\}\)\(118\)
The approximation error bounds depend on‖π∗−πref‖\\\|\\pi^\{\*\}\-\\pi\_\{\\text\{ref\}\}\\\|, which is controlled byβ\\betaregardless of the preference structure\. Specifically, for any responseyy:

\|1π∗​\(y\|x\)−1πref​\(y\|x\)\|≤1min\{π∗\(y\|x\),πref\(y\|x\)\}2⋅\|π∗\(y\|x\)−πref\(y\|x\)\|\\left\|\\frac\{1\}\{\\pi^\{\*\}\(y\|x\)\}\-\\frac\{1\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\\right\|\\leq\\frac\{1\}\{\\min\\\{\\pi^\{\*\}\(y\|x\),\\pi\_\{\\text\{ref\}\}\(y\|x\)\\\}^\{2\}\}\\cdot\|\\pi^\{\*\}\(y\|x\)\-\\pi\_\{\\text\{ref\}\}\(y\|x\)\|\(119\)
This bound is independent of how many timesyyappears in preference pairs\. Therefore, all theoretical guarantees \(approximation accuracy, convergence, absolute advantage\) transfer to the general case with multiple appearances\. ∎

## Appendix GPreference Learning as Reranking

### G\.1DPO as Soft Margin Ranking

Proof of Proposition[4\.1](https://arxiv.org/html/2605.20834#S4.Thmtheorem1)\(DPO as Soft Margin Ranking\)\.

###### Proof\.

Definez=δπθ−δπrefz=\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\. The DPO loss can be written as:

ℒDPO=log⁡\(1\+exp⁡\(−β​z\)\)=softplus​\(−β​z\)\\mathcal\{L\}\_\{\\text\{DPO\}\}=\\log\(1\+\\exp\(\-\\beta z\)\)=\\text\{softplus\}\(\-\\beta z\)\(120\)
We analyze the asymptotic behavior for three cases:

Case 1 \(z\>0z\>0, margin satisfied\):Whenδπθ\>δπref\\delta\_\{\\pi\_\{\\theta\}\}\>\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, asβ→∞\\beta\\to\\inftywithz\>0z\>0fixed, we haveexp⁡\(−β​z\)→0\\exp\(\-\\beta z\)\\to 0\. Using the Taylor expansionlog⁡\(1\+ϵ\)≈ϵ\\log\(1\+\\epsilon\)\\approx\\epsilonfor smallϵ\\epsilon:

ℒDPO≈exp⁡\(−β​z\)→0\\mathcal\{L\}\_\{\\text\{DPO\}\}\\approx\\exp\(\-\\beta z\)\\to 0\(121\)Therefore:

limβ→∞1β​ℒDPO=limβ→∞exp⁡\(−β​z\)β=0=max⁡\(0,−z\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\\lim\_\{\\beta\\to\\infty\}\\frac\{\\exp\(\-\\beta z\)\}\{\\beta\}=0=\\max\(0,\-z\)\(122\)
Case 2 \(z<0z<0, margin violated\):Whenδπθ<δπref\\delta\_\{\\pi\_\{\\theta\}\}<\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, we rewrite:

ℒDPO\\displaystyle\\mathcal\{L\}\_\{\\text\{DPO\}\}=log⁡\(1\+exp⁡\(−β​z\)\)\\displaystyle=\\log\(1\+\\exp\(\-\\beta z\)\)\(123\)=log⁡\(exp⁡\(−β​z\)​\(1\+exp⁡\(β​z\)\)\)\\displaystyle=\\log\(\\exp\(\-\\beta z\)\(1\+\\exp\(\\beta z\)\)\)\(124\)=−β​z\+log⁡\(1\+exp⁡\(β​z\)\)\\displaystyle=\-\\beta z\+\\log\(1\+\\exp\(\\beta z\)\)\(125\)Asβ→∞\\beta\\to\\inftywithz<0z<0fixed, we haveexp⁡\(β​z\)→0\\exp\(\\beta z\)\\to 0\(sinceβ​z<0\\beta z<0\)\. Thus:

log⁡\(1\+exp⁡\(β​z\)\)≈exp⁡\(β​z\)→0\\log\(1\+\\exp\(\\beta z\)\)\\approx\\exp\(\\beta z\)\\to 0\(126\)Therefore:

ℒDPO≈−β​z=β​\(δπref−δπθ\)\\mathcal\{L\}\_\{\\text\{DPO\}\}\\approx\-\\beta z=\\beta\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\(127\)and:

limβ→∞1β​ℒDPO=−z=δπref−δπθ=max⁡\(0,−z\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\-z=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}=\\max\(0,\-z\)\(128\)
Case 3 \(z=0z=0, boundary\):Whenδπθ=δπref\\delta\_\{\\pi\_\{\\theta\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}:

ℒDPO=log⁡\(1\+e0\)=log⁡2\\mathcal\{L\}\_\{\\text\{DPO\}\}=\\log\(1\+e^\{0\}\)=\\log 2\(129\)Therefore:

limβ→∞1β​ℒDPO=limβ→∞log⁡2β=0=max⁡\(0,0\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\\lim\_\{\\beta\\to\\infty\}\\frac\{\\log 2\}\{\\beta\}=0=\\max\(0,0\)\(130\)
Combining all cases:

limβ→∞1β​ℒDPO=\{0if​z\>0−zif​z<00if​z=0=max⁡\(0,−z\)=max⁡\(0,δπref−δπθ\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{DPO\}\}=\\begin\{cases\}0&\\text\{if \}z\>0\\\\ \-z&\\text\{if \}z<0\\\\ 0&\\text\{if \}z=0\\end\{cases\}=\\max\(0,\-z\)=\\max\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\delta\_\{\\pi\_\{\\theta\}\}\)\(131\)∎

### G\.2Proof of CPO as Corrected Soft Margin Ranking

Proof of Theorem[4\.2](https://arxiv.org/html/2605.20834#S4.Thmtheorem2)\(CPO as Corrected Soft Margin Ranking\)\.

###### Proof\.

Definez=δπθ−δπref−2​γ/βz=\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-2\\gamma/\\beta\. The CPO loss is:

ℒCPO=log⁡\(1\+exp⁡\(−β​z\)\)\\mathcal\{L\}\_\{\\text\{CPO\}\}=\\log\(1\+\\exp\(\-\\beta z\)\)\(132\)
Following the same analysis as Proposition[4\.1](https://arxiv.org/html/2605.20834#S4.Thmtheorem1):

Case 1 \(z\>0z\>0\):Asβ→∞\\beta\\to\\infty,exp⁡\(−β​z\)→0\\exp\(\-\\beta z\)\\to 0, so:

limβ→∞1β​ℒCPO=0\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=0\(133\)
Case 2 \(z<0z<0\):Asβ→∞\\beta\\to\\infty:

ℒCPO≈−β​z=β​\(δπref\+2​γβ−δπθ\)\\mathcal\{L\}\_\{\\text\{CPO\}\}\\approx\-\\beta z=\\beta\\left\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{2\\gamma\}\{\\beta\}\-\\delta\_\{\\pi\_\{\\theta\}\}\\right\)\(134\)Therefore:

limβ→∞1β​ℒCPO=δπref\+2​γβ−δπθ\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{2\\gamma\}\{\\beta\}\-\\delta\_\{\\pi\_\{\\theta\}\}\(135\)
Case 3 \(z=0z=0\):limβ→∞1β​ℒCPO=0\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=0\.

Combining:limβ→∞1β​ℒCPO=max⁡\(0,δπref\+2​γ/β−δπθ\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{CPO\}\}=\\max\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+2\\gamma/\\beta\-\\delta\_\{\\pi\_\{\\theta\}\}\)\.

By Corollary[3\.9](https://arxiv.org/html/2605.20834#S3.Thmtheorem9), whenγ≥γ∗\\gamma\\geq\\gamma^\{\*\}:

γ∗=β2​max\(x,yw,yl\)∈𝒟⁡max⁡\{0,−δπref​\(x,yw,yl\)−r∗​\(yw\)−r∗​\(yl\)β\}\\gamma^\{\*\}=\\frac\{\\beta\}\{2\}\\max\_\{\(x,y\_\{w\},y\_\{l\}\)\\in\\mathcal\{D\}\}\\max\\left\\\{0,\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\(x,y\_\{w\},y\_\{l\}\)\-\\frac\{r^\{\*\}\(y\_\{w\}\)\-r^\{\*\}\(y\_\{l\}\)\}\{\\beta\}\\right\\\}\(136\)
This ensures that for all preference pairs:

meff∗=δπref\+2​γ∗β≥max⁡\{0,δπref\+Δ​r∗/β\}\>0m^\{\*\}\_\{\\text\{eff\}\}=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\frac\{2\\gamma^\{\*\}\}\{\\beta\}\\geq\\max\\\{0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Delta r^\{\*\}/\\beta\\\}\>0\(137\)
Therefore, the optimization only stops whenδπθ\>meff∗\>0\\delta\_\{\\pi\_\{\\theta\}\}\>m^\{\*\}\_\{\\text\{eff\}\}\>0, guaranteeing absolute preference alignment\. ∎

### G\.3Proof of E\-CPOC as Adaptive Margin Ranking

Proof of Theorem[4\.3](https://arxiv.org/html/2605.20834#S4.Thmtheorem3)\(E\-CPOC as Adaptive Margin Ranking\)\.

###### Proof\.

Definez=δπθ−δπref−Φcons​\(δπref\)z=\\delta\_\{\\pi\_\{\\theta\}\}\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\-\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\), whereΦcons​\(δπref\)=Φ​\(δπref,0\)\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\Phi\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\},0\)is the conservative margin function\. Following the same analysis as Proposition[4\.1](https://arxiv.org/html/2605.20834#S4.Thmtheorem1), we obtain:

limβ→∞1β​ℒE\-CPOC=max⁡\(0,−z\)=max⁡\(0,δπref\+Φcons​\(δπref\)−δπθ\)\\lim\_\{\\beta\\to\\infty\}\\frac\{1\}\{\\beta\}\\mathcal\{L\}\_\{\\text\{E\-CPOC\}\}=\\max\(0,\-z\)=\\max\(0,\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\-\\delta\_\{\\pi\_\{\\theta\}\}\)\(138\)
The effective target margin is:

m∗​\(δπref\)=δπref\+Φcons​\(δπref\)m^\{\*\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)=\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\+\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\(139\)
By PropositionLABEL:prop:phi\_properties,Φcons​\(δπref\)≥0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\geq 0for allδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}, and:

- •Whenδπref≪0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\ll 0\(difficult\):Φcons​\(δπref\)≈γ−δπref\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx\\gamma\-\\delta\_\{\\pi\_\{\\text\{ref\}\}\}by the softplus approximation, som∗≈γ\>0m^\{\*\}\\approx\\gamma\>0
- •Whenδπref≫0\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\\gg 0\(easy\):Φcons​\(δπref\)≈0\\Phi\_\{\\text\{cons\}\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\\approx 0, som∗≈δπref\>0m^\{\*\}\\approx\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\>0

Therefore,m∗​\(δπref\)\>0m^\{\*\}\(\\delta\_\{\\pi\_\{\\text\{ref\}\}\}\)\>0for allδπref\\delta\_\{\\pi\_\{\\text\{ref\}\}\}whenγ\>0\\gamma\>0, ensuring absolute preference alignment\. ∎