Normalized Rewards for Preference Optimization

# Normalized Rewards for Preference Optimization
Source: [https://arxiv.org/html/2607.16240](https://arxiv.org/html/2607.16240)
###### Abstract

Direct Alignment Algorithms \(DAAs\) such as DPO have become a common way to post\-train and align LLMs with human preferences\. However, DAAs have been observed to over\-optimize their implicit reward model and decrease the likelihood of preferred responses\. This results in a decrease in the total likelihood assigned to responses seen in the preference dataset, potentially resulting in undesirable behavior\. To counteract this undesired side\-effect of DAAs, we examine the effect of using objectives that add a regularization term to maintain the total length\-normalized probabilities of the chosen and rejected responses\. To better understand over\-optimization, we investigate how response likelihood changes are distributed over the tokens with and without regularization\. We find that a significant portion of the likelihood changes are due to a small set of outlier tokens, which explains how DAAs improve generation quality despite decreasing the likelihoods of chosen responses\. We apply the proposed regularization to reference\-based \(DPO\) and reference\-free \(SimPO\) methods and find \(1\) improved trade\-offs between generation quality and general benchmark capability and \(2\) improvements in reward modeling across datasets\. For example, on Llama\-3\.1\-8B\-Instruct, we see both a\>20%\>20\\%relative increase in AlpacaEval2 scores and\>9%\>9\\%relative performance gains on general benchmarks\. Additionally, we find that the added regularization term effectively mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low\-likelihood tokens\.

Direct Alignment, RLHF, DPO

## 1Introduction

With the rise in interactions between large language models \(LLMs\) and humans, training LLMs to produce responses that are considered desirable by human users has become a vital step\. Such training is commonly performed with methods that learn what it means for a response to be desirable from a dataset of paired preferred and non\-preferred responses, such as Reinforcement Learning from Human Feedback \(RLHF\)\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.16240#bib.bib4)\)and Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib5)\)\. The main difference between the two approaches is RLHF relies on a two\-step training process while DPO uses only one\. RLHF first learns a reward function that assigns more value to the preferred versus non\-preferred response, and then uses the reward function to train an LLM policy\. DPO directly updates the LLM by maximizing the likelihood of the preferred versus non\-preferred responses\. Due to the simplicity and reduced computational cost of DPO, there has been a rise in the use and development of Direct Alignment Algorithms \(DAAs\), which train directly on the preferred and non\-preferred paired dataset\.

Despite their popularity, recent work has identified a critical limitation of DAAs: they can over\-optimize their implicit reward model and ultimately constrain improvements to the quality of the LLM’s generations\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib48); Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7); Huanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib6)\)\. A common, concerning manifestation of over\-optimization is likelihood displacement\(Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7)\), a phenomenon whereby the likelihoods of both the preferred and the non\-preferred responses drop simultaneously, potentially resulting in harmful behavior\. However, despite these documented issues, DAAs are nonetheless still successfully used in post\-training\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib12); Groeneveldet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib13)\)\.

Several hypotheses have been proposed to explain the causes of over\-optimization in DAAs: high embedding or textual similarity between preferred and non\-preferred responses\(Palet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib10); Tajwaret al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib9); Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7)\), or the insufficient regularization provided by the shape of the implicit reward function\(Huanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib6); Guptaet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib8)\)\. However, no single explanation satisfactorily generalizes across all DAA objectives\. For example, a common difference between DAAs is the presence versus absence of a reference model in the reward computation \(e\.g\., DPO uses a reference model for its reward, while SimPO\(Menget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib11)\)is reference\-free\), and hypotheses that explain over\-optimization for a reference\-free method do not hold or have not been applied to methods without a reference model and vice versa\. This is evidenced by the fact that existing analyses on over\-optimization\(Yoonet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib23); Huanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib6); Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7)\)focus on single instances of DAA, either reference\-based or reference\-free, but not both\. The limitations of the current hypotheses motivate our research question:how can we explain and mitigate reward over\-optimization for both reference\-based and reference\-free rewards?

In this work, we propose that reward over\-optimization—particularly likelihood displacement—stems from a lack of normalization of the implicit reward\. To counteract this, we introduce a regularization term designed to conserve the total response probability within preferred and non\-preferred response pairs\. To test the validity of our regularization term, we evaluate its impact on both reference\-based \(DPO\) and reference\-free \(SimPO\) methods\. We find that its inclusion leads to \(1\) improved trade\-offs between generation quality and general capability benchmarks, and \(2\) comparable or better reward modeling across datasets\. Furthermore, our analysis reveals new insights into the mechanics of likelihood displacement\. We discover that this phenomenon is highly concentrated, with a small subset of outlier tokens accounting for the majority of likelihood shift\. Related observations of alignment primarily focusing on a small subset of tokens have been seen in works such as\(Linet al\.,[2023a](https://arxiv.org/html/2607.16240#bib.bib50); Qiet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib51)\)\. Building on a gradient analysis fromYoonet al\.\([2025](https://arxiv.org/html/2607.16240#bib.bib23)\), we find that low\-likelihood tokens dominate the gradient, potentially causing the large shifts in outlier tokens, and our regularization term mitigates these shifts\. These findings further support the use of our regularization term, and shed more light on why DAAs, like DPO and SimPO, improve generation despite causing likelihood displacement\.

We provide a summary of the key results below:

1. 1\.We illustrate how the inability to fully account for the distribution of unseen responses leads to phenomena such as likelihood displacement\.
2. 2\.We propose a mechanistic explanation of likelihood displacement, connecting its concentration on outlier tokens to perturbations of the partition function, and show via gradient analysis that our regularizer mitigates their outsized impact, with the corrective updates concentrated on low\-likelihood tokens\.
3. 3\.We demonstrate that our modified objective improves trade\-offs between generation quality and benchmark performance\(Linet al\.,[2023b](https://arxiv.org/html/2607.16240#bib.bib39)\)for both DPO and SimPO\.

![Refer to caption](https://arxiv.org/html/2607.16240v1/x1.png)Figure 1:Illustrative example of likelihood displacement and how our regularization mitigates it\. The reference model assigns likelihoods 0\.4, 0\.3, 0\.3 to responsesAA,BB,CC, with matching probabilities since the partition function equals 1\. We then train on the single preferenceA<BA<B; the responseCCis never observed\. The example highlights the distinction between unnormalized likelihoods and normalized probabilities: under DPO, the likelihoods of bothAAandBBdecrease whileCC’s likelihood is unchanged, yet the resulting drop in the partition function spuriously inflatesCC’s probability\. To counteract this, we introduce a regularization term that conserves the total likelihood over the seen pair\(A,B\)\(A,B\)\. The preference is still learned \(AA’s likelihood falls,BB’s rises\), but the partition function \(and with it the probability of the unseen responseCC\) is preserved\.
## 2Background

In this section, we introduce how RLHF and DPO utilize preference data\.

#### RLHF\.

The preference learning component of RLHF is composed of two stages\. The first consists of training a reward modelrϕ​\(x,y\)r\_\{\\phi\}\(x,y\)parameterized byϕ\\phion pairwise comparisons to assign scores to generated responsesyyto a given promptxx\. Given a pairwise preferenceyw≻yly\_\{w\}\\succ y\_\{l\}, the reward modelrϕ​\(x,y\)r\_\{\\phi\}\(x,y\)is trained to minimize the negative log\-likelihood of the reward assignments under the Bradley\-Terry model

−log⁡p​\(yw≻yl\)=−log⁡σ​\(r​\(x,yw\)−r​\(x,yl\)\),\-\\log p\(y\_\{w\}\\succ y\_\{l\}\)=\-\\log\\sigma\(r\(x,y\_\{w\}\)\-r\(x,y\_\{l\}\)\),\(1\)whereσ\\sigmais the logistic function\. Using this reward model in the second stage, the language modelπθ\\pi\_\{\\theta\}is then trained to maximize the expected reward of its responses under a KL constraint—added to mitigate drifting too far from the original model\. The objective can be written as

𝔼x∼D,y∼πθ\(⋅\|x\)\[r\(x,y\)−β𝕂𝕃\[πθ\(⋅\|x\)\|\|πref\(⋅\|x\)\]\],\\mathbb\{E\}\_\{x\\sim D,y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\\big\[r\(x,y\)\-\\beta\\mathbb\{KL\}\[\\pi\_\{\\theta\}\(\\cdot\|x\)\|\|\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\]\\big\],\(2\)whereπθ\\pi\_\{\\theta\}is the current model,πref\\pi\_\{\\text\{ref\}\}is the reference model, andβ\\betais a hyperparameter for the KL constraint\.

#### DPO\.

DPO is a direct alignment algorithm derived from the RLHF objective which removes the need for an external reward model and directly updates the language model using preference data\. This is done by utilizing the fact that the optimal policyπ∗\\pi^\{\*\}under the RLHF objective can be written as

π∗​\(y\|x\)=1Z​\(x\)​πref​\(y\|x\)​exp⁡\(r​\(x,y\)β\),\\pi^\{\*\}\(y\|x\)=\\frac\{1\}\{Z\(x\)\}\\pi\_\{\\text\{ref\}\}\(y\|x\)\\exp\\left\(\\frac\{r\(x,y\)\}\{\\beta\}\\right\),\(3\)whereZ​\(x\)Z\(x\)is defined as

Z​\(x\)=∑yπref​\(y\|x\)​exp⁡\(r​\(x,y\)β\),Z\(x\)=\\sum\_\{y\}\\pi\_\{\\text\{ref\}\}\(y\|x\)\\exp\\left\(\\frac\{r\(x,y\)\}\{\\beta\}\\right\),\(4\)and denotes the partition function, which normalizes the output distribution ofπ∗​\(y\|x\)\\pi^\{\*\}\(y\|x\), ensuring the sum of response probabilities is 1\. From this, we can write the rewardr​\(x,y\)r\(x,y\)in terms of the optimal policy and the reference policy

r​\(x,y\)=β​log⁡π∗​\(y\|x\)πref​\(y\|x\)\+β​log⁡Z​\(x\)\.r\(x,y\)=\\beta\\log\\frac\{\\pi^\{\*\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\+\\beta\\log Z\(x\)\.\(5\)Then, by defining

rθ​\(x,y\)=β​log⁡πθ​\(y\|x\)πref​\(y\|x\)\+β​log⁡Z​\(x\),r\_\{\\theta\}\(x,y\)=\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\+\\beta\\log Z\(x\),\(6\)which corresponds to the reward model under whichπθ\\pi\_\{\\theta\}is optimal\. We note that the partition function acts as an offset that ensures the rewards are calibrated and only when the partition function is 1 do the rewards directly correspond to log\-likelihood ratios\. By maximizing the likelihood of the policy under the Bradley\-Terry model, the reward model and policy are simultaneously optimized, and under mild conditions, should result in the same optimal policy as RLHF\. Expanding the reward terms, we have that the DPO objectiveℒDPO​\(πθ\)\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\}\)is given by

𝔼\(x,yw,yl\)∼𝒟​\[−log⁡σ​\(β​log⁡πθ​\(yw\|x\)​πref​\(yl\|x\)πθ​\(yl\|x\)​πref​\(yw\|x\)\)\],\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\left\[\-\\log\\sigma\\left\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\}\\right\)\\right\],\(7\)with the partition function term cancelling out since the objective tracks the difference of rewards\.

#### SimPO\.

A variant of DPO introduced to further simplify alignment training is SimPO\. This variant removes the need for a reference model by using only the current model’s likelihood of a response as the reward, and considers length\-normalized probabilities\. We can write the SimPO rewardrθ​\(x,y\)r\_\{\\theta\}\(x,y\)for promptxxand responseyyas

rθ​\(x,y\)=β​π¯θ​\(y\|x\)\+Z​\(x\),r\_\{\\theta\}\(x,y\)=\\beta\\bar\{\\pi\}\_\{\\theta\}\(y\|x\)\+Z\(x\),\(8\)whereπ¯θ​\(y\|x\)=πθ​\(y\|x\)1/\|y\|\\bar\{\\pi\}\_\{\\theta\}\(y\|x\)=\\pi\_\{\\theta\}\(y\|x\)^\{1/\|y\|\}is the length\-normalized likelihood of a response for some modelπθ\\pi\_\{\\theta\}with\|y\|\|y\|being the length of the response andZ​\(x\)Z\(x\)is the partition function defined as

Z​\(x\)=∑yπref​\(y\|x\)​exp⁡\(rθ​\(x,y\)\|y\|β\)\.Z\(x\)=\\sum\_\{y\}\\pi\_\{\\text\{ref\}\}\(y\|x\)\\exp\\left\(\\frac\{r\_\{\\theta\}\(x,y\)^\{\|y\|\}\}\{\\beta\}\\right\)\.\(9\)In addition, SimPO introduces the use of a margin term to further separate the likelihoods of the preferred and non\-preferred responses\. Finally, the SimPO objectiveℒSimPO\\mathcal\{L\}\_\{\\text\{SimPO\}\}can be written as follows:

𝔼\(x,yw,yl\)∼D​\[−log⁡σ​\(β​log⁡π¯θ​\(yw\|x\)π¯θ​\(yl\|x\)−β​γ\)\],\\mathbb\{E\}\_\{\(x,y\_\{w\},y\_\{l\}\)\\sim D\}\\left\[\-\\log\\sigma\\left\(\\beta\\log\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\}\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\-\\beta\\gamma\\right\)\\right\],\(10\)whereγ\\gammais a hyperparameter for the margin size\.

#### Dangers of neglecting the partition function

Crucially, neither the DPO nor the SimPO objectives contain the partition functionZ​\(x\)Z\(x\)\. Indeed, since they consider the*differences*in rewards for each pair of responses, theZ​\(x\)Z\(x\)terms cancel out in the respective objective formulations for DPO and SimPO, effectively rendering them invariant to changes inZ​\(x\)Z\(x\)\. While mathematically convenient, this simplification deprives the model of an important factor — it does not incentivize the preservation of a good estimate ofZ​\(x\)Z\(x\)\. Furthermore, the limited set of responses available in the training set compared to the set of all possible responses makes proper normalization difficult\. This difficulty in properly normalizing the rewards provides a compelling explanation for why likelihood displacement often occurs: there is insufficient information in the objective to learn calibrated likelihoods\.

#### Example

We provide an illustrative example of how this offset results in undesirable behavior in the trained model, shown in Figure[1](https://arxiv.org/html/2607.16240#S1.F1)\. We start with a reference model that assigns a likelihood of 0\.4 to optionAAand 0\.3 to optionsBBandCC\. Since we are considering a proper distribution, the partition function based on these likelihoods is 1\. We can then consider training on a single preferenceA<BA<B, whereCCis not present\. As outlined in\(Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7)\), over\-optimization might cause the likelihoods ofAAandBBto drop simultaneously, which in turn causes the probability of the unseen responseCCto spuriously increase\. If instead we incentivize the conservation of the total likelihood betweenAAandBB, we can still learn the preference correctly, while minimizing changes to responses outside the training distribution, preserving the probability of outputting the unseen responseCC\. This allows the model to more reliably generate responses that more closely resemble the given preferences, and avoids generating unseen and potentially harmful responses\.

## 3Method

We first consider what the partition function for the rewards should be\. If we consider any parameterized language modelπθ\\pi\_\{\\theta\}that applies softmax to its outputs, then the output distribution is always normalized and the partition function for the model is 1\. As a result, we have the key property thatthe partition function should remain fixed at 1 throughout training\.

However, the implicit DPO reward does not account for this constraint as the objective does not contain a term for the partition function resulting in phenomena such as likelihood displacement\(Razinet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib7)\)\. Due to difficulties in effectively estimating the partition function for preference datasets \(each prompt has only a single pair of responses\), we propose a regularization term motivated by the insight of a fixed partition function while working around the challenge of accounting for unseen responses\. We enforce normalization in the rewards by adding a regularization term that maintains the probability mass over the set of responses seen for a prompt\. Given a preference data point with promptxxand responsesyw,yly\_\{w\},y\_\{l\}, we start with a regularization penalty of

λ​\(log⁡πθ​\(yw\|x\)\+πθ​\(yl\|x\)πref​\(yw\|x\)\+πref​\(yl\|x\)\)2,\\lambda\\left\(\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\|x\)\+\\pi\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)^\{2\},\(11\)whereλ\\lambdais a hyperparameter\. The penalty aims to keep the ratio of the total response probability close to11\. Notice that the regularization term is minimized when the ratio of the total response probabilities is11and the total probability assigned to the two responses under the optimized model is the same as that under the original model\. By maintaining the total probability, we mitigate offsets in likelihood that would occur given the lack of information about the full response distribution, and as a result, implicitly avoid large spurious changes\. Unlike other regularization methods such as adding an SFT term on the preferred responses or DPO\-P\(Palet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib10)\)that unconditionally increase the response likelihoods, our method aims to mitigate changes to unseen responses\. However, response probabilities decrease exponentially with length and as responses are often hundreds of tokens long, the ratio of response probabilities may be sensitive to differences in length or small changes in per\-token probabilities\. To have a more stable penalty and to mitigate length bias, we use length\-normalized probabilities\. Usingπ¯θ,π¯ref\\bar\{\\pi\}\_\{\\theta\},\\bar\{\\pi\}\_\{\\text\{ref\}\}to denote length\-normalized probabilities, we have the following regularization penalty:

ℛ​\(πθ,πref,x,yw,yl\)=λ​\(log⁡π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)π¯ref​\(yw\|x\)\+π¯ref​\(yl\|x\)\)2\.\\mathcal\{R\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\},x,y\_\{w\},y\_\{l\}\)=\\lambda\\left\(\\log\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)^\{2\}\.\(12\)For consistency with the regularization, we modify the original objective with length\-normalization which, with the regularization, gives the following objective for DPO:

ℒN\-DPO​\(θ\)=ℒDPO​\(πθ\)\+ℛ​\(πθ,πref,x,yw,yl\)\.\\mathcal\{L\}\_\{\\text\{N\-DPO\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\}\)\+\\mathcal\{R\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\},x,y\_\{w\},y\_\{l\}\)\.\(13\)We refer to the modified version of DPO as N\-DPO\. We also modify SimPO with the same form of regularization, but since SimPO is already length normalized, we simply add the regularization term resulting in:

ℒN\-SimPO​\(θ\)=ℒSimPO​\(πθ\)\+ℛ​\(πθ,πref,x,yw,yl\),\\mathcal\{L\}\_\{\\text\{N\-SimPO\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{SimPO\}\}\(\\pi\_\{\\theta\}\)\+\\mathcal\{R\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\},x,y\_\{w\},y\_\{l\}\),\(14\)which we refer to as N\-SimPO\.

## 4Analysis

In this section, we consider an analysis of the effect of our regularization term by observing how it changes the likelihoods of responses over the course of training, at the level of the full response and at a token level\. We track the dynamics of the response likelihoods over the course of training with and without our regularization\. We then explore how the reward for each token changes by studying empirically the distribution of token\-wise rewards with and without regularization\. Through a theoretical gradient analysis, we reveal how likelihood displacement is distributed across tokens and how our regularization term uses low\-likelihood tokens to reshape the reward distribution and response likelihoods\.

### 4\.1Effect of Regularization

We analyze the dynamics of the response likelihoods for each of the methods on UltraFeedback\(Cuiet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib14)\)over the course of training, which we show in Figure[2](https://arxiv.org/html/2607.16240#S4.F2)\. We demonstrate that adding regularization results in less likelihood displacement, particularly for DPO where the chosen responses maintain similar likelihood throughout training\. In the case of SimPO, adding regularization keeps the chosen likelihoods similar while significantly decreasing the rejected response likelihoods, allowing the model to more strongly distinguish preferences without making the chosen response less likely\. While we can see that the regularization does effectively mitigate displacement or result in stronger preference learning without worsening displacement, the response likelihoods do not explain where displacement comes from and how exactly the regularization targets displacement\.

To better understand displacement and the effect of regularization, we consider the overall token\-wise reward distribution for each model and method as well as the distribution of the minimum token\-wise reward per sample\. The token\-wise reward of a responseyyislog⁡πθ​\(y\(i\)\)πref​\(y\(i\)\)\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y^\{\(i\)\}\)\}wherey\(i\)y^\{\(i\)\}is theiith token\. The minimum token\-wise reward ismini∈\[L\]⁡log⁡πθ​\(y\(i\)\|x\)πref​\(y\(i\)\|x\)\\min\_\{i\\in\[L\]\}\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y^\{\(i\)\}\|x\)\}whereLLis the length of the response\. To provide a clear comparison across settings, we use the change in log\-likelihood per token as a normalized reward\. We provide the results for Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib12)\)in Figure[3](https://arxiv.org/html/2607.16240#S4.F3)\. When using DPO the peak of the minimum reward distribution is around \-10 while the overall reward distribution lies mostly within \-2\.5 and 2\.5, suggesting that for many samples when using DPO, the minimum reward lies far outside the typical range\. This suggests that a significant part of likelihood displacement comes from outlier tokens significantly dropping the response likelihood\. We demonstrate that this change does in fact reduce likelihood displacement by plotting the likelihood of responses over training in Figure[2](https://arxiv.org/html/2607.16240#S4.F2)\. We also observe that while the overall token distribution does not change much between DPO and N\-DPO, we see that there is a large shift in the minimum reward distribution\. This suggests that N\-DPO primarily mitigates the effect of these outlier tokens while maintaining the reward otherwise\. We demonstrate how our regularization term mitigates these outlier tokens when total likelihood decreases as seen with Llama\-3\.1\-8B\-Instruct\.

Llama\-3\.1\-8B\-Instruct![Refer to caption](https://arxiv.org/html/2607.16240v1/x2.png)\(a\)Response likelihood over training with DPO vs\. N\-DPO
![Refer to caption](https://arxiv.org/html/2607.16240v1/x3.png)\(b\)Response likelihood over training with SimPO vs\. N\-SimPO

Figure 2:Comparison of response likelihoods between DPO/SimPO \(left\) and N\-DPO/N\-SimPO \(right\) for Llama\-3\.1\-8B\-Instruct\. In blue is the chosen response likelihood and in red the rejected response likelihood\.
### 4\.2Token\-wise Gradient Analysis

We consider the gradient of

ℛ​\(πθ,πref,x,yw,yl\)=λ​\(log⁡π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)π¯ref​\(yw\|x\)\+π¯ref​\(yl\|x\)\)2\\mathcal\{R\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\},x,y\_\{w\},y\_\{l\}\)=\\lambda\\left\(\\log\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)^\{2\}\(15\)with respect toθ\\theta\. First, we define the necessary notation\. LetPθ​\(x\),Pref​\(x\)P\_\{\\theta\}\(x\),P\_\{\\text\{ref\}\}\(x\)be the total length\-normalized response probabilities under the trained and reference model respectively and letπθ​\(yw/l\(i\)\)\\pi\_\{\\theta\}\(y\_\{w/l\}^\{\(i\)\}\)be the likelihood of theiith token in a response given the previous tokens\. Then, we can write the gradient of the regularization term with respect to the parametersθ\\thetaas

∇θℛ=2​λPθ​\(x\)log\(Pθ​\(x\)Pref​\(x\)\)\(π¯θ​\(yw\|x\)\|yw\|∑i=1\|yw\|∇θ\(πθ​\(yw\(i\)\)\)πθ​\(yw\(i\)\)\+π¯θ​\(yl\|x\)\|yl\|∑i=1\|yl\|∇θ\(πθ​\(yl\(i\)\)\)πθ​\(yl\(i\)\)\)\.\\begin\{split\}\\nabla\_\{\\theta\}\\mathcal\{R\}&=\\frac\{2\\lambda\}\{P\_\{\\theta\}\(x\)\}\\log\\left\(\\frac\{P\_\{\\theta\}\(x\)\}\{P\_\{\\text\{ref\}\}\(x\)\}\\right\)\\Bigg\(\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\}\{\|y\_\{w\}\|\}\\sum\_\{i=1\}^\{\|y\_\{w\}\|\}\\frac\{\\nabla\_\{\\theta\}\(\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\)\}\{\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\}\\\\ &\+\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\|y\_\{l\}\|\}\\sum\_\{i=1\}^\{\|y\_\{l\}\|\}\\frac\{\\nabla\_\{\\theta\}\(\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\)\}\{\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\}\\Bigg\)\.\\end\{split\}\(16\)Notice that all of the token\-wise contributions to the gradient have the same sign\. The sign of the gradients are determined bylog⁡\(Pθ​\(x\)Pref​\(x\)\)\\log\\left\(\\frac\{P\_\{\\theta\}\(x\)\}\{P\_\{\\text\{ref\}\}\(x\)\}\\right\)where if the total response probability has decreased, the gradients will be negative increasing token probabilities\. If the total probability increases, the opposite will occur\. Furthermore, looking at each token\-wise gradient, we have∇θ\(πθ​\(y\(i\)\)\)πθ​\(y\(i\)\)\\frac\{\\nabla\_\{\\theta\}\(\\pi\_\{\\theta\}\(y^\{\(i\)\}\)\)\}\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\)\}which is inversely proportional to the likelihood of each token\. If a token has small likelihood \(e\.g\.,1​e−71e^\{\-7\}times smaller than other tokens\) the low\-likelihood token’s gradient will dominate\. This has been observed in the gradient analysis in the ConfPO paper\(Yoonet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib23)\)\. Then, if there is an outlier token with small likelihood and the response probability has decreased as seen with models such as Llama\-3\.1\-8B\-Instruct, the regularization term will strongly increase the likelihood of the outlier tokens\. More generally, if the response likelihood has decreased significantly, the regularization term will prioritize updating the lowest likelihood tokens to increase the overall response likelihood\. In this way, when the response likelihood decreases, the regularization term primarily shifts large negative rewards closer to 0\.

We can also consider the case when the total response probability has increased\. Now, the regularization term has gradients that will result in a decrease in response probability, but similar to before, these gradients will be dominated by the low likelihood tokens\. Then, when total response probability has increased compared to the original, the regularization corrects this primarily by decreasing the likelihood of low likelihood tokens\. In this way, the overall response probability is maintained with minimal changes to the most likely tokens, which are also most relevant for generation\. In this way, the regularization term mitigates overly large likelihoods and does so with minimal distribution shift\.

The gradient analysis reveals that introducing the regularization term effectively mitigates shifts in response likelihood and mitigates the presence of outlier tokens\. Furthermore, we find that the regularization term does so primarily through low likelihood tokens which have a smaller effect on the overall sampling distribution\. In this way, our regularization term demonstrates that low likelihood tokens not only provide an approximation of the gradient\(Yoonet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib23)\)but also can be utilized to shape the reward and likelihood distribution\.

Llama\-3\.1\-8B\-Instruct![Refer to caption](https://arxiv.org/html/2607.16240v1/x4.png)\(a\)Minimum of token likelihood changes per sample for DPO
![Refer to caption](https://arxiv.org/html/2607.16240v1/x5.png)\(b\)Overall distribution of token likelihood changes for DPO
![Refer to caption](https://arxiv.org/html/2607.16240v1/x6.png)\(c\)Minimum of token likelihood changes per sample for N\-DPO
![Refer to caption](https://arxiv.org/html/2607.16240v1/x7.png)\(d\)Overall distribution of token likelihood changes for N\-DPO

Figure 3:Comparison of token\-wise reward distributions between DPO and N\-DPO for Llama\-3\.1\-8B\-Instruct\. On the left is the distribution of the minimum token reward per sample, and on the right is the distribution of all token rewards\.

## 5Experiments

We evaluate the impact of accounting for reward normalization on downstream performance on instruction\-following tasks, common sense and reasoning, and implicit reward modeling\.

### 5\.1Evaluation

We focus our evaluation on understanding the effect of our modifications by comparing DPO with N\-DPO and SimPO with N\-SimPO\. We expect that due to the more strongly enforced normalization of rewards, likelihood displacement would be mitigated, and more generally, the model can learn from the preference data with less of a distribution shift\. As a result, we expect to see better trade\-offs between generation quality and benchmark performance when using N\-DPO or N\-SimPO\. We also expect that with better normalized rewards, we may see better generalization of the implicit reward\. We compare our methods to other forms of regularization such as DPO\+SFT and DPOP\(Palet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib10)\)\.

We train Mistral\-7B\-Instruct\(Jianget al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib28)\), Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib12)\), and OLMo\-7B\-SFT\(Groeneveldet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib13)\)on Ultrafeedback\(Cuiet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib14)\)using DPO, N\-DPO, SimPO, N\-SimPO, DPO\+SFT, and DPOP\. For all runs, we train for 1 epoch with a cosine learning rate scheduler\. We select hyperparameters using AlpacaEval1 results\. We evaluate generation quality using AlpacaEval2\(Duboiset al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib15)\)and assess the model on common sense and reasoning benchmarks \(ARC, MMLU, HellaSwag, PIQA, SciQ, WinoGrande\)\. We evaluate reward modeling for a range of datasets \(Ultrafeedback, HH\-RLHF\(Baiet al\.,[2022](https://arxiv.org/html/2607.16240#bib.bib40)\), HelpSteer2\(Wanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib41)\)\)\.

#### Instruction Following\.

AlpacaEval\(Duboiset al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib15)\)is a benchmark that evaluates a model based on its win\-rate compared to GPT\-3\.5\(OpenAI,[2022](https://arxiv.org/html/2607.16240#bib.bib42)\)for AlpacaEval1 and GPT\-4\(Achiamet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib33)\)for AlpacaEval2 using an LLM\-as\-a\-judge in an instruction\-following setting\. For AlpacaEval2 both a raw win\-rate \(WR\) and a length\-controlled win\-rate \(LC\) are provided\. Table[1](https://arxiv.org/html/2607.16240#S5.T1)shows the win\-rates of the models on AlpacaEval1 and AlpacaEval2, where we can see that N\-DPO and N\-SimPO generally have improved performance over DPO and SimPO, respectively\. In particular, we see over a 20% increase for AlpacaEval2 \(LC\) for Llama\-3\.1\-8B\-Instruct between DPO and N\-DPO and a large increase in both AlpacaEval1 and AlpacaEval2 \(WR\) between SimPO and N\-SimPO for OLMo\-7B\-SFT with over a 75% increase for AlpacaEval2 \(WR\)\. While we see a small decrease in instruction\-following quality using N\-SimPO for Llama\-3\.1\-8B\-Instruct, the benchmark performance noticeably improves\. Furthermore, we find that N\-DPO and N\-SimPO perform better than both DPO\+SFT and DPOP on AlpacaEval2, demonstrating the benefit of maintaining the total response probabilities rather than unconditionally increasing the likelihood of preferred responses\.

Table 1:AlpacaEval scores across methods, with standard errors as subscripts\. A1 corresponds to the AlpacaEval1 win\-rate, while WR and LC correspond to the raw and length\-controlled AlpacaEval2 win\-rates, respectively\. A1 results should be treated as validation metrics\.Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTA1WRLCA1WRLCA1WRLCReference93\.1793\.1714\.54±1\.1414\.54\_\{\\pm 1\.14\}18\.17±0\.4918\.17\_\{\\pm 0\.49\}90\.0090\.0025\.09±1\.3925\.09\_\{\\pm 1\.39\}18\.84±0\.1718\.84\_\{\\pm 0\.17\}58\.1558\.152\.92±0\.542\.92\_\{\\pm 0\.54\}4\.50±0\.284\.50\_\{\\pm 0\.28\}DPO94\.66\\mathbf\{94\.66\}18\.13±1\.2418\.13\_\{\\pm 1\.24\}23\.66±0\.4923\.66\_\{\\pm 0\.49\}91\.67\\mathbf\{91\.67\}24\.18±1\.3624\.18\_\{\\pm 1\.36\}21\.72±0\.2721\.72\_\{\\pm 0\.27\}79\.6379\.638\.12±0\.898\.12\_\{\\pm 0\.89\}8\.96±0\.348\.96\_\{\\pm 0\.34\}N\-DPO94\.2894\.2820\.08±1\.28\\mathbf\{20\.08\}\_\{\\pm 1\.28\}25\.48±0\.50\\mathbf\{25\.48\}\_\{\\pm 0\.50\}91\.2891\.2827\.57±1\.43\\mathbf\{27\.57\}\_\{\\pm 1\.43\}24\.97±0\.20\\mathbf\{24\.97\}\_\{\\pm 0\.20\}81\.24\\mathbf\{81\.24\}11\.79±1\.04\\mathbf\{11\.79\}\_\{\\pm 1\.04\}9\.88±0\.29\\mathbf\{9\.88\}\_\{\\pm 0\.29\}SimPO90\.8790\.8719\.68±1\.2819\.68\_\{\\pm 1\.28\}26\.34±0\.42\\mathbf\{26\.34\}\_\{\\pm 0\.42\}88\.93\\mathbf\{88\.93\}28\.68±1\.46\\mathbf\{28\.68\}\_\{\\pm 1\.46\}26\.74±0\.12\\mathbf\{26\.74\}\_\{\\pm 0\.12\}71\.2171\.213\.66±0\.613\.66\_\{\\pm 0\.61\}5\.15±0\.285\.15\_\{\\pm 0\.28\}N\-SimPO92\.72\\mathbf\{92\.72\}20\.38±1\.30\\mathbf\{20\.38\}\_\{\\pm 1\.30\}26\.05±0\.4526\.05\_\{\\pm 0\.45\}85\.4585\.4527\.11±1\.4327\.11\_\{\\pm 1\.43\}25\.54±0\.1825\.54\_\{\\pm 0\.18\}79\.63\\mathbf\{79\.63\}10\.02±0\.96\\mathbf\{10\.02\}\_\{\\pm 0\.96\}8\.70±0\.31\\mathbf\{8\.70\}\_\{\\pm 0\.31\}DPO\+SFT92\.9892\.9818\.32±1\.2418\.32\_\{\\pm 1\.24\}20\.11±0\.5320\.11\_\{\\pm 0\.53\}76\.4376\.434\.07±0\.634\.07\_\{\\pm 0\.63\}4\.12±0\.194\.12\_\{\\pm 0\.19\}66\.8366\.833\.70±0\.613\.70\_\{\\pm 0\.61\}2\.69±0\.152\.69\_\{\\pm 0\.15\}DPOP94\.5294\.5214\.76±1\.1314\.76\_\{\\pm 1\.13\}18\.22±0\.4818\.22\_\{\\pm 0\.48\}91\.2891\.2820\.73±1\.2820\.73\_\{\\pm 1\.28\}15\.92±0\.1715\.92\_\{\\pm 0\.17\}70\.5070\.503\.86±0\.623\.86\_\{\\pm 0\.62\}4\.42±0\.274\.42\_\{\\pm 0\.27\}
#### Common Sense and Reasoning\.

In addition to instruction\-following quality, we perform evaluation on various benchmarks from LM Evaluation Harness\(Gaoet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib3)\)to see how well the model maintains its general common sense and reasoning capabilities\. We expect reducing likelihood displacement to also reduce distribution shift allowing for better benchmark performance\. To quantify how well the model maintains benchmark performance, we consider the difference in scores between the reference model and the fine\-tuned model\. The results for the evaluation are shown in Table[2](https://arxiv.org/html/2607.16240#S5.T2)\. We can see that for Llama\-3\.1\-8B\-Instruct, N\-DPO and N\-SimPO result not only in better maintenance of benchmark performance, but also improve scores on average by over 3%\. For Mistral\-7B\-Instruct, we also see an improvement between DPO versus N\-DPO along with a small drop in benchmark performance that is accompanied by an increase in generation quality by a larger margin than the drop in benchmark performance\. For OLMo\-7B\-SFT, we find that N\-DPO improves benchmark performance while improving generation quality\. We note that for N\-SimPO, there is a drop in benchmark performance, but the results suggest this is due to difficulties with applying SimPO to OLMo: generation quality increased the most with the smallest learning rate, margin, and beta\. This suggests that larger updates using SimPO do not benefit OLMo\-7B\-SFT, and that N\-SimPO creates a better optimization landscape for OLMo\-7B\-SFT, allowing for more dramatic changes in weaker base models that improve generation quality\. Compared to DPO\+SFT and DPOP, across models, we see that N\-DPO and N\-SimPO have comparable benchmark performance\. We provide the hyperparameter sweep range and final hyperparameters in Appendix[B](https://arxiv.org/html/2607.16240#A2)\.

Table 2:Common sense and reasoning benchmarks performance across methods, with standard errors as subscripts\. DPOP uses the best setting for each model\.Mistral\-7B\-InstructMMLUARC ChalARC EasyHellaSwagPiQASciQWinoGAvgReference44\.32±0\.4144\.32\_\{\\pm 0\.41\}49\.49±1\.4649\.49\_\{\\pm 1\.46\}70\.33±0\.9470\.33\_\{\\pm 0\.94\}62\.86±0\.4862\.86\_\{\\pm 0\.48\}75\.19±1\.0175\.19\_\{\\pm 1\.01\}90\.90±0\.9190\.90\_\{\\pm 0\.91\}61\.48±1\.3761\.48\_\{\\pm 1\.37\}64\.9464\.94DPO−1\.34±0\.41\-1\.34\_\{\\pm 0\.41\}−3\.41±1\.46\-3\.41\_\{\\pm 1\.46\}−5\.05±0\.98\-5\.05\_\{\\pm 0\.98\}−1\.45±0\.49\-1\.45\_\{\\pm 0\.49\}−2\.67±1\.04\-2\.67\_\{\\pm 1\.04\}−3\.00±1\.03\-3\.00\_\{\\pm 1\.03\}−2\.68±1\.38\-2\.68\_\{\\pm 1\.38\}−2\.80\-2\.80N\-DPO\+0\.17±0\.41\\mathbf\{\+0\.17\}\_\{\\pm 0\.41\}−1\.11±1\.46\\mathbf\{\-1\.11\}\_\{\\pm 1\.46\}−1\.90±0\.95\\mathbf\{\-1\.90\}\_\{\\pm 0\.95\}−0\.95±0\.48\\mathbf\{\-0\.95\}\_\{\\pm 0\.48\}−1\.96±1\.03\\mathbf\{\-1\.96\}\_\{\\pm 1\.03\}−0\.80±0\.94\\mathbf\{\-0\.80\}\_\{\\pm 0\.94\}−0\.31±1\.37\\mathbf\{\-0\.31\}\_\{\\pm 1\.37\}−0\.98\\mathbf\{\-0\.98\}SimPO−0\.70±0\.41\\mathbf\{\-0\.70\}\_\{\\pm 0\.41\}−3\.84±1\.46\\mathbf\{\-3\.84\}\_\{\\pm 1\.46\}−3\.12±0\.96\\mathbf\{\-3\.12\}\_\{\\pm 0\.96\}−7\.47±0\.50\-7\.47\_\{\\pm 0\.50\}−5\.06±1\.07\-5\.06\_\{\\pm 1\.07\}−0\.50±0\.93\\mathbf\{\-0\.50\}\_\{\\pm 0\.93\}−2\.21±1\.38\\mathbf\{\-2\.21\}\_\{\\pm 1\.38\}−3\.27\\mathbf\{\-3\.27\}N\-SimPO−0\.84±0\.41\-0\.84\_\{\\pm 0\.41\}−4\.10±1\.46\-4\.10\_\{\\pm 1\.46\}−3\.62±0\.97\-3\.62\_\{\\pm 0\.97\}−6\.87±0\.49\\mathbf\{\-6\.87\}\_\{\\pm 0\.49\}−4\.79±1\.06\\mathbf\{\-4\.79\}\_\{\\pm 1\.06\}−1\.20±0\.96\-1\.20\_\{\\pm 0\.96\}−2\.44±1\.38\-2\.44\_\{\\pm 1\.38\}−3\.41\-3\.41DPO\+SFT\+2\.57±0\.29\\mathbf\{\+2\.57\}\_\{\\pm 0\.29\}\+6\.14±1\.45\\mathbf\{\+6\.14\}\_\{\\pm 1\.45\}\+4\.75±0\.89\\mathbf\{\+4\.75\}\_\{\\pm 0\.89\}\+2\.34±0\.48\\mathbf\{\+2\.34\}\_\{\\pm 0\.48\}\+3\.37±0\.96\\mathbf\{\+3\.37\}\_\{\\pm 0\.96\}\+1\.60±0\.83\\mathbf\{\+1\.60\}\_\{\\pm 0\.83\}\+6\.40±1\.31\\mathbf\{\+6\.40\}\_\{\\pm 1\.31\}\+3\.88\\mathbf\{\+3\.88\}DPOP\+1\.65±0\.29\+1\.65\_\{\\pm 0\.29\}\+0\.25±1\.46\+0\.25\_\{\\pm 1\.46\}\+0\.80±0\.93\+0\.80\_\{\\pm 0\.93\}\+1\.38±0\.48\+1\.38\_\{\\pm 0\.48\}\+0\.44±1\.00\+0\.44\_\{\\pm 1\.00\}\+0\.30±0\.90\+0\.30\_\{\\pm 0\.90\}\+0\.87±1\.36\+0\.87\_\{\\pm 1\.36\}\+0\.81\+0\.81
Llama\-3\.1\-8B\-InstructMMLUARC ChalARC EasyHellaSwagPiQASciQWinoGAvgReference43\.28±0\.4143\.28\_\{\\pm 0\.41\}52\.82±1\.4652\.82\_\{\\pm 1\.46\}81\.44±0\.8081\.44\_\{\\pm 0\.80\}57\.52±0\.4957\.52\_\{\\pm 0\.49\}79\.76±0\.9479\.76\_\{\\pm 0\.94\}95\.20±0\.6895\.20\_\{\\pm 0\.68\}67\.40±1\.3267\.40\_\{\\pm 1\.32\}68\.2068\.20DPO\+4\.71±0\.41\\mathbf\{\+4\.71\}\_\{\\pm 0\.41\}−4\.78±1\.46\-4\.78\_\{\\pm 1\.46\}−9\.51±0\.92\-9\.51\_\{\\pm 0\.92\}\+6\.43±0\.48\\mathbf\{\+6\.43\}\_\{\\pm 0\.48\}−5\.82±1\.02\-5\.82\_\{\\pm 1\.02\}−4\.20±0\.91\-4\.20\_\{\\pm 0\.91\}−9\.55±1\.39\-9\.55\_\{\\pm 1\.39\}−3\.25\-3\.25N\-DPO\+4\.24±0\.41\+4\.24\_\{\\pm 0\.41\}\+3\.84±1\.45\\mathbf\{\+3\.84\}\_\{\\pm 1\.45\}\+0\.80±0\.78\\mathbf\{\+0\.80\}\_\{\\pm 0\.78\}\+6\.24±0\.48\+6\.24\_\{\\pm 0\.48\}\+0\.82±0\.92\\mathbf\{\+0\.82\}\_\{\\pm 0\.92\}\+0\.20±0\.66\\mathbf\{\+0\.20\}\_\{\\pm 0\.66\}\+5\.13±1\.25\\mathbf\{\+5\.13\}\_\{\\pm 1\.25\}\+3\.04\\mathbf\{\+3\.04\}SimPO\+1\.69±0\.41\+1\.69\_\{\\pm 0\.41\}\+4\.77±1\.44\+4\.77\_\{\\pm 1\.44\}\+0\.17±0\.80\+0\.17\_\{\\pm 0\.80\}−5\.49±0\.50\-5\.49\_\{\\pm 0\.50\}−0\.54±0\.82\-0\.54\_\{\\pm 0\.82\}\+0\.80±0\.62\+0\.80\_\{\\pm 0\.62\}\+6\.55±1\.23\\mathbf\{\+6\.55\}\_\{\\pm 1\.23\}\+1\.14\+1\.14N\-SimPO\+8\.49±0\.41\\mathbf\{\+8\.49\}\_\{\\pm 0\.41\}\+9\.21±1\.42\\mathbf\{\+9\.21\}\_\{\\pm 1\.42\}\+2\.27±0\.76\\mathbf\{\+2\.27\}\_\{\\pm 0\.76\}−0\.92±0\.49\\mathbf\{\-0\.92\}\_\{\\pm 0\.49\}\+1\.36±0\.91\\mathbf\{\+1\.36\}\_\{\\pm 0\.91\}\+1\.10±0\.60\\mathbf\{\+1\.10\}\_\{\\pm 0\.60\}\+4\.33±1\.26\+4\.33\_\{\\pm 1\.26\}\+3\.69\\mathbf\{\+3\.69\}DPO\+SFT\+5\.61±0\.29\+5\.61\_\{\\pm 0\.29\}−4\.61±1\.46\-4\.61\_\{\\pm 1\.46\}−0\.59±0\.81\-0\.59\_\{\\pm 0\.81\}−1\.65±0\.50\-1\.65\_\{\\pm 0\.50\}\+0\.27±0\.93\+0\.27\_\{\\pm 0\.93\}\+0\.40±0\.65\+0\.40\_\{\\pm 0\.65\}\+4\.27±1\.27\+4\.27\_\{\\pm 1\.27\}\+0\.53\+0\.53DPOP\+7\.08±0\.29\+7\.08\_\{\\pm 0\.29\}\+0\.41±1\.46\+0\.41\_\{\\pm 1\.46\}\+0\.59±0\.79\+0\.59\_\{\\pm 0\.79\}\+0\.36±0\.49\+0\.36\_\{\\pm 0\.49\}\+0\.22±0\.93\+0\.22\_\{\\pm 0\.93\}\+0\.00±0\.68\+0\.00\_\{\\pm 0\.68\}\+1\.03±1\.31\+1\.03\_\{\\pm 1\.31\}\+1\.38\+1\.38
OLMo\-7B\-SFTMMLUARC ChalARC EasyHellaSwagPiQASciQWinoGAvgReference37\.94±0\.4037\.94\_\{\\pm 0\.40\}39\.33±1\.4339\.33\_\{\\pm 1\.43\}67\.97±0\.9667\.97\_\{\\pm 0\.96\}53\.54±0\.5053\.54\_\{\\pm 0\.50\}76\.66±0\.9976\.66\_\{\\pm 0\.99\}91\.10±0\.9091\.10\_\{\\pm 0\.90\}63\.93±1\.3563\.93\_\{\\pm 1\.35\}61\.5061\.50DPO−0\.15±0\.40\-0\.15\_\{\\pm 0\.40\}−0\.85±1\.42\-0\.85\_\{\\pm 1\.42\}−2\.06±0\.97\-2\.06\_\{\\pm 0\.97\}\+0\.27±0\.50\+0\.27\_\{\\pm 0\.50\}−1\.42±1\.01\-1\.42\_\{\\pm 1\.01\}−4\.80±1\.09\-4\.80\_\{\\pm 1\.09\}−2\.05±1\.36\-2\.05\_\{\\pm 1\.36\}−1\.58\-1\.58N\-DPO\+0\.47±0\.40\\mathbf\{\+0\.47\}\_\{\\pm 0\.40\}\+1\.88±1\.44\\mathbf\{\+1\.88\}\_\{\\pm 1\.44\}\+2\.11±0\.94\\mathbf\{\+2\.11\}\_\{\\pm 0\.94\}\+3\.52±0\.49\\mathbf\{\+3\.52\}\_\{\\pm 0\.49\}−0\.82±1\.00\\mathbf\{\-0\.82\}\_\{\\pm 1\.00\}−1\.20±0\.95\\mathbf\{\-1\.20\}\_\{\\pm 0\.95\}−0\.95±1\.36\\mathbf\{\-0\.95\}\_\{\\pm 1\.36\}\+0\.72\\mathbf\{\+0\.72\}SimPO−0\.04±0\.40\-0\.04\_\{\\pm 0\.40\}−0\.34±1\.43\\mathbf\{\-0\.34\}\_\{\\pm 1\.43\}−0\.25±0\.96\\mathbf\{\-0\.25\}\_\{\\pm 0\.96\}−1\.17±0\.50\\mathbf\{\-1\.17\}\_\{\\pm 0\.50\}−1\.09±1\.02\\mathbf\{\-1\.09\}\_\{\\pm 1\.02\}−0\.70±0\.93\\mathbf\{\-0\.70\}\_\{\\pm 0\.93\}−1\.50±1\.36\\mathbf\{\-1\.50\}\_\{\\pm 1\.36\}−0\.73\\mathbf\{\-0\.73\}N\-SimPO\+0\.10±0\.40\\mathbf\{\+0\.10\}\_\{\\pm 0\.40\}−1\.53±1\.42\-1\.53\_\{\\pm 1\.42\}−14\.01±0\.98\-14\.01\_\{\\pm 0\.98\}−3\.49±0\.50\-3\.49\_\{\\pm 0\.50\}−4\.79±1\.05\-4\.79\_\{\\pm 1\.05\}−4\.20±1\.07\-4\.20\_\{\\pm 1\.07\}−5\.76±1\.39\-5\.76\_\{\\pm 1\.39\}−4\.81\-4\.81DPO\+SFT−0\.62±0\.28\-0\.62\_\{\\pm 0\.28\}−4\.18±1\.40\-4\.18\_\{\\pm 1\.40\}−8\.21±1\.01\-8\.21\_\{\\pm 1\.01\}−3\.46±0\.50\-3\.46\_\{\\pm 0\.50\}−0\.49±0\.99\-0\.49\_\{\\pm 0\.99\}−12\.80±1\.30\-12\.80\_\{\\pm 1\.30\}−4\.50±1\.38\-4\.50\_\{\\pm 1\.38\}−4\.89\-4\.89DPOP\+1\.30±0\.28\\mathbf\{\+1\.30\}\_\{\\pm 0\.28\}\+0\.00±1\.42\+0\.00\_\{\\pm 1\.42\}\+2\.23±0\.94\\mathbf\{\+2\.23\}\_\{\\pm 0\.94\}\+2\.34±0\.50\+2\.34\_\{\\pm 0\.50\}\+2\.23±0\.96\\mathbf\{\+2\.23\}\_\{\\pm 0\.96\}−1\.20±0\.94\-1\.20\_\{\\pm 0\.94\}\+2\.05±1\.34\\mathbf\{\+2\.05\}\_\{\\pm 1\.34\}\+1\.28\\mathbf\{\+1\.28\}

#### Implicit Reward\.

We further evaluate the trained models on their implicit reward modeling capabilities on both UltraFeedback and datasets not seen during training \(HH\-RLHF, HelpSteer2\)\. Table[3](https://arxiv.org/html/2607.16240#S5.T3)shows the reward accuracy on the eval splits for these datasets using a length\-normalized reward\. We find that DPO\+SFT and DPOP have a larger drop in reward accuracy compared to other methods\.

Table 3:Reward accuracy on UltraFeedback \(UF\), HH\-RLHF \(HH\), and HelpSteer2 \(HS\) across methods\.Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTUFHHHSUFHHHSUFHHHSDPO80\.7355\.1266\.1574\.7159\.5568\.2371\.1253\.3064\.06N\-DPO94\.2881\.9456\.3377\.2058\.7375\.0073\.2654\.6364\.32SimPO82\.1257\.1764\.3276\.2257\.0367\.1960\.1955\.4753\.65N\-SimPO82\.0656\.8263\.2875\.8157\.6267\.9765\.0555\.6856\.77DPO\+SFT53\.1850\.9659\.1137\.6746\.8632\.0359\.1451\.9166\.15DPOP72\.7456\.7455\.7361\.9259\.2855\.2167\.0154\.9055\.99

### 5\.2Regularization Ablation

We perform an ablation on the regularization term by settingλ=0\\lambda=0to demonstrate that maintaining the probability mass helps improve performance beyond using only length\-normalization \(L\-DPO\)\. We provide the results on AlpacaEval in Appendix[A](https://arxiv.org/html/2607.16240#A1), where we can see that performance improves whenλ\\lambdais a non\-zero value\. In Appendix[A](https://arxiv.org/html/2607.16240#A1)we additionally report the sensitivity of reward accuracy and reward margins to the choice ofλ\\lambda, confirming that N\-DPO remains stable across the swept range\.

## 6Related Works

#### DAA Methods\.

A wide range of works have explored various approaches to improve existing DAA methods like DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib5)\)\. One class of changes is modifying the function of the reward margin\(Azaret al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib35); Zhaoet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib36); Tanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib34)\)\. A range of works have approached the problem of over\-optimization and likelihood displacement through modifying the reward function based on alternatives to KL regularization\(Huanget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib6); Guptaet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib8); Wanget al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib22)\), focusing on select tokens\(Yoonet al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib23)\), mitigating length\-normalization exploits\([Guptaet al\.,](https://arxiv.org/html/2607.16240#bib.bib47)\), distilling explicit reward models\(Fischet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib49)\), or adding regularization\(Liuet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib24); Palet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib10)\)\. Another line of work has proposed modifications to the DPO objective for robustness such as rDPO\(Chowdhuryet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib16)\)and ROPO\(Lianget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib17)\), and other works have explored weighting samples\(Zhouet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib18)\)or using rejection sampling\(Xionget al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib19); Zhaoet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib20); Liuet al\.,[2023](https://arxiv.org/html/2607.16240#bib.bib21)\)\. A range of works have considered various forms of data for aligning language models using reward data\(Chenet al\.,[2024a](https://arxiv.org/html/2607.16240#bib.bib43)\)or data from self play\(Wuet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib44);[Guptaet al\.,](https://arxiv.org/html/2607.16240#bib.bib45); Tanget al\.,[2025](https://arxiv.org/html/2607.16240#bib.bib46)\)\. Other objectives include KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib37)\), which uses prospect theory to motivate an objective that considers a set of desirable responses and undesirable responses, and ORPO\(Honget al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib38)\), which utilizes the log odds ratio between the preferred and dispreferred responses for optimization\.

#### Reward over\-optimization\.

Outside of proposing new DAAs to mitigate over\-optimization, works have also analyzed factors that may lead to over\-optimization and likelihood displacement\(Rafailovet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib48)\)\.Razinet al\.\([2024](https://arxiv.org/html/2607.16240#bib.bib7)\)studies under a simplified model how the embedding geometry may lead to likelihood displacement\.Palet al\.\([2024](https://arxiv.org/html/2607.16240#bib.bib10)\)provides an analysis of how likelihood displacement can arise given preference pairs with small edit distance\. Reward over\-optimization is also a generally observed phenomenon outside of DAAs with RLHF\(Chenet al\.,[2024b](https://arxiv.org/html/2607.16240#bib.bib27)\)and other reinforcement learning settings\(Skalseet al\.,[2022](https://arxiv.org/html/2607.16240#bib.bib25); Ibarzet al\.,[2018](https://arxiv.org/html/2607.16240#bib.bib26)\)\.

## 7Discussion

Our results suggest that a common factor in reward over\-optimization and likelihood displacement across methods and, in particular, across both reference\-based and reference\-free methods is the lack of reward normalization\. The objectives modified to normalize rewards, N\-DPO and N\-SimPO, demonstrate better trade\-offs between generation quality and benchmark performance, sometimes improving both while also maintaining reward modeling abilities\. We note that these modifications require tuning an additional hyperparameter, although our sensitivity analysis in Appendix[A](https://arxiv.org/html/2607.16240#A1)shows that N\-DPO is robust to choices ofλ\\lambda\. The improvement across various axes suggests that reward normalization has a significant role in DAAs and that enforcing such constraints can be an effective addition to methods\. Furthermore, our analysis of token\-wise rewards demonstrates that likelihood displacement does not affect the model broadly, but rather primarily on a limited number of tokens, explaining why generation improves despite decreasing likelihood\. A token\-wise analysis of the gradient of our proposed regularization term demonstrates that our regularization term effectively mitigates such outlier tokens and more generally utilizes low likelihood tokens to reshape the reward and likelihood distribution, see Appendix[A\.1](https://arxiv.org/html/2607.16240#A1.SS1)and[A\.2](https://arxiv.org/html/2607.16240#A1.SS2)\. These analyses provide insight into the role of different tokens in preference optimization and demonstrate the need for finer\-grained analyses of reward model behavior\. While our analysis and regularization term do not generalize to online settings or methods such as KTO\(Ethayarajhet al\.,[2024](https://arxiv.org/html/2607.16240#bib.bib37)\), we hope our analysis can motivate more general analyses and methodologies\.

## Impact Statement

Post\-training with methods such as DPO and other DAAs are essential for allowing interactions between people and models to be beneficial and safe\. Our study of likelihood displacement and regularization aims to allow for more robust learning from human preferences and to better understand how post\-training affects models\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.SSS0.Px1.p1.1)\.
- M\. G\. Azar, Z\. D\. Guo, B\. Piot, R\. Munos, M\. Rowland, M\. Valko, and D\. Calandriello \(2024\)A general theoretical paradigm to understand learning from human preferences\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 4447–4455\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- H\. Chen, G\. He, L\. Yuan, G\. Cui, H\. Su, and J\. Zhu \(2024a\)Noise contrastive alignment of language models with explicit rewards\.Advances in Neural Information Processing Systems37,pp\. 117784–117812\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Chen, C\. Zhu, D\. Soselia, J\. Chen, T\. Zhou, T\. Goldstein, H\. Huang, M\. Shoeybi, and B\. Catanzaro \(2024b\)Odin: disentangled reward mitigates hacking in rlhf\.arXiv preprint arXiv:2402\.07319\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- S\. R\. Chowdhury, A\. Kini, and N\. Natarajan \(2024\)Provably robust dpo: aligning language models with noisy feedback\.arXiv preprint arXiv:2403\.00409\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Cui, L\. Yuan, N\. Ding, G\. Yao, W\. Zhu, Y\. Ni, G\. Xie, Z\. Liu, and M\. Sun \(2023\)Ultrafeedback: boosting language models with high\-quality feedback\.Cited by:[§4\.1](https://arxiv.org/html/2607.16240#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.16240#S4.SS1.p2.6),[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela \(2024\)Kto: model alignment as prospect theoretic optimization\.arXiv preprint arXiv:2402\.01306\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2607.16240#S7.p1.1)\.
- A\. Fisch, J\. Eisenstein, V\. Zayats, A\. Agarwal, A\. Beirami, C\. Nagpal, P\. Shaw, and J\. Berant \(2024\)Robust preference optimization through reward model distillation\.arXiv preprint arXiv:2405\.19316\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Groeneveld, I\. Beltagy, P\. Walsh, A\. Bhagia, R\. Kinney, O\. Tafjord, A\. H\. Jha, H\. Ivison, I\. Magnusson, Y\. Wang,et al\.\(2024\)Olmo: accelerating the science of language models\.arXiv preprint arXiv:2402\.00838\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p2.1),[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- A\. Gupta, S\. Tang, Q\. Song, S\. Zhu, J\. Hong, A\. Saha, V\. Gupta, N\. Lee, E\. Kim, S\. Zhu,et al\.\(2025\)AlphaPO: reward shape matters for llm alignment\.arXiv preprint arXiv:2501\.03884\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p3.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- \[15\]T\. Gupta, R\. Madhavan, X\. Zhang, C\. Bansal, and S\. RajmohanAMPO: active multi preference optimization for self\-play preference selection\.InForty\-second International Conference on Machine Learning,Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- \[16\]T\. Gupta, R\. Madhavan, X\. Zhang, C\. Bansal, and S\. RajmohanREFA: reference free alignment with fine\-grained length control\.InSecond Conference on Language Modeling,Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)Orpo: monolithic preference optimization without reference model\.arXiv preprint arXiv:2403\.07691\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Huang, W\. Zhan, T\. Xie, J\. D\. Lee, W\. Sun, A\. Krishnamurthy, and D\. J\. Foster \(2024\)Correcting the mythos of kl\-regularization: direct alignment without overoptimization via chi\-squared preference optimization\.arXiv preprint arXiv:2407\.13399\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p2.1),[§1](https://arxiv.org/html/2607.16240#S1.p3.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- B\. Ibarz, J\. Leike, T\. Pohlen, G\. Irving, S\. Legg, and D\. Amodei \(2018\)Reward learning from human preferences and demonstrations in atari\.Advances in neural information processing systems31\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- X\. Liang, C\. Chen, S\. Qiu, J\. Wang, Y\. Wu, Z\. Fu, Z\. Shi, F\. Wu, and J\. Ye \(2024\)ROPO: robust preference optimization for large language models\.arXiv preprint arXiv:2404\.04102\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- B\. Y\. Lin, A\. Ravichander, X\. Lu, N\. Dziri, M\. Sclar, K\. Chandu, C\. Bhagavatula, and Y\. Choi \(2023a\)The unlocking spell on base llms: rethinking alignment via in\-context learning\.arXiv preprint arXiv:2312\.01552\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p4.1)\.
- Y\. Lin, H\. Lin, W\. Xiong, S\. Diao, J\. Liu, J\. Zhang, R\. Pan, H\. Wang, W\. Hu, H\. Zhang,et al\.\(2023b\)Mitigating the alignment tax of rlhf\.arXiv preprint arXiv:2309\.06256\.Cited by:[item 3](https://arxiv.org/html/2607.16240#S1.I1.i3.p1.1)\.
- T\. Liu, Y\. Zhao, R\. Joshi, M\. Khalman, M\. Saleh, P\. J\. Liu, and J\. Liu \(2023\)Statistical rejection sampling improves preference optimization\.arXiv preprint arXiv:2309\.06657\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, M\. Lu, S\. Zhang, B\. Liu, H\. Guo, Y\. Yang, J\. Blanchet, and Z\. Wang \(2024\)Provably mitigating overoptimization in rlhf: your sft loss is implicitly an adversarial regularizer\.Advances in Neural Information Processing Systems37,pp\. 138663–138697\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)Simpo: simple preference optimization with a reference\-free reward\.Advances in Neural Information Processing Systems37,pp\. 124198–124235\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p3.1)\.
- OpenAI \(2022\)OpenAI gpt\-3\.5 \[text\-davinci\-003\]\.Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p1.1)\.
- A\. Pal, D\. Karkhanis, S\. Dooley, M\. Roberts, S\. Naidu, and C\. White \(2024\)Smaug: fixing failure modes of preference optimisation with dpo\-positive\.arXiv preprint arXiv:2402\.13228\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p3.1),[§3](https://arxiv.org/html/2607.16240#S3.p2.6),[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p1.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- X\. Qi, A\. Panda, K\. Lyu, X\. Ma, S\. Roy, A\. Beirami, P\. Mittal, and P\. Henderson \(2024\)Safety alignment should be made more than just a few tokens deep\.arXiv preprint arXiv:2406\.05946\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p4.1)\.
- R\. Rafailov, Y\. Chittepu, R\. Park, H\. S\. Sikchi, J\. Hejna, B\. Knox, C\. Finn, and S\. Niekum \(2024\)Scaling laws for reward model overoptimization in direct alignment algorithms\.Advances in Neural Information Processing Systems37,pp\. 126207–126242\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p2.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p1.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- N\. Razin, S\. Malladi, A\. Bhaskar, D\. Chen, S\. Arora, and B\. Hanin \(2024\)Unintentional unalignment: likelihood displacement in direct preference optimization\.arXiv preprint arXiv:2410\.08847\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p2.1),[§1](https://arxiv.org/html/2607.16240#S1.p3.1),[§2](https://arxiv.org/html/2607.16240#S2.SS0.SSS0.Px5.p1.11),[§3](https://arxiv.org/html/2607.16240#S3.p2.2),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward gaming\.Advances in Neural Information Processing Systems35,pp\. 9460–9471\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px2.p1.1)\.
- F\. Tajwar, A\. Singh, A\. Sharma, R\. Rafailov, J\. Schneider, T\. Xie, S\. Ermon, C\. Finn, and A\. Kumar \(2024\)Preference fine\-tuning of llms should leverage suboptimal, on\-policy data\.arXiv preprint arXiv:2404\.14367\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p3.1)\.
- X\. Tang, S\. Yoon, S\. Son, H\. Yuan, Q\. Gu, and I\. Bogunovic \(2025\)Game\-theoretic regularized self\-play alignment of large language models\.arXiv preprint arXiv:2503\.00030\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Tang, Z\. D\. Guo, Z\. Zheng, D\. Calandriello, R\. Munos, M\. Rowland, P\. H\. Richemond, M\. Valko, B\. Á\. Pires, and B\. Piot \(2024\)Generalized preference optimization: a unified approach to offline alignment\.arXiv preprint arXiv:2402\.05749\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Wang, Y\. Jiang, C\. Yang, H\. Liu, and Y\. Chen \(2023\)Beyond reverse kl: generalizing direct preference optimization with diverse divergence constraints\.arXiv preprint arXiv:2309\.16240\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Wang, A\. Bukharin, O\. Delalleau, D\. Egert, G\. Shen, J\. Zeng, O\. Kuchaiev, and Y\. Dong \(2024\)HelpSteer2\-preference: complementing ratings with preferences\.External Links:2410\.01257,[Link](https://arxiv.org/abs/2410.01257)Cited by:[§5\.1](https://arxiv.org/html/2607.16240#S5.SS1.p2.1)\.
- Y\. Wu, Z\. Sun, H\. Yuan, K\. Ji, Y\. Yang, and Q\. Gu \(2024\)Self\-play preference optimization for language model alignment\.arXiv preprint arXiv:2405\.00675\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- W\. Xiong, H\. Dong, C\. Ye, Z\. Wang, H\. Zhong, H\. Ji, N\. Jiang, and T\. Zhang \(2023\)Iterative preference learning from human feedback: bridging theory and practice for rlhf under kl\-constraint\.arXiv preprint arXiv:2312\.11456\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- H\. S\. Yoon, E\. Yoon, M\. A\. Hasegawa\-Johnson, S\. Kim, and C\. D\. Yoo \(2025\)ConfPO: exploiting policy model confidence for critical token selection in large language model preference optimization\.arXiv preprint arXiv:2506\.08712\.Cited by:[§1](https://arxiv.org/html/2607.16240#S1.p3.1),[§1](https://arxiv.org/html/2607.16240#S1.p4.1),[§4\.2](https://arxiv.org/html/2607.16240#S4.SS2.p1.8),[§4\.2](https://arxiv.org/html/2607.16240#S4.SS2.p3.1),[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- H\. Zhao, G\. I\. Winata, A\. Das, S\. Zhang, D\. D\. Yao, W\. Tang, and S\. Sahu \(2024\)Rainbowpo: a unified framework for combining improvements in preference optimization\.arXiv preprint arXiv:2410\.04203\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu \(2023\)Slic\-hf: sequence likelihood calibration with human feedback\.arXiv preprint arXiv:2305\.10425\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.
- W\. Zhou, R\. Agrawal, S\. Zhang, S\. R\. Indurthi, S\. Zhao, K\. Song, S\. Xu, and C\. Zhu \(2024\)Wpo: enhancing rlhf with weighted preference optimization\.arXiv preprint arXiv:2406\.11827\.Cited by:[§6](https://arxiv.org/html/2607.16240#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Results

We provide results with and without regularization for length\-normalized DPO\.

Table 4:AlpacaEval scores for length\-normalized DPO with and without regularization\. WR corresponds to the raw win\-rate and LC corresponds to the length\-controlled win\-rate\.Mistral\-7B\-InstructAlpacaEval1AlpacaEval2 \(WR\)AlpacaEval2 \(LC\)λ=0\\lambda=093\.0316\.5420\.21N\-DPO94\.2818\.1622\.10
Llama\-3\.1\-8B\-InstructAlpacaEval1AlpacaEval2 \(WR\)AlpacaEval2 \(LC\)λ=0\\lambda=091\.2530\.5628\.50N\-DPO91\.2831\.8429\.41
OLMo\-7B\-SFTAlpacaEval1AlpacaEval2 \(WR\)AlpacaEval2 \(LC\)λ=0\\lambda=078\.438\.767\.34N\-DPO81\.249\.218\.02
We provide AlpacaEval scores for the DPO\+SFT method in Table[5](https://arxiv.org/html/2607.16240#A1.T5)\.

Table 5:AlpacaEval scores \(win\-rate\) across methods\.Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTAlpacaEval1AlpacaEval1AlpacaEval1Reference93\.1790\.0058\.15DPO94\.6691\.6779\.63N\-DPO94\.2891\.2881\.24DPO\+SFT92\.1593\.5266\.50We report the sensitivity of reward accuracy and reward margins to the regularization strengthλ\\lambdain Tables[6](https://arxiv.org/html/2607.16240#A1.T6)and[7](https://arxiv.org/html/2607.16240#A1.T7)\.

Table 6:Sensitivity of reward accuracy to the regularization strengthλ\\lambda\. Each cell reports the reward accuracy of the best hyperparameter configuration \(selected by AlpacaEval1 win\-rate\) at the correspondingλ\\lambda, from a joint sweep overβ\\beta,γ\\gamma, and learning rate\.Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTλ\\lambdaN\-DPON\-SimPON\-DPON\-SimPON\-DPON\-SimPO0\.02584\.6182\.1879\.0578\.5975\.9865\.220\.0584\.2082\.0679\.1778\.9475\.8763\.250\.183\.2881\.7778\.9479\.3476\.1662\.62Table 7:Sensitivity of reward margins to the regularization strengthλ\\lambda, under the same evaluation setup as Table[6](https://arxiv.org/html/2607.16240#A1.T6)\. Margins decrease monotonically withλ\\lambdafor both methods, as expected: stronger regularization limits how far the model can separate chosen from rejected responses\.Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTλ\\lambdaN\-DPON\-SimPON\-DPON\-SimPON\-DPON\-SimPO0\.0252\.3557\.2233\.3016\.1432\.5880\.3180\.052\.2387\.0843\.2215\.7842\.4440\.2680\.12\.0926\.8563\.0505\.4392\.2990\.213Outlier token comparisons for DPO vs\. N\-DPO, and SimPO vs\. N\-SimPO for both Mistral\-7B\-Instruct\-v0\.2 and OLMo\-7B\-SFT\-hf\.

### A\.1Mistral\-7B\-Instruct\-v0\.2

Table 8:Outlier category distribution: DPO vs N\-DPO for Mistral\-7B\-Instruct\-v0\.2\.CategoryDPON\-DPOChangeCount%Count%\(Δ\\Delta%\)function\_word54218\.857019\.4\+0\.6number692\.4672\.3\-0\.1other\_single\_char00\.030\.1\+0\.1punctuation35112\.233711\.5\-0\.7single\_letter1575\.51475\.0\-0\.4special\_token461\.6361\.2\-0\.4subword\_fragment154153\.5154952\.8\-0\.7symbol180\.6210\.7\+0\.1whitespace1565\.42036\.9\+1\.5Table 9:Outlier summary: DPO vs N\-DPO or Mistral\-7B\-Instruct\-v0\.2\.StatisticDPON\-DPOTotal tokens58,65558,655Num outliers2,8802,933Outlier %4\.95\.0Threshold\-2\.0000\-3\.3151Reward mean\-0\.2955\-0\.4805Reward std1\.05741\.5650Outlier reward mean\-4\.0886\-6\.0234Outlier reward min\-22\.6446\-40\.2568Table 10:Shared and unique outlier tokens: DPO vs N\-DPO for Mistral\-7B\-Instruct\-v0\.2\.TokenCategoryCountShared outliers\.punctuation135<newline\>whitespace117thefunction\_word77,punctuation60asingle\_letter53andfunction\_word50</s\>special\_token46infunction\_word35isfunction\_word33<whitespace\>whitespace31DPO onlypowerfulsubword\_fragment3hsingle\_letter3artsubword\_fragment2actuallysubword\_fragment1personalitysubword\_fragment1Becausesubword\_fragment1Silsubword\_fragment1‘,subword\_fragment1parsubword\_fragment1glesubword\_fragment1N\-DPO onlydependsubword\_fragment2technologysubword\_fragment2thingssubword\_fragment1equationssubword\_fragment1rotsubword\_fragment1accessiblesubword\_fragment1customsubword\_fragment1staysubword\_fragment1\]\);punctuation1Listsubword\_fragment1Table 11:Outlier category distribution: SimPO vs N\-SimPO for Mistral\-7B\-Instruct\-v0\.2\.CategorySimPON\-SimPOChangeCount%Count%\(Δ\\Delta%\)function\_word72824\.868523\.4\-1\.5number301\.0361\.2\+0\.2punctuation44015\.042014\.3\-0\.7single\_letter1665\.71685\.7\+0\.1special\_token130\.4160\.6\+0\.1subword\_fragment136446\.5142548\.6\+2\.1symbol130\.4130\.4\+0\.0whitespace1796\.11705\.8\-0\.3Table 12:Outlier summary: SimPO vs N\-SimPO for Mistral\-7B\-Instruct\-v0\.2\.StatisticSimPON\-SimPOTotal tokens58,65558,655Num outliers2,9332,933Outlier %5\.05\.0Threshold\-6\.1987\-5\.4903Reward mean\-0\.9325\-0\.8133Reward std2\.49942\.2270Outlier reward mean\-9\.5135\-8\.5701Outlier reward min\-31\.5423\-32\.0055Table 13:Shared and unique outlier tokens: SimPO vs N\-SimPO for Mistral\-7B\-Instruct\-v0\.2TokenCategoryCountShared outliers\.punctuation199<newline\>whitespace147thefunction\_word118,punctuation95asingle\_letter90andfunction\_word87isfunction\_word47thatfunction\_word41tofunction\_word38infunction\_word36SimPO onlyByfunction\_word2oversubword\_fragment2knowssubword\_fragment1approachsubword\_fragment1wagessubword\_fragment1stoppingsubword\_fragment1applicationsubword\_fragment1latestsubword\_fragment1touchsubword\_fragment1firstsubword\_fragment1N\-SimPO onlycertainsubword\_fragment2Becausesubword\_fragment1",punctuation1seemssubword\_fragment1emsubword\_fragment1followsubword\_fragment1bsingle\_letter1Chesubword\_fragment1Whosubword\_fragment1distinguishsubword\_fragment1
### A\.2OLMo\-7B\-SFT\-hf

Table 14:Outlier category distribution: DPO vs N\-DPO for OLMo\-7B\-SFT\-hf\.CategoryDPON\-DPOChangeCount%Count%\(Δ\\Delta%\)content\_word36235\.6133948\.4\+12\.8function\_word13913\.741014\.8\+1\.1number383\.7752\.7\-1\.0other\_single\_char30\.360\.2\-0\.1punctuation706\.91786\.4\-0\.5single\_letter545\.31204\.3\-1\.0special\_token373\.6582\.1\-1\.5subword\_fragment929\.12609\.4\+0\.3symbol212\.1692\.5\+0\.4whitespace20019\.72499\.0\-10\.7Table 15:Outlier summary: DPO vs N\-DPO for OLMo\-7B\-SFT\-hf\.StatisticDPON\-DPOTotal tokens55,26155,261Num outliers1,0162,764Outlier %1\.85\.0Threshold\-2\.0000\-2\.4403Reward mean\-0\.1585\-0\.3777Reward std1\.38221\.3455Outlier reward mean\-5\.9049\-4\.5450Outlier reward min\-57\.3944\-27\.8938Table 16:Shared and unique outlier tokens: DPO vs N\-DPO for OLMo\-7B\-SFT\-hf\.TokenCategoryCountShared outliers<newline\>whitespace197<\|endoftext\|\>special\_token37\.punctuation24<space\>Isingle\_letter18<space\>thefunction\_word13U\+FFFDsymbol12<space\>isfunction\_word11<space\>andfunction\_word101number8‘‘‘subword\_fragment8DPO only<space\>85number3’dsubword\_fragment2Currentsubword\_fragment1<space\>psingle\_letter1Porsubword\_fragment1<space\>generatecontent\_word1<space\>Partcontent\_word1<space\>untilcontent\_word1<space\>Fellowcontent\_word1<space\>confidencecontent\_word1N\-DPO only<space\>wecontent\_word12<space\>havefunction\_word12<space\>morecontent\_word8<space\>Byfunction\_word7<space\>knowcontent\_word6<space\>helpcontent\_word5<space\>themcontent\_word53number5Cellsubword\_fragment5magneticsubword\_fragment5Table 17:Outlier category distribution: SimPO vs N\-SimPO for OLMo\-7B\-SFT\-hf\.CategorySimPON\-SimPOChangeCount%Count%\(Δ\\Delta%\)content\_word14933\.920447\.8\+13\.9function\_word4610\.47517\.6\+7\.1number122\.7133\.0\+0\.3punctuation245\.5368\.4\+3\.0single\_letter4610\.44911\.5\+1\.0special\_token102\.392\.1\-0\.2subword\_fragment133\.0174\.0\+1\.0symbol10\.220\.5\+0\.2whitespace13931\.6225\.2\-26\.4Table 18:Outlier summary: SimPO vs N\-SimPO for OLMo\-7B\-SFT\-hf\.StatisticSimPON\-SimPOTotal tokens55,26155,261Num outliers440427Outlier %0\.80\.8Threshold\-2\.0000\-2\.0000Reward mean\-0\.0609\-0\.0576Reward std0\.51550\.4505Outlier reward mean\-4\.1299\-3\.2282Outlier reward min\-15\.0571\-9\.6681Table 19:Shared and unique outlier tokens: SimPO vs N\-SimPO for OLMo\-7B\-SFT\-hf\.TokenCategoryCountShared outliers<newline\>whitespace139<space\>Isingle\_letter41<\|endoftext\|\>special\_token10\.punctuation9<space\>youcontent\_word8\!punctuation7<space\>andfunction\_word7<space\>herecontent\_word7’msubword\_fragment62number6SimPO only<space\>Howcontent\_word1<space\>Partcontent\_word1N\-SimPO only<space\>isfunction\_word3<space\>basedcontent\_word2<space\>notfunction\_word2<space\>writecontent\_word1<space\>Greatcontent\_word1<space\>mayfunction\_word1<space\>certainlycontent\_word1<space\>modelscontent\_word1<space\>servicescontent\_word130number1

## Appendix BHyperparameters

We provide the set of hyperparameters used to perform hyperparameter sweeps for each method and model\.

Mistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTDPO\[0\.03, 0\.1, 0\.3\]\[0\.01, 0\.03, 0\.1\]\[0\.03, 0\.1, 0\.3\]N\-DPO\[0\.3, 1\.0, 3\.0\]\[0\.3, 1\.0, 3\.0\]\[0\.3, 1\.0, 3\.0\]SimPO\[0\.3, 1\.0, 3\.0\]\[0\.3, 1\.0, 3\.0\]\[0\.3, 1\.0, 3\.0\]N\-SimPO\[3\.0\]\[3\.0\]\[0\.3\]Table 20:Set ofβ\\betaused for hyperparameter sweep for each model/methodMistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTSimPO\[0\.4, 0\.8, 1\.2, 1\.6, 2\.0\]\[0\.4, 0\.8, 1\.2, 1\.6, 2\.0\]\[0\.4, 0\.8, 1\.2, 1\.6, 2\.0\]N\-SimPO\[0\.8, 1\.2, 1\.6, 2\.0\]\[0\.8, 1\.2, 1\.6, 2\.0\]\[0\.8, 1\.2, 1\.6, 2\.0\]Table 21:Set ofγ\\gammaused for hyperparameter sweep for each model/methodMistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTN\-DPO\[0\.0, 0\.025, 0\.05, 0\.075, 0\.1\]\[0\.0, 0\.025, 0\.05, 0\.075, 0\.1\]\[0\.0, 0\.025, 0\.05, 0\.075, 0\.1\]N\-SimPO\[0\.025, 0\.05, 0\.075, 0\.1\]\[0\.025, 0\.05, 0\.075, 0\.1\]\[0\.025, 0\.05, 0\.075, 0\.1\]Table 22:Set ofλ\\lambdaused for hyperparameter sweep for each model/methodMistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFT\[3​e−83e\-8,1​e−71e\-7,3​e−73e\-7\]\[1​e−71e\-7,3​e−73e\-7,1​e−61e\-6\]\[1​e−71e\-7,3​e−73e\-7,1​e−61e\-6\]Table 23:Set of learning rates used for hyperparameter sweep for each modelMistral\-7B\-InstructLlama\-3\.1\-8B\-InstructOLMo\-7B\-SFTβ\\beta\(0\.03/1\.0/3\.0/3\.0\)\(0\.01/3\.0/3\.0/3\.0\)\(0\.03/3\.0/0\.3/0\.3\)γ\\gamma\(0/0/1\.6/2\.0\)\(0/0/0\.8/2\.0\)\(0/0/0\.4/1\.6\)λ\\lambda\(0/0\.1/0/0\.1\)\(0/0\.025/0/0\.025\)\(0/0\.05/0/0\.025\)Learning Rate\(1e\-7/3e\-7/1e\-7/1e\-7\)\(3e\-7/1e\-6/3e\-7/1e\-6\)\(3e\-7/1e\-6/1e\-7/3e\-7\)Table 24:Hyperparameters used for model evaluation\. \(DPO/N\-DPO/SimPO/N\-SimPO\)
## Appendix CGradient Derivation

We provide a derivation of the gradient of the regularization term

ℛ​\(πθ,πref,x,yw,yl\)=λ​\(log⁡π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)π¯ref​\(yw\|x\)\+π¯ref​\(yl\|x\)\)2\\mathcal\{R\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\},x,y\_\{w\},y\_\{l\}\)=\\lambda\\left\(\\log\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\)^\{2\}\(17\)with respect to parametersθ\\theta\. Lettingg​\(πθ\)=\(log⁡π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)π¯ref​\(yw\|x\)\+π¯ref​\(yl\|x\)\)g\(\\pi\_\{\\theta\}\)=\\left\(\\log\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}\\right\), we have that

∇θℛ=2​λ​g​\(πθ\)​∇θg​\(πθ\)\\nabla\_\{\\theta\}\\mathcal\{R\}=2\\lambda g\(\\pi\_\{\\theta\}\)\\nabla\_\{\\theta\}g\(\\pi\_\{\\theta\}\)\(18\)Now, definingh​\(πθ\)=π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)π¯ref​\(yw\|x\)\+π¯ref​\(yl\|x\)h\(\\pi\_\{\\theta\}\)=\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\text\{ref\}\}\(y\_\{l\}\|x\)\}and denoting the denominator ofh​\(x\)h\(x\)asPref​\(x\)P\_\{\\text\{ref\}\}\(x\), we have

∇θg​\(πθ\)=1h​\(πθ\)​∇θh​\(πθ\)=Pref​\(x\)π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)​1Pref​\(x\)​∇θ\(π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)\)\\nabla\_\{\\theta\}g\(\\pi\_\{\\theta\}\)=\\frac\{1\}\{h\(\\pi\_\{\\theta\}\)\}\\nabla\_\{\\theta\}h\(\\pi\_\{\\theta\}\)=\\frac\{P\_\{\\text\{ref\}\}\(x\)\}\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\\frac\{1\}\{P\_\{\\text\{ref\}\}\(x\)\}\\nabla\_\{\\theta\}\(\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\)\(19\)Writingπ¯θ​\(y\|x\)\\bar\{\\pi\}\_\{\\theta\}\(y\|x\)asπθ​\(y\|x\)1/\|y\|\\pi\_\{\\theta\}\(y\|x\)^\{1/\|y\|\}and decomposingπθ​\(y\|x\)1/\|y\|\\pi\_\{\\theta\}\(y\|x\)^\{1/\|y\|\}as

exp⁡\(1\|y\|​∑i=1\|y\|log⁡πθ​\(y\(i\)\)\)\\exp\\left\(\\frac\{1\}\{\|y\|\}\\sum\_\{i=1\}^\{\|y\|\}\\log\\pi\_\{\\theta\}\(y^\{\(i\)\}\)\\right\)\(20\)we have

∇θ\(π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)\)=\(π¯θ​\(yw\|x\)\|yw\|​∑i=1\|yw\|∇θπθ​\(yw\(i\)\)πθ​\(yw\(i\)\)\+π¯θ​\(yl\|x\)\|yl\|​∑i=1\|yl\|∇θπθ​\(yl\(i\)\)πθ​\(yl\(i\)\)\)\\nabla\_\{\\theta\}\(\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\)=\\left\(\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\}\{\|y\_\{w\}\|\}\\sum\_\{i=1\}^\{\|y\_\{w\}\|\}\\frac\{\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\}\{\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\}\+\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\|y\_\{l\}\|\}\\sum\_\{i=1\}^\{\|y\_\{l\}\|\}\\frac\{\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\}\{\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\}\\right\)\(21\)Through the chain rule and lettingPθ​\(x\)=π¯θ​\(yw\|x\)\+π¯θ​\(yl\|x\)P\_\{\\theta\}\(x\)=\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\+\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\), we have that

∇θℛ=2​λPθ​\(x\)​log⁡\(Pθ​\(x\)Pref​\(x\)\)​\(π¯θ​\(yw\|x\)\|yw\|​∑i=1\|yw\|∇θ\(πθ​\(yw\(i\)\)\)πθ​\(yw\(i\)\)\+π¯θ​\(yl\|x\)\|yl\|​∑i=1\|yl\|∇θ\(πθ​\(yl\(i\)\)\)πθ​\(yl\(i\)\)\)\.\\nabla\_\{\\theta\}\\mathcal\{R\}=\\frac\{2\\lambda\}\{P\_\{\\theta\}\(x\)\}\\log\\left\(\\frac\{P\_\{\\theta\}\(x\)\}\{P\_\{\\text\{ref\}\}\(x\)\}\\right\)\\left\(\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{w\}\|x\)\}\{\|y\_\{w\}\|\}\\sum\_\{i=1\}^\{\|y\_\{w\}\|\}\\frac\{\\nabla\_\{\\theta\}\(\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\)\}\{\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\)\}\+\\frac\{\\bar\{\\pi\}\_\{\\theta\}\(y\_\{l\}\|x\)\}\{\|y\_\{l\}\|\}\\sum\_\{i=1\}^\{\|y\_\{l\}\|\}\\frac\{\\nabla\_\{\\theta\}\(\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\)\}\{\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\)\}\\right\)\.\(22\)