Reliability-Aware LLM Alignment from Inconsistent Human Feedback

# Reliability-Aware LLM Alignment from Inconsistent Human Feedback
Source: [https://arxiv.org/html/2607.20515](https://arxiv.org/html/2607.20515)
###### Abstract

Reinforcement Learning from Human Feedback \(RLHF\) is critical for aligning Large Language Models \(LLMs\) with human preferences\. However, its efficacy is often compromised by the inherent inconsistency and subjectivity of human annotations\. Existing preference optimization frameworks, such as Direct Preference Optimization \(DPO\), typically treat ambiguous pairs with high annotator disagreement identically to those with unanimous consensus, forcing models to overfit to inconsistent supervision signals and leading to suboptimal alignment\. In this work, we proposeReliability\-Guided Preference Optimization\(RGPO\), a robust framework designed to mitigate the impact of inconsistent human feedback\. RGPO estimates annotator reliability and infers latent ground truth labels from noisy human feedback to identify robust preferences\. Furthermore, we introduce a reliability\-aware consistency optimization that dynamically modulates the training objective based on the consensus level of annotations, ensuring the model prioritizes high\-consensus supervision signals\. Extensive experiments on LLM alignment benchmarks demonstrate that RGPO effectively reduces inconsistency and noise in training data and achieves superior performance compared to widely adopted RLHF baselines\. Our code and configurations are available at[https://github\.com/GenieHuang/RGPO](https://github.com/GenieHuang/RGPO)\.

Machine Learning, ICML

## 1Introduction

With the widespread adoption of Large Language Models \(LLMs\) across diverse domains, ranging from creative writing to complex decision support, it has become an essential objective to ensure these systems are both helpful and harmless\(Askell et al\.,[2021](https://arxiv.org/html/2607.20515#bib.bib1)\)\. As LLMs become increasingly capable, the focus shifts from generating coherent text to aligning model outputs with complex human preferences and intents\(Christiano et al\.,[2017](https://arxiv.org/html/2607.20515#bib.bib8); Bai et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib3)\)\. Consequently, alignment techniques that leverage human feedback have emerged as essential components in the modern training pipeline\(Glaese et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib17)\)\. They bridge the gap between token prediction and user\-centric interaction\(Ouyang et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib28)\)\. This alignment is critical not only for improving user satisfaction but also for mitigating safety risks such as toxicity or hallucination, which cannot be easily defined by static rules\(Ganguli et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib15)\)\.

Unlike mathematical reasoning or code generation tasks, which typically have objective golden answers, Reinforcement Learning from Human Feedback \(RLHF\) tasks are predominantly open\-ended\. In this context, there is rarely a single correct response, making the annotation process heavily rely on human annotators to demonstrate or rank preferable behaviors based on nuanced criteria\(Menick et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib26)\)\. To leverage the human preference annotations, researchers have developed frameworks such as Proximal Policy Optimization \(PPO\), which trains a separate reward model to provide simulated feedback to the policy model, and constrains the training process by a KL\-divergence penalty to prevent mode collapse\(Schulman et al\.,[2017](https://arxiv.org/html/2607.20515#bib.bib35); Stiennon et al\.,[2020](https://arxiv.org/html/2607.20515#bib.bib38)\)\. Alternatively, Direct Preference Optimization \(DPO\) has emerged by eliminating the reward model entirely\. It constructs “chosen” and “rejected” pairs to directly optimize the policy based on the likelihood margin between preferred and dispreferred responses\(Rafailov et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib31)\)\. However, both approaches face limitations: PPO requires significant computational resources to maintain separate reward and value models, while DPO’s efficacy is critically sensitive to the quality and reliability of the constructed preference pairs\(Azar et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib2); Meng et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib25)\)\.

![Refer to caption](https://arxiv.org/html/2607.20515v1/assets/overview_rgpo.jpg)Figure 1:Uniform Ensemble ignores the varying reliability levels of annotators, while existing RLHF frameworks such as DPO treat samples with strong disagreement identically to those with unanimous preference, thereby introducing potential inconsistency\.A more fundamental challenge lies in the nature of the annotation itself\. Because RLHF tasks are open\-ended, the quality of the supervision signal depends heavily on the capability, attention, and cultural background of the annotators, creating significant variance in their reliability\(Sap et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib34); Santurkar et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib33)\)\. Beyond these individual differences, human annotations are inherently subjective, and potential inconsistency is introduced in this process\(Bakker et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib4)\)\. This inconsistency often manifests through high levels of disagreement among different annotators, where noise becomes inevitable due to conflicting personal preferences or genuine divergence regarding ambiguous prompts\(Chowdhury et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib7)\)\. Yet, most existing frameworks overlook this complexity by aggregating conflicting judgments into binary labels\(Gordon et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib18)\)\. For instance, as illustrated in Figure[1](https://arxiv.org/html/2607.20515#S1.F1), if a pair of responses receives a contentious 3\-to\-2 vote, existing methods treat it identically to a unanimous 5\-to\-0 vote, forcing the model to maximize the margin between the chosen and rejected response with full confidence\. This disregard for reliability and uncertainty leads to models that overfit to inconsistency, learning with unjustified confidence from ambiguous data and potentially degrading performance on clear instructions\(Gao et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib16)\)\.

To address these challenges, we propose a framework that explicitly models annotator reliability to resolve data inconsistency\. We apply maximum likelihood estimation directly to the LLM alignment loop to statistically distinguish credible signals from noise\(Raykar et al\.,[2010](https://arxiv.org/html/2607.20515#bib.bib32)\)\. By jointly estimating latent ground truth and annotator reliability, we identify high\-confidence preference labels rather than relying on simple aggregation\. Additionally, we regulate the training process by adaptive scaling based on the level of reliability\-aware consistency to prioritize feedback from the most reliable sources\. Our key contributions are summarized as follows:

- •We propose the*Reliability\-GuidedPreferenceOptimization*\(RGPO\) framework to systematically mitigate the performance degradation caused by subjective noise and annotator inconsistency in multi\-annotated RLHF datasets\.
- •We propose*Iterative Latent Reliability Estimation*to resolve the ambiguity of inconsistent annotations, leveraging iterative maximum likelihood estimation to infer annotator reliability with latent ground truth labels\.
- •We design*Reliability\-Aware Consistency Optimization*to prevent the model from overfitting to inconsistent data, dynamically scaling the optimization objective based on the reliability\-based consensus level of supervision signals\.
- •We conduct extensive experiments to demonstrate that enhancing baselines with RGPO leads to superior stability and alignment performance compared to the standard baselines\.

## 2Related Work

#### Reinforcement Learning from Human Feedback\.

Reinforcement Learning from Human Feedback \(RLHF\) has become the cornerstone for aligning Large Language Models \(LLMs\) with human preference\(Christiano et al\.,[2017](https://arxiv.org/html/2607.20515#bib.bib8)\)\. It shifts models from text generators to helpful assistants\(Ouyang et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib28); Bai et al\.,[2022](https://arxiv.org/html/2607.20515#bib.bib3)\)\. The standard pipeline typically involves training a reward model on human preference pairs followed by Proximal Policy Optimization \(PPO\)\(Schulman et al\.,[2017](https://arxiv.org/html/2607.20515#bib.bib35); Stiennon et al\.,[2020](https://arxiv.org/html/2607.20515#bib.bib38)\)\. However, the efficacy of RLHF highly depends on the quality of the underlying human data, which is often inconsistent and inherently subjective\(Casper et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib6); Fernandes et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib14)\)\. Recent studies have demonstrated that human annotations frequently contain significant noise due to task ambiguity, annotator fatigue, or lack of domain expertise\(Larson et al\.,[2019](https://arxiv.org/html/2607.20515#bib.bib21)\)\. Specifically, some research highlights that inconsistent preference labels, where the annotated “winner” contradicts the underlying ground truth, can catastrophically degrade the performance of aligned models\(Chowdhury et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib7)\)\. Despite this, most of the existing RLHF frameworks treat annotations as ground truth directly or aggregate conflicting labels using a simple uniform ensemble\(Wu et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib44)\), ignoring the variance of annotator reliability\(Peng et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib29)\)\. While some recent approaches have attempted to model inconsistency via robust loss functions\(Liang et al\.,[2025](https://arxiv.org/html/2607.20515#bib.bib23); Zhu et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib48)\), they often fail to explicitly model the latent labels distinct from the observed noisy labels, particularly in multi\-annotator settings where disagreement frequently occurs\(Uma et al\.,[2021](https://arxiv.org/html/2607.20515#bib.bib40); Plank,[2022](https://arxiv.org/html/2607.20515#bib.bib30)\)\. In contrast, our RGPO framework addresses this limitation with latent reliability estimation to identify the ground truth and guide policy optimization with robust supervision signals\(Dawid & Skene,[1979](https://arxiv.org/html/2607.20515#bib.bib9)\)\.

#### Direct Preference Optimization and derivative alignment frameworks\.

To circumvent the instability and computational complexity of PPO, Direct Preference Optimization \(DPO\) was introduced to optimize the policy directly on paired preference data\(Rafailov et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib31)\)\. The training objective is formulated as:

ℒDPO\(πθ;πref\)=−𝔼\(q,yw,yl\)∼𝒟\[logσ\(βlogπθ​\(yw\|q\)πref​\(yw\|q\)−βlogπθ​\(yl\|q\)πref​\(yl\|q\)\)\]\\small\\begin\{split\}\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\pi\_\{\\theta\};\\pi\_\{\\text\{ref\}\}\)=\-\\mathbb\{E\}\_\{\(q,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\Big\[\\log\\sigma\\Big\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}\|q\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}\|q\)\}\\\\ \-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{l\}\|q\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}\|q\)\}\\Big\)\\Big\]\\end\{split\}\(1\)This shift has led to the development of numerous derivative frameworks aiming to enhance stability and performance\(Zhao et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib47); Swamy et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib39)\)\. For instance, IPO\(Azar et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib2)\)defines a squared error objective on the log\-likelihood ratio, effectively regressing the preference gap to a fixed margin\. Meanwhile, KTO\(Ethayarajh et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib13)\)eliminates the need for paired data entirely by adopting a prospect\-theoretic approach\. Further advancements include SimPO\(Meng et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib25)\), which removes the reference model to improve memory efficiency\. Its training objective is defined as:

ℒSimPO\(πθ\)=−𝔼\(q,yw,yl\)∼𝒟\[logσ\(β\|yw\|logπθ\(yw\|q\)−β\|yl\|logπθ\(yl\|q\)−γ\)\]\\small\\begin\{split\}\\mathcal\{L\}\_\{\\text\{SimPO\}\}\(\\pi\_\{\\theta\}\)=\-\\mathbb\{E\}\_\{\(q,y\_\{w\},y\_\{l\}\)\\sim\\mathcal\{D\}\}\\Big\[\\log\\sigma\\Big\(\\frac\{\\beta\}\{\|y\_\{w\}\|\}\\log\\pi\_\{\\theta\}\(y\_\{w\}\|q\)\\\\ \-\\frac\{\\beta\}\{\|y\_\{l\}\|\}\\log\\pi\_\{\\theta\}\(y\_\{l\}\|q\)\-\\gamma\\Big\)\\Big\]\\end\{split\}\(2\)Additionally, ORPO\(Hong et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib19)\)integrates preference alignment directly into the pre\-training or supervised fine\-tuning stage\. Despite these architectural innovations, a critical limitation persists across these methods: they largely treat all preference samples as equally informative\(Liu et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib24); Yuan et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib46)\)\. Current loss functions typically assign a uniform weight to every preference pair, disregarding the reliability level of annotators or the consistency of annotations\(Santurkar et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib33); Dong et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib10)\)\. Consequently, a pair with unanimous expert agreement is optimized with the same gradient magnitude as a controversial pair with significant disagreement among annotators\(Sharma et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib36)\)\. This uniform treatment is suboptimal for noisy datasets, as it forces the model to overfit to ambiguous or potentially incorrect labels\(Xu et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib45); Song et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib37)\)\. Instead, RGPO employs consistency\-weighted optimization based on annotator reliability to fully leverage an adaptive training mechanism\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2607.20515v1/assets/pipeline_rgpo.jpg)Figure 2:*Iterative Latent Reliability Estimation*\(blue box\) and*Reliability\-Aware Consistency Optimization*\(green box\)\. To mitigate the inconsistency introduced during the annotation process, RGPO utilizes iterative estimation to infer annotator reliability, and optimizes the policy with scaled consistency weight to prioritize robust supervision signals\.As shown in Figure[2](https://arxiv.org/html/2607.20515#S3.F2), RGPO has two main modules to process inconsistent preference data and optimize the policy:

- •*Iterative Latent Reliability Estimation*to apply iterative maximum likelihood estimation for annotator reliability assessment with latent ground truth identification from conflicting multi\-annotator judgments\.
- •*Reliability\-Aware Consistency Optimization*to calculate the scaled reliability\-based consistency weight of the annotator distribution for each sample and modulate the gradient magnitude of the optimization objective during training\.

### 3\.1Iterative Latent Reliability Estimation

Human preference data collected from annotation platforms often contains significant annotator disagreement, especially for controversial or subjective topics, where multiple annotators provide conflicting preferences for the same comparison\. To address this inconsistency, we propose a principled probabilistic strategy that explicitly models annotator reliability, utilizing maximum likelihood estimation to jointly and iteratively recover reliability scores and latent preferences\.

Given a queryqq, the pool of annotators is denoted by\{a1,a2,…,aM\}\\\{a\_\{1\},a\_\{2\},\\ldots,a\_\{M\}\\\}and two candidate responses are denoted byrAr\_\{A\}andrBr\_\{B\}\. Each annotatoraja\_\{j\}is asked to evaluate each response and provide scores for bothrAr\_\{A\}andrBr\_\{B\}, or a direct preference label for the comparison\. Letyjy\_\{j\}denote the absolute difference ofrAr\_\{A\}andrBr\_\{B\}scores or the direct preference label provided by the annotatoraja\_\{j\}\. We projectyjy\_\{j\}to a 3\-scale voting withyj∈\{−1,0,\+1\}y\_\{j\}\\in\\\{\-1,0,\+1\\\}, where\+1\+1indicates preference for responserAr\_\{A\},−1\-1indicates preference for responserBr\_\{B\}, and0indicates a tie \(i\.e\., the annotator perceives both responses as equally good\)\.𝒜i⊆\{a1,…,aM\}\\mathcal\{A\}\_\{i\}\\subseteq\\\{a\_\{1\},\\ldots,a\_\{M\}\\\}is used to denote the set of annotators who labeled the comparisonii, andyi​jy\_\{ij\}to denote the label provided by annotatoraja\_\{j\}for comparisonii\.

#### Assumption: A latent true preference exists\.

We assume that there exists a latent true preference for each comparison that reflects the ground\-truth quality difference between the two candidate responses\. This true preference is unobserved, and our goal is to infer it from the inconsistent annotations while simultaneously estimating each annotator’s reliability\. This latent preference is not intended to force a single objectively correct answer\. It serves as a statistical estimator that aggregates diverse and potentially conflicting annotations in to a coherent preference signal\. The central insight is that not all annotators are equally reliable: some might have domain expertise and provide more accurate and reliable labels, while others may provide noisy or adversarial labels\.

To capture annotator\-specific labeling behavior, we utilize a confusion matrix𝐂\(j\)∈ℝK×K\\mathbf\{C\}^\{\(j\)\}\\in\\mathbb\{R\}^\{K\\times K\}for each annotatoraja\_\{j\}, whereK=3K=3is the number of possible labelsyj∈\{−1,0,\+1\}y\_\{j\}\\in\\\{\-1,0,\+1\\\}\. The entryCt​k\(j\)C^\{\(j\)\}\_\{tk\}represents the probability that annotatoraja\_\{j\}provides labelkkwhen the true preference istt:

Ct​k\(j\)=P​\(yi​j=k∣Ti=t\)\\small C^\{\(j\)\}\_\{tk\}=P\(y\_\{ij\}=k\\mid T\_\{i\}=t\)\(3\)whereTiT\_\{i\}denotes the latent true preference of comparisonii\. Each row of the confusion matrix sums to one,∑kCt​k\(j\)=1\\sum\_\{k\}C^\{\(j\)\}\_\{tk\}=1for alltt\. The diagonal entriesCt​t\(j\)C^\{\(j\)\}\_\{tt\}represent the probability that annotatoraja\_\{j\}correctly identifies the true preference, while off\-diagonal entries capture systematic errors or biases\. For instance, an annotator who tends to avoid committing to a preference might have elevated values in the column corresponding to ties \(Ct,0\(j\)≫0C^\{\(j\)\}\_\{t,0\}\\gg 0fort≠0t\\neq 0\)\. This setting allows our model to capture diverse annotator behaviors beyond simple accuracy\. Additionally, we use a prior distribution𝝅t\\boldsymbol\{\\pi\}\_\{t\}over the true preference labels, whereπt=P​\(Ti=t\)\\pi\_\{t\}=P\(T\_\{i\}=t\)represents the probability that a randomly selected comparison has true preferencett\. This prior captures the base rates of different preference outcomes in the data and is learned jointly with the confusion matrices\.

#### Maximum likelihood estimation\.

We use𝜽=\{\{𝐂\(j\)\}j=1M,𝝅\}\\boldsymbol\{\\theta\}=\\\{\\\{\\mathbf\{C\}^\{\(j\)\}\\\}\_\{j=1\}^\{M\},\\boldsymbol\{\\pi\}\\\}to denote the complete set of parameters including confusion matrices for annotatorsa1,a2,…,aM\{a\_\{1\},a\_\{2\},\.\.\.,a\_\{M\}\}and prior distributions\. We then estimate these parameters by maximizing the likelihood of the observed annotations\. For a comparisoniiwith annotations\{yi​j\}aj∈𝒜i\\\{y\_\{ij\}\\\}\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}, the marginal likelihood is obtained by summing over all possible values of the latent true preference:

P​\(\{yi​j\}aj∈𝒜i;𝜽\)=∑t∈\{−1,0,\+1\}πt​∏aj∈𝒜iCt,yi​j\(j\)\\small P\(\\\{y\_\{ij\}\\\}\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\};\\boldsymbol\{\\theta\}\)=\\sum\_\{t\\in\\\{\-1,0,\+1\\\}\}\\pi\_\{t\}\\prod\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}C^\{\(j\)\}\_\{t,y\_\{ij\}\}\(4\)The full log\-likelihood over allNNcomparisons is:

ℒ​\(𝜽\)=∑i=1Nlog⁡P​\(\{yi​j\}aj∈𝒜i;𝜽\)=∑i=1Nlog⁡\(∑t∈\{−1,0,\+1\}πt​∏aj∈𝒜iCt,yi​j\(j\)\)\\small\\begin\{split\}\\mathcal\{L\}\(\\boldsymbol\{\\theta\}\)&=\\sum\_\{i=1\}^\{N\}\\log P\(\\\{y\_\{ij\}\\\}\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\};\\boldsymbol\{\\theta\}\)\\\\ &=\\sum\_\{i=1\}^\{N\}\\log\\left\(\\sum\_\{t\\in\\\{\-1,0,\+1\\\}\}\\pi\_\{t\}\\prod\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}C^\{\(j\)\}\_\{t,y\_\{ij\}\}\\right\)\\end\{split\}\(5\)
Directly optimizing this objective is difficult because the true preference labelsTiT\_\{i\}are latent\. To address this, we use an iterative optimization process that alternates between inferring the true preferences and updating the model’s parameters\. In the expectation step, we calculateγi​t\\gamma\_\{it\}, which represents the posterior probability that the true preference for comparisoniiistt, given how well the observed annotations align with each category under the current model\. Theγi​t\\gamma\_\{it\}can be formulated as:

γi​t=P​\(Ti=t∣\{yi​j\}aj∈𝒜i;𝜽\)=πt​∏aj∈𝒜iCt,yi​j\(j\)∑t′∈\{−1,0,\+1\}πt′​∏aj∈𝒜iCt′,yi​j\(j\)\\small\\begin\{split\}\\gamma\_\{it\}&=P\(T\_\{i\}=t\\mid\\\{y\_\{ij\}\\\}\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\};\\boldsymbol\{\\theta\}\)\\\\ &=\\frac\{\\pi\_\{t\}\\prod\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}C^\{\(j\)\}\_\{t,y\_\{ij\}\}\}\{\\sum\_\{t^\{\\prime\}\\in\\\{\-1,0,\+1\\\}\}\\pi\_\{t^\{\\prime\}\}\\prod\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}C^\{\(j\)\}\_\{t^\{\\prime\},y\_\{ij\}\}\}\\end\{split\}\(6\)
In the maximization step, we update the confusion matrices and class priors to maximize the likelihood of the observed data given the estimated true preferences\. Specifically, the update for the confusion matrix entryCt​k\(j\)C^\{\(j\)\}\_\{tk\}aggregates evidence from all comparisons where annotatoraja\_\{j\}participated, which effectively calculates the proportion of times annotatoraja\_\{j\}assigned labelkk, weighted by the probability that the true preference istt\. Simultaneously, the prior distribution is updated as the average posterior\. These updates are formulated as:

Ct​k\(j\)←∑i:aj∈𝒜iγi​t⋅𝕀​\(yi​j=k\)∑i:aj∈𝒜iγi​t\\small C^\{\(j\)\}\_\{tk\}\\leftarrow\\frac\{\\sum\_\{i:a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}\\gamma\_\{it\}\\cdot\\mathbb\{I\}\(y\_\{ij\}=k\)\}\{\\sum\_\{i:a\_\{j\}\\in\\mathcal\{A\}\_\{i\}\}\\gamma\_\{it\}\}\(7\)πt←1N​∑i=1Nγi​t\\small\\pi\_\{t\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\gamma\_\{it\}\(8\)This iterative procedure is guaranteed to monotonically increase the log\-likelihood at each iteration and converge to a local optimum\.

#### Annotator reliability scoreqjq\_\{j\}\.

Upon convergence, we extract a scalar reliability scoreqjq\_\{j\}for each annotatoraja\_\{j\}by averaging the diagonal entries of their learned confusion matrix\.

qj=1K​∑t=1KCt​t\(j\)\\small q\_\{j\}=\\frac\{1\}\{K\}\\sum\_\{t=1\}^\{K\}C^\{\(j\)\}\_\{tt\}\(9\)This scoreqj∈\[0,1\]q\_\{j\}\\in\[0,1\]quantifies the annotator’s overall accuracy across all true preference categories, with higher values indicating more reliable annotators\. Importantly, because the confusion matrices are estimated jointly with the true preferences, reliability scores account for the difficulty of comparison tasks\. That is, an annotator who labeled many ambiguous comparisons will not be unfairly penalized compared to one who labeled easier cases\.

#### Preference pair construction\.

For comparisonii, we determine the predicted preference using the Maximum A Posteriori \(MAP\) estimate\(Bishop & Nasrabadi,[2006](https://arxiv.org/html/2607.20515#bib.bib5)\):

T^i=arg⁡maxt∈\{−1,0,\+1\}⁡γi​t\\small\\hat\{T\}\_\{i\}=\\arg\\max\_\{t\\in\\\{\-1,0,\+1\\\}\}\\gamma\_\{it\}\(10\)The chosen response isrAr\_\{A\}ifT^i=\+1\\hat\{T\}\_\{i\}=\+1, otherwiserBr\_\{B\}\. Comparisons where the model predicts a tie \(T^i=0\\hat\{T\}\_\{i\}=0\) are filtered from the training set, as preference optimization requires a strict preference direction\.

### 3\.2Reliability\-Aware Consistency Optimization

We further extend the derived annotator reliability scores to explicitly modulate the policy optimization\. While the estimation determines the latent preference labels, it does not directly address the challenge of how confident we should be in that preference when training the model\. Therefore, we propose a regulated optimization based on the reliability\-aware consistency measure to gauge supervision quality\. Rather than simply smoothing disagreement through weighted aggregation, this consistency measure evaluates how strongly reliable annotators agree on the preference direction for each comparison\. Since imposing strong updates on inconsistent samples risks overfitting to noise, this regulated optimization dynamically scales the training objective, ensuring that the magnitude of policy update is strictly governed by the reliability\-based consistency of each sample\.

#### Reliability\-based consistency measurement\.

For each comparisonii, we first identify the subset of annotators who expressed a clear directional preference, denoted𝒜i\(≠0\)=\{aj∈𝒜i:yi​j≠0\}\\mathcal\{A\}\_\{i\}^\{\(\\neq 0\)\}=\\\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}:y\_\{ij\}\\neq 0\\\}\. We exclude tie votes from the consistency computation for two reasons: \(i\) a tie vote \(yi​j=0y\_\{ij\}=0\) typically represents annotator uncertainty or indifference rather than a conflicting preference, so including it would confuse neutrality with active disagreement; \(ii\) when computing agreement on preference direction, only votes expressing a direction \(\+1\+1or−1\-1\) provide a relevant signal\. Note that we exclude these tie votes only for the consistency measurement, while all votes remain in use for the prior preference estimation\. Among the annotators with directional votes, we compute normalized reliability weights:

wj=qj∑ak∈𝒜i\(≠0\)qk\\small w\_\{j\}=\\frac\{q\_\{j\}\}\{\\sum\_\{a\_\{k\}\\in\\mathcal\{A\}\_\{i\}^\{\(\\neq 0\)\}\}q\_\{k\}\}\(11\)whereqjq\_\{j\}is the reliability score for annotatoraja\_\{j\}obtained from the maximum likelihood estimation\. This normalization ensures that∑aj∈𝒜i\(≠0\)wj=1\\sum\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}^\{\(\\neq 0\)\}\}w\_\{j\}=1\. Using these reliability\-weighted contributions, we compute the probability massPk\(i\)P\_\{k\}^\{\(i\)\}assigned to each preference directionk∈\{\+1,−1\}k\\in\\\{\+1,\-1\\\}\(corresponding to choosingrAr\_\{A\}andrBr\_\{B\}\) and quantify the resulting uncertainty using binary entropyHiH\_\{i\}:

Pk\(i\)=∑aj∈𝒜i\(≠0\)wj⋅𝕀​\(yi​j=k\)\\small P\_\{k\}^\{\(i\)\}=\\sum\_\{a\_\{j\}\\in\\mathcal\{A\}\_\{i\}^\{\(\\neq 0\)\}\}w\_\{j\}\\cdot\\mathbb\{I\}\(y\_\{ij\}=k\)\(12\)Hi=−∑k∈\{\+1,−1\}Pk\(i\)​log2⁡Pk\(i\)\\small H\_\{i\}=\-\\sum\_\{k\\in\\\{\+1,\-1\\\}\}P\_\{k\}^\{\(i\)\}\\log\_\{2\}P\_\{k\}^\{\(i\)\}\(13\)Since tie votes are excluded and weights are normalized, the probabilities sum to one \(∑Pk\(i\)=1\\sum P\_\{k\}^\{\(i\)\}=1\)\. Finally, we define the consistency weightsis\_\{i\}as the complement of the entropy:si=1−His\_\{i\}=1\-H\_\{i\}\. This metricsi∈\[0,1\]s\_\{i\}\\in\[0,1\]serves as a direct measure of agreement: it reaches its peak \(si=1s\_\{i\}=1\) when there is perfect consensus among reliable annotators and is minimized \(si=0s\_\{i\}=0\) when there is maximum disagreement\.

#### Consistency weight scaling\.

To fully adapt the consistency measurement to the training process, we calibrate the contribution of each comparison based on annotator consensus using weighting\. The weights~i\\tilde\{s\}\_\{i\}is formulated by:

s~i=12​\(tanh⁡\(siλ\)\+1\)\\small\\tilde\{s\}\_\{i\}=\\frac\{1\}\{2\}\\left\(\\tanh\\left\(\\frac\{s\_\{i\}\}\{\\lambda\}\\right\)\+1\\right\)\(14\)whereλ\>0\\lambda\>0is a scaling coefficient to control the steepness of this differentiation\. A smallλ\\lambdaaggressively prioritizes high\-confidence samples, while a large one produces a flatter distribution\. This formulation maps the input within\[0,1\]\[0,1\], and assigns higher weights to comparisons where reliable annotators strongly agree \(si→1s\_\{i\}\\to 1\) and lower weights to comparisons with high disagreement\.

The scaled weightss~i\\tilde\{s\}\_\{i\}are incorporated into the DPO objective as sample\-specific loss multipliers\. Let\(qi,rA\(i\),rB\(i\)\)\(q\_\{i\},r\_\{A\}^\{\(i\)\},r\_\{B\}^\{\(i\)\}\)denote the query and two responses for comparisonii, and let\(yw\(i\),yl\(i\)\)\(y\_\{w\}^\{\(i\)\},y\_\{l\}^\{\(i\)\}\)denote the chosen and rejected responses as determined by the MAP estimateT^i\\hat\{T\}\_\{i\}\. The reliability\-weighted DPO loss is:

ℒRGPO=−𝔼\(qi,yw\(i\),yl\(i\)\)∼𝒟\[s~i⋅logσ\(βlogπθ​\(yw\(i\)∣qi\)πref​\(yw\(i\)∣qi\)−βlogπθ​\(yl\(i\)∣qi\)πref​\(yl\(i\)∣qi\)\)\]\\small\\begin\{split\}\\mathcal\{L\}\_\{\\text\{RGPO\}\}=\-\\mathbb\{E\}\_\{\(q\_\{i\},y\_\{w\}^\{\(i\)\},y\_\{l\}^\{\(i\)\}\)\\sim\\mathcal\{D\}\}\\Big\[\\tilde\{s\}\_\{i\}\\cdot\\log\\sigma\\Big\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{w\}^\{\(i\)\}\\mid q\_\{i\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{w\}^\{\(i\)\}\\mid q\_\{i\}\)\}\\\\ \-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{l\}^\{\(i\)\}\\mid q\_\{i\}\)\}\{\\pi\_\{\\text\{ref\}\}\(y\_\{l\}^\{\(i\)\}\\mid q\_\{i\}\)\}\\Big\)\\Big\]\\end\{split\}\(15\)whereπθ\\pi\_\{\\theta\}is the policy being optimized,πref\\pi\_\{\\text\{ref\}\}is the reference policy, andβ\\betais the KL\-divergence constraint parameter\. By scaling the gradient contribution withs~i\\tilde\{s\}\_\{i\}, the model effectively focuses on learning from high\-reliability signals while down\-weighting ambiguous or controversial comparisons\.

## 4Experiments

### 4\.1Experiments Settings

#### Training datasets\.

We perform our experiments using two human\-annotated preference datasets with multiple judgments: MultiPref\(Miranda et al\.,[2025](https://arxiv.org/html/2607.20515#bib.bib27)\)and HelpSteer2\(Wang et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib41)\)\. MultiPref contains 10,461 pairwise comparison samples from 227 annotators, including both normal annotators and domain experts\. For HelpSteer2, we use the disagreement collection\(Wang et al\.,[2025](https://arxiv.org/html/2607.20515#bib.bib42)\), which consists of 11,824 paired samples from 6 annotators\. For both datasets, we focus exclusively on the “helpfulness” annotation dimension\.

#### Hyperparameters\.

Across all groups of baselines and RGPO\-enhanced models, we maintain identical training hyperparameters to ensure fair comparisons\. We employ Low\-Rank Adaptation \(LoRA\) for parameter\-efficient fine\-tuning, and use differentβ\\betato adapt different algorithms and base models\. All training runs are with 3NVIDIA RTX 6000 AdaGPUs\. See Appendix[A](https://arxiv.org/html/2607.20515#A1)for detailed training settings\.

#### Evaluation benchmarks\.

We evaluate the performance of aligned models using two widely used benchmarks: AlpacaEval 2\(Dubois et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib11),[2024](https://arxiv.org/html/2607.20515#bib.bib12)\)and Arena\-Hard\(Li et al\.,[2025](https://arxiv.org/html/2607.20515#bib.bib22)\)\. AlpacaEval 2 assesses conversational quality by computing win rates againstgpt\-4\-1106\-preview, reporting both the raw win rate and the length\-controlled win rate to mitigate verbosity biases\. Arena\-Hard evaluates the model’s capability on challenging user queries requiring complex reasoning againstgpt\-4\-0314\. See Appendix[A](https://arxiv.org/html/2607.20515#A1)for detailed inference and evaluation settings\.

#### Baselines\.

To evaluate the effectiveness of our proposed framework, we integrate RGPO into widely adopted RLHF methods: Direct Preference Optimization \(DPO\)\(Rafailov et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib31)\), Simple Preference Optimization \(SimPO\)\(Meng et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib25)\)and Identity\-PO \(IPO\)\(Azar et al\.,[2024](https://arxiv.org/html/2607.20515#bib.bib2)\)\. We employ uniform ensemble for baseline preference construction and conduct these evaluations across two base models,Llama\-3\-8B\-InstructandQwen2\.5\-7B\-Instruct\. We systematically compare the performance of each baseline with and without the RGPO to quantify improvements in preference alignment\. For fair comparisons and strict chosen\-rejected pair construction, we filter out all tie votes for both baselines and RGPO\-enhanced models to strictly establish the direction of preference, since tie votes do not provide a valid pairwise preference signal\.

### 4\.2Main Results

As illustrated in Table[1](https://arxiv.org/html/2607.20515#S4.T1), we evaluate the performance of RGPO\-enhanced models compared with models trained with standard RLHF frameworks\. We summarize our key observations as follows:

Effectiveness of RGPO framework\.The consistent performance improvements observed across all benchmarks demonstrate the efficacy of our approach\. By integrating RGPO, the model achieves superior Win Rates compared to the standard DPO, SimPO and IPO baselines\. This indicates that reliability\-aware supervision effectively steers the model towards human\-preferred behaviors\. Notably, improvements are evident across different frameworks: RGPO raises the DPO baseline’s Arena\-Hard Win Rate from43\.70%43\.70\\%to46\.60%46\.60\\%, and IPO’s from51\.90%51\.90\\%to53\.10%53\.10\\%on MultiPref with Llama\-3\-Instruct model\. The advantages are most significant for SimPO, where RGPO propels the Qwen2\.5\-7B\-Instruct model’s AlapaEval 2 Length\-Controlled \(LC\) Win Rate from73\.09%73\.09\\%to80\.05%80\.05\\%\. This substantial gain underscores the stability of RGPO, suggesting that consistency\-weighted optimization encourages genuine alignment patterns rather than merely exploiting generation length\.

Scalability across different base models\.Our experiments indicate that RGPO generalizes effectively to more capable base models\. While the Qwen2\.5\-7B\-Instruct model exhibits a naturally stronger baseline performance compared to Llama\-3\-8B\-Instruct, the integration of RGPO consistently amplifies this advantage\. The highest overall performance across the entire evaluation is achieved by the Qwen2\.5\-7B\-Instruct model trained on MultiPref with RGPO, with a remarkable85\.36%85\.36\\%Raw Win Rate and80\.05%80\.05\\%LC Win Rate on AlpacaEval 2, and a competitive85\.20%85\.20\\%Win Rate on Arena\-Hard\. This demonstrates that RGPO can effectively leverage the superior reasoning capabilities of advanced backbones, and refine their outputs even further by providing reliability\-weighted preference signals\.

Robustness against optimization instability\.The results highlight the critical role of RGPO in stabilizing training, particularly for optimizations sensitive to data quality\. As observed in Table[1](https://arxiv.org/html/2607.20515#S4.T1), the SimPO framework shows significant instability with extreme low values on the challenging HelpSteer2 dataset, such as a12\.31%12\.31\\%Raw Win Rate for Llama\-3, which significantly underperforms the model baseline without alignment of31\.61%31\.61\\%\. This is likely because of SimPO’s sensitivity to ambiguous labels in difficult datasets\. However, when integrated with RGPO, the SimPO model recovers dramatically, with a25\.60%25\.60\\%Llama\-3 win rate on HelpSteer2\. Importantly, this stabilization extends beyond only recovery in terms of Length\-Controlled Win Rate, the RGPO\-enhanced SimPO model \(40\.32%40\.32\\%\) surpasses the Llama\-3\-Instruct model baseline \(30\.28%30\.28\\%\)\. This confirms that RGPO serves as a robust stabilizer, transforming potentially fragile optimization objectives into highly competitive alignments by mitigating the noise that causes divergence\.

Table 1:Evaluation results of standard RLHF baselines versus RGPO\-enhanced models on AlpacaEval 2 and Arena\-Hard benchmarks\. Statistically significant improvements over the corresponding baselines are marked with \* \(p<0\.05p<0\.05\)\.Llama\-3\-8B\-InstructQwen2\.5\-7B\-InstructAlpacaEval 2Arena\-HardAlpacaEval 2Arena\-HardMethodsRaw Win Rate\(%\)LC Win Rate\(%\)Win Rate\(%\)Raw Win Rate\(%\)LC Win Rate\(%\)Win Rate\(%\)MultiPrefBase Model31\.6130\.2836\.9043\.2849\.1274\.40DPO39\.5135\.5543\.7049\.7350\.8278\.20w/ RGPO40\.7438\.30\*46\.60\*51\.1751\.9577\.10SimPO52\.6460\.4536\.1069\.0773\.0982\.80w/ RGPO53\.3463\.29\*37\.6085\.36\*80\.05\*85\.20\*IPO43\.0542\.0451\.9057\.8854\.5681\.30w/ RGPO43\.8442\.9453\.1059\.0555\.82\*80\.90Helpsteer2\-DisagreementDPO28\.9528\.7835\.0040\.3144\.3474\.80w/ RGPO29\.4031\.82\*36\.7039\.8346\.42\*76\.90SimPO12\.3128\.626\.1023\.2931\.0156\.70w/ RGPO25\.60\*40\.32\*26\.10\*31\.40\*38\.46\*64\.90\*IPO31\.3038\.7240\.0040\.6047\.5676\.40w/ RGPO32\.6338\.7841\.8040\.0448\.1978\.50
### 4\.3Analysis of Annotator Reliability

We analyze the distribution of the estimated reliability scoresqjq\_\{j\}obtained via interative reliability estimation\. Figure[3](https://arxiv.org/html/2607.20515#S4.F3)illustrates the reliability variance across both datasets\. For the MultiPref dataset, as shown in Figure[3\(a\)](https://arxiv.org/html/2607.20515#S4.F3.sf1), the reliability scores follow an approximately normal distribution centered at a mean of0\.6460\.646\. Most annotators exhibit consistent performance, with scores clustered between0\.50\.5and0\.80\.8, indicating that they provide signals that are better than random\. Notably, the distribution features a long tail of annotators with extremely low reliability \(qj<0\.4q\_\{j\}<0\.4\)\. Our investigation reveals that these outliers are primarily due to data sparsity: these annotators labeled a very limited number of samples, preventing the maximum likelihood estimation from converging to a stable estimate of their true competence\. Conversely, the HelpSteer2 dataset, as depicted in Figure[3\(b\)](https://arxiv.org/html/2607.20515#S4.F3.sf2), demonstrates significant variability in annotation quality in a small group of annotators\. The scores range from a low of0\.4790\.479to a high of0\.6990\.699, with a mean of0\.6170\.617\. This substantial gap suggests that even within small and curated groups, treating all annotators uniformly is suboptimal\. The observed reliability variance in both large\-scale and small\-team settings validates our motivation that simple aggregation or uniform smoothing would erroneously upweight unreliable judgments or dilute high\-quality ones, whereas our optimization with reliability\-based consistency dynamically adjusts for these discrepancies\.

\(a\)MultiPref: 227 Annotators\(b\)Helpsteer2: 6 Annotators
Figure 3:Distributions of annotator reliabilityqjq\_\{j\}\.
### 4\.4Ablation Analysis of RGPO

To assess the contribution of RGPO’s each core component, we conduct ablation analyses of*Iterative Latent Reliability Estimation*\(ILRE\) and*Reliability\-Aware Consistency Optimization*\(RACO\)\.

The ablation results show that the complete RGPO achieves the strongest performance, while removing either ILRE or RACO weakens the overall gains\. As shown in Table[2](https://arxiv.org/html/2607.20515#S4.T2), removing ILRE leads to a substantial performance drop relative to full RGPO, especially on AlpacaEval 2 and Arena\-Hard, indicating that reliability\-aware preference estimation is crucial for constructing effective training signals\. Training without RACO still preserves part of the improvement brought by ILRE, but it consistently falls behind full RGPO across all metrics\. This drop demonstrates the contribution of RACO, showing that reliability\-aware consistency optimization further improves alignment beyond preference\-pair construction alone\. Overall, these results suggest that ILRE provides the main benefit through cleaner preference estimation, while RACO further strengthens optimization through reliability\-aware consistency measurement that guides the training objective\.

Table 2:Ablation results of Iterative Latent Reliability Estimation \(ILRE\) and Reliability\-Aware Consistency Optimization \(RACO\)\.AlpacaEval 2Arena\-HardMethodsWR \(%\)LC WR \(%\)WR \(%\)DPO39\.5135\.5543\.70w/ RGPO40\.7438\.3046\.60w/o ILRE38\.9735\.7742\.50w/o RACO40\.1337\.4845\.40#### Preference label distribution\.

We further analyze the preference pairs constructed by ILRE to validate the effectiveness of RGPO’s reliability\-aware estimation\. We compare the distribution of preference labels inferred by RGPO with that produced by the baseline uniform ensemble in Table[3](https://arxiv.org/html/2607.20515#S4.T3)\. We observe a substantial shift in the label distribution, specifically a marked increase in the proportion ofTiecases across both datasets, rising from12\.27%12\.27\\%to29\.03%29\.03\\%on MultiPref and from11\.71%11\.71\\%to31\.90%31\.90\\%on HelpSteer2\. This trend indicates that the uniform ensemble often forces a binary decision on ambiguous samples where annotators disagree, thereby introducing noise into the training signal\. In contrast, through annotator reliability\-aware estimation, RGPO effectively identifies such ambiguity and filters out inconsistent, noisy samples that would otherwise confuse the model\. Together with the ablation results in Table[2](https://arxiv.org/html/2607.20515#S4.T2), these findings suggest that learning from a cleaner and more reliable set of preference pairs enables RGPO to achieve stronger alignment performance\.

Table 3:Comparison of preference label distribution between Uniform Ensemble and RGPO’s reliability\-aware estimation\.DatasetMethodsrAr\_\{A\}TierBr\_\{B\}MultiPref\(n=10,461\)Uniform Ensemble6,225\(59\.51%\)1,284\(12\.27%\)2,952\(28\.22%\)RGPO5,278\(50\.45%\)3,037\(29\.03%\)2,146\(20\.51%\)HelpSteer2\(n=11,824\)Uniform Ensemble4,904\(41\.47%\)1,384\(11\.71%\)5,536\(46\.82%\)RGPO4,391\(37\.14%\)3,772\(31\.90%\)3,661\(30\.96%\)
#### Comparison to confidence\-based filtering\.

To investigate the contribution of RGPO’s reliability\-based preference estimation beyond direct confidence\-based filtering, we construct a confidence\-filtered variant of DPO using the annotator\-provided confidence labels in MultiPref\. Specifically, we remove the least confident samples from the uniform\-ensemble training set and use the filtered data for DPO training\. We then compare this confidence\-filtered variant with RGPO\-enhanced DPO\. As shown in Table[4](https://arxiv.org/html/2607.20515#S4.T4), confidence\-based filtering provides modest improvements over standard DPO in LC win rate and Arena\-Hard performance, but slightly reduces the raw win rate on AlpacaEval 2\. In contrast, RGPO achieves the strongest performance across all metrics, suggesting that its gains are not simply due to removing low\-confidence samples\. Instead, the EM\-based estimation step provides a more principled inference of latent preferences and annotator reliability from inconsistent annotations\. See Appendix[A](https://arxiv.org/html/2607.20515#A1)for the confidence distribution of MultiPref\.

Table 4:Effect of confidence\-based filtering and RGPO\-based preference pair construction on DPO training\.AlpacaEval 2Arena\-HardMethodsWR \(%\)LC WR \(%\)WR \(%\)DPO39\.5135\.5543\.70w/ Confidence Filtering38\.6736\.0845\.20w/ RGPO40\.7438\.3046\.60

### 4\.5Analysis of Consistency Weight Scaling

To evaluate the scaling strategy of consistency weight in RGPO, we compare the unscaled weights with four scaling methods\. As illustrated in Figure[4\(a\)](https://arxiv.org/html/2607.20515#S4.F4.sf1), the model utilizing the scaling strategy achieves consistently higher training accuracy compared to the model trained without scaling\. This suggests that raw consistency scores alone may not provide sufficient signal for robust optimization\. Further insights are provided in Figure[4\(b\)](https://arxiv.org/html/2607.20515#S4.F4.sf2)and Appendix[C\.2](https://arxiv.org/html/2607.20515#A3.SS2), which track the evolution of consistency weights and accuracies throughout the training\. Among the evaluated methods, Tanh scaling shows the highest stability\. In contrast, methods such as Sigmoid, Min\-Max \(Power\), and the unscaled baseline exhibit significantly lower overall weight magnitudes, which result in diminished optimization strength and unstable training signal\. In Table[5](https://arxiv.org/html/2607.20515#S4.T5), while the Min\-Max \(Power\) scaling achieves a slightly higher Raw Win Rate \(41\.00%41\.00\\%\), the Tanh scaling outperforms all others in the Length\-Controlled Win Rate \(38\.30%38\.30\\%\)\. This indicates that Tanh scaling effectively balances optimization stability with alignment quality, resulting in more helpful responses\. We further discuss the sensitivity analysis of the Tanh scaling coefficientλ\\lambdain Appendix[C\.1](https://arxiv.org/html/2607.20515#A3.SS1)\.

\(a\)Training Accuracy\(b\)Consistency Weight
Figure 4:Effectiveness of consistency weight scaling\.Table 5:Evaluation results of DPO\+RGPO models trained with different consistency weight scaling strategies\.AlpacaEval 2Methodsf​\(x\)f\(x\)WR \(%\)LC WR \(%\)w/o Scalingxx39\.0635\.54Tanh12​\(tanh⁡\(x\)\+1\)\\frac\{1\}\{2\}\\left\(\\tanh\\left\(x\\right\)\+1\\right\)40\.7438\.30Sigmoid11\+e−x−μσ\\frac\{1\}\{1\+e^\{\-\\frac\{x\-\\mu\}\{\\sigma\}\}\}39\.7937\.15Min\-Max \(Power\)xn−min⁡\(xn\)max⁡\(xn\)−min⁡\(xn\)\\frac\{x^\{n\}\-\\min\(x^\{n\}\)\}\{\\max\(x^\{n\}\)\-\\min\(x^\{n\}\)\}41\.0037\.54Mean Centeringx−μ\+1x\-\\mu\+139\.1036\.36
### 4\.6Analysis of Annotator ID Proxies for HelpSteer2

As annotator IDs are unavailable in HelpSteer2, we use annotation indices as proxy annotator IDs when estimating reliability\. These proxies are not intended to recover the true annotator identities\. Instead, they serve as a practical surrogate for modeling heterogeneous annotation behavior when explicit annotator metadata is unavailable\. If the proxies reflect systematic differences in annotation quality or scoring patterns, a reliability\-aware estimation model can still capture meaningful variation in annotation reliability across proxies\(Raykar et al\.,[2010](https://arxiv.org/html/2607.20515#bib.bib32); Whitehill et al\.,[2009](https://arxiv.org/html/2607.20515#bib.bib43)\)\. We observe three pieces of evidence for such systematic differences in HelpSteer2: \(i\) the inferred reliability scores in Figure[3\(b\)](https://arxiv.org/html/2607.20515#S4.F3.sf2)vary substantially across proxies, suggesting non\-uniform annotation behavior; \(ii\) the behavior\-profile analysis in Figure[5](https://arxiv.org/html/2607.20515#S4.F5)shows structured differences across proxies in score distributions and candidate\-pair score margins; \(iii\) RGPO achieves consistent performance gains on HelpSteer2, as shown in Table[1](https://arxiv.org/html/2607.20515#S4.T1), supporting the view that these proxy IDs capture useful signals of annotation inconsistency\. This setting is practically important because many open\-source preference datasets do not release annotator identity metadata\. The gains observed on both MultiPref, where annotator IDs are available, and HelpSteer2, where only proxy IDs are used, suggest that RGPO can be applied under varying levels of annotation metadata availability\.

Figure 5:Annotation behavior profiles across proxy annotator IDs on HelpSteer2\. The z\-score normalized profiles reveal clear structured differences across proxies: proxies 0, 1, and 2 assign higher mean scores with lower score variance, proxies 3 and 4 produce larger candidate\-pair score margins, and proxy 5 has the smallest pairwise margins\.

## 5Discussion

#### Conclusion\.

We propose the*Reliability\-Guided Preference Optimization*\(RGPO\) framework to address the critical challenge of annotation inconsistency and subjectivity in RLHF, which stems from the reliance on multi\-annotated human feedback\. Existing methods often disregard the valuable signal contained in annotator disagreement by treating all preference pairs as equally valid\. To overcome this, our approach leverages*Iterative Latent Reliability Estimation*, which not only estimates robust ground\-truth labels but also infers specific annotator reliability scores\. These estimated reliability scores directly drive our*Reliability\-Aware Consistency Optimization*, which regulates the gradient magnitude dynamically\. Our findings confirm that explicitly modeling annotator consistency into the training loop significantly enhance model robustness and alignment performance\.

#### Limitations\.

Our study has two main limitations\. First, due to limited computational resources, we conducted experiments with models up to 8B parameters using Low\-Rank Adaptation \(LoRA\)\. However, larger models may have better robustness to noise but does not eliminate systematic conflicts from inconsistent annotator preferences\. The efficacy of RGPO on larger\-scale or full\-parameter models remains to be verified\. Second, there are very few public datasets available that provide multiple annotations for each sample\. As a result, we assessed our framework on the MultiPref and HelpSteer2\-Disagreement datasets only\. A key difference between these sources is that MultiPref provides unique annotator IDs, while HelpSteer2 lacks such metadata\. For HelpSteer2, we used annotation indices as a substitute to group annotations\. Although our analysis suggests that annotation\-index proxies can capture useful systematic differences in HelpSteer2, this approximation may still introduce additional noise into reliability estimation, potentially making the resulting gains less precise than those obtained with true annotator IDs in MultiPref\.

## Impact Statement

As LLMs are integrated into diverse aspects of daily life, it is critical to ensure their alignment with human intent\. Our work addresses the challenge of alignment on inconsistent and noisy human feedback, which is particularly prevalent in the subjective tasks common to social usage\. We hope this work stimulates broader discussion and further optimization regarding annotation quality in alignment pipelines, and fosters the development of AI systems that are not only more robust but also safer and more consistent in their interactions with users\.

## References

- Askell et al\. \(2021\)Askell, A\., Bai, Y\., Chen, A\., Drain, D\., Ganguli, D\., Henighan, T\., Jones, A\., Joseph, N\., Mann, B\., DasSarma, N\., et al\.A general language assistant as a laboratory for alignment\.*arXiv preprint arXiv:2112\.00861*, 2021\.
- Azar et al\. \(2024\)Azar, M\. G\., Guo, Z\. D\., Piot, B\., Munos, R\., Rowland, M\., Valko, M\., and Calandriello, D\.A general theoretical paradigm to understand learning from human preferences\.In*International Conference on Artificial Intelligence and Statistics*, pp\. 4447–4455\. PMLR, 2024\.
- Bai et al\. \(2022\)Bai, Y\., Jones, A\., Ndousse, K\., Askell, A\., Chen, A\., DasSarma, N\., Drain, D\., Fort, S\., Ganguli, D\., Henighan, T\., et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.*CoRR*, 2022\.
- Bakker et al\. \(2022\)Bakker, M\., Chadwick, M\., Sheahan, H\., Tessler, M\., Campbell\-Gillingham, L\., Balaguer, J\., McAleese, N\., Glaese, A\., Aslanides, J\., Botvinick, M\., et al\.Fine\-tuning language models to find agreement among humans with diverse preferences\.*Advances in neural information processing systems*, 35:38176–38189, 2022\.
- Bishop & Nasrabadi \(2006\)Bishop, C\. M\. and Nasrabadi, N\. M\.*Pattern recognition and machine learning*, volume 4\.Springer, 2006\.
- Casper et al\. \(2023\)Casper, S\., Davies, X\., Shi, C\., Gilbert, T\. K\., Scheurer, J\., Rando, J\., Freedman, R\., Korbak, T\., Lindner, D\., Freire, P\., Wang, T\. T\., Marks, S\., Segerie, C\.\-R\., Carroll, M\., Peng, A\., Christoffersen, P\. J\., Damani, M\., Slocum, S\., Anwar, U\., Siththaranjan, A\., Nadeau, M\., Michaud, E\. J\., Pfau, J\., Krasheninnikov, D\., Chen, X\., Langosco, L\., Hase, P\., Biyik, E\., Dragan, A\., Krueger, D\., Sadigh, D\., and Hadfield\-Menell, D\.Open problems and fundamental limitations of reinforcement learning from human feedback\.*Transactions on Machine Learning Research*, 2023\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=bx24KpJ4Eb](https://openreview.net/forum?id=bx24KpJ4Eb)\.Survey Certification, Featured Certification\.
- Chowdhury et al\. \(2024\)Chowdhury, S\. R\., Kini, A\., and Natarajan, N\.Provably robust DPO: Aligning language models with noisy feedback\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=yhpDKSw7yA](https://openreview.net/forum?id=yhpDKSw7yA)\.
- Christiano et al\. \(2017\)Christiano, P\. F\., Leike, J\., Brown, T\., Martic, M\., Legg, S\., and Amodei, D\.Deep reinforcement learning from human preferences\.*Advances in neural information processing systems*, 30, 2017\.
- Dawid & Skene \(1979\)Dawid, A\. P\. and Skene, A\. M\.Maximum likelihood estimation of observer error\-rates using the em algorithm\.*Journal of the Royal Statistical Society: Series C \(Applied Statistics\)*, 28\(1\):20–28, 1979\.
- Dong et al\. \(2023\)Dong, H\., Xiong, W\., Goyal, D\., Zhang, Y\., Chow, W\., Pan, R\., Diao, S\., Zhang, J\., SHUM, K\., and Zhang, T\.RAFT: Reward ranked finetuning for generative foundation model alignment\.*Transactions on Machine Learning Research*, 2023\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=m7p5O7zblY](https://openreview.net/forum?id=m7p5O7zblY)\.
- Dubois et al\. \(2023\)Dubois, Y\., Li, X\., Taori, R\., Zhang, T\., Gulrajani, I\., Ba, J\., Guestrin, C\., Liang, P\., and Hashimoto, T\.Alpacafarm: A simulation framework for methods that learn from human feedback\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=4hturzLcKX](https://openreview.net/forum?id=4hturzLcKX)\.
- Dubois et al\. \(2024\)Dubois, Y\., Liang, P\., and Hashimoto, T\.Length\-controlled alpacaeval: A simple debiasing of automatic evaluators\.In*First Conference on Language Modeling*, 2024\.
- Ethayarajh et al\. \(2024\)Ethayarajh, K\., Xu, W\., Muennighoff, N\., Jurafsky, D\., and Kiela, D\.Model alignment as prospect theoretic optimization\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=iUwHnoENnl](https://openreview.net/forum?id=iUwHnoENnl)\.
- Fernandes et al\. \(2023\)Fernandes, P\., Madaan, A\., Liu, E\., Farinhas, A\., Martins, P\. H\., Bertsch, A\., de Souza, J\. G\. C\., Zhou, S\., Wu, T\., Neubig, G\., and Martins, A\. F\. T\.Bridging the gap: A survey on integrating \(human\) feedback for natural language generation\.*Transactions of the Association for Computational Linguistics*, 11:1643–1668, 2023\.doi:10\.1162/tacl˙a˙00626\.URL[https://aclanthology\.org/2023\.tacl\-1\.92/](https://aclanthology.org/2023.tacl-1.92/)\.
- Ganguli et al\. \(2022\)Ganguli, D\., Lovitt, L\., Kernion, J\., Askell, A\., Bai, Y\., Kadavath, S\., Mann, B\., Perez, E\., Schiefer, N\., Ndousse, K\., et al\.Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned\.*CoRR*, 2022\.
- Gao et al\. \(2023\)Gao, L\., Schulman, J\., and Hilton, J\.Scaling laws for reward model overoptimization\.In*Proceedings of the 40th International Conference on Machine Learning*, pp\. 10835–10866, 2023\.
- Glaese et al\. \(2022\)Glaese, A\., McAleese, N\., Trkebacz, M\., Aslanides, J\., Firoiu, V\., Ewalds, T\., Rauh, M\., Weidinger, L\., Chadwick, M\., Thacker, P\., Campbell\-Gillingham, L\., Uesato, J\., Huang, P\.\-S\., Comanescu, R\., Yang, F\., See, A\., Dathathri, S\., Greig, R\., Chen, C\., Fritz, D\., Elias, J\. S\., Green, R\., Mokr’a, S\., Fernando, N\., Wu, B\., Foley, R\., Young, S\., Gabriel, I\., Isaac, W\. S\., Mellor, J\. F\. J\., Hassabis, D\., Kavukcuoglu, K\., Hendricks, L\. A\., and Irving, G\.Improving alignment of dialogue agents via targeted human judgements\.*ArXiv*, abs/2209\.14375, 2022\.URL[https://api\.semanticscholar\.org/CorpusID:252596089](https://api.semanticscholar.org/CorpusID:252596089)\.
- Gordon et al\. \(2022\)Gordon, M\. L\., Lam, M\. S\., Park, J\. S\., Patel, K\., Hancock, J\., Hashimoto, T\., and Bernstein, M\. S\.Jury learning: Integrating dissenting voices into machine learning models\.In*Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*, pp\. 1–19, 2022\.
- Hong et al\. \(2024\)Hong, J\., Lee, N\., and Thorne, J\.Orpo: Monolithic preference optimization without reference model\.In*2024 Conference on Empirical Methods in Natural Language Processing*\. Association for Computational Linguistics, 2024\.
- Kwon et al\. \(2023\)Kwon, W\., Li, Z\., Zhuang, S\., Sheng, Y\., Zheng, L\., Yu, C\. H\., Gonzalez, J\., Zhang, H\., and Stoica, I\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the 29th symposium on operating systems principles*, pp\. 611–626, 2023\.
- Larson et al\. \(2019\)Larson, S\., Mahendran, A\., Lee, A\., Kummerfeld, J\. K\., Hill, P\., Laurenzano, M\. A\., Hauswald, J\., Tang, L\., and Mars, J\.Outlier detection for improved data quality and diversity in dialog systems\.In Burstein, J\., Doran, C\., and Solorio, T\. \(eds\.\),*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pp\. 517–527, Minneapolis, Minnesota, June 2019\. Association for Computational Linguistics\.doi:10\.18653/v1/N19\-1051\.URL[https://aclanthology\.org/N19\-1051/](https://aclanthology.org/N19-1051/)\.
- Li et al\. \(2025\)Li, T\., Chiang, W\.\-L\., Frick, E\., Dunlap, L\., Wu, T\., Zhu, B\., Gonzalez, J\. E\., and Stoica, I\.From crowdsourced data to high\-quality benchmarks: Arena\-hard and benchbuilder pipeline\.In*Forty\-second International Conference on Machine Learning*, 2025\.URL[https://openreview\.net/forum?id=KfTf9vFvSn](https://openreview.net/forum?id=KfTf9vFvSn)\.
- Liang et al\. \(2025\)Liang, X\., Chen, C\., Qiu, S\., Wang, J\., Wu, Y\., Fu, Z\., Chen, H\., Wu, F\., and Ye, J\.ROPO: Robust preference optimization for large language models\.In*Forty\-second International Conference on Machine Learning*, 2025\.URL[https://openreview\.net/forum?id=5WEmyTooVV](https://openreview.net/forum?id=5WEmyTooVV)\.
- Liu et al\. \(2024\)Liu, T\., Zhao, Y\., Joshi, R\., Khalman, M\., Saleh, M\., Liu, P\. J\., and Liu, J\.Statistical rejection sampling improves preference optimization\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=xbjSwwrQOe](https://openreview.net/forum?id=xbjSwwrQOe)\.
- Meng et al\. \(2024\)Meng, Y\., Xia, M\., and Chen, D\.Simpo: simple preference optimization with a reference\-free reward\.In*Proceedings of the 38th International Conference on Neural Information Processing Systems*, pp\. 124198–124235, 2024\.
- Menick et al\. \(2022\)Menick, J\., Trebacz, M\., Mikulik, V\., Aslanides, J\., Song, F\., Chadwick, M\., Glaese, M\., Young, S\., Campbell\-Gillingham, L\., Irving, G\., et al\.Teaching language models to support answers with verified quotes\.*arXiv preprint arXiv:2203\.11147*, 2022\.
- Miranda et al\. \(2025\)Miranda, L\. J\. V\., Wang, Y\., Elazar, Y\., Kumar, S\., Pyatkin, V\., Brahman, F\., Smith, N\. A\., Hajishirzi, H\., and Dasigi, P\.Hybrid preferences: Learning to route instances for human vs\. ai feedback\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 7162–7200, 2025\.
- Ouyang et al\. \(2022\)Ouyang, L\., Wu, J\., Jiang, X\., Almeida, D\., Wainwright, C\., Mishkin, P\., Zhang, C\., Agarwal, S\., Slama, K\., Ray, A\., et al\.Training language models to follow instructions with human feedback\.*Advances in neural information processing systems*, 35:27730–27744, 2022\.
- Peng et al\. \(2023\)Peng, B\., Li, C\., He, P\., Galley, M\., and Gao, J\.Instruction tuning with gpt\-4\.*arXiv preprint arXiv:2304\.03277*, 2023\.
- Plank \(2022\)Plank, B\.The “problem” of human label variation: On ground truth in data, modeling and evaluation\.In Goldberg, Y\., Kozareva, Z\., and Zhang, Y\. \(eds\.\),*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp\. 10671–10682, Abu Dhabi, United Arab Emirates, December 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.emnlp\-main\.731\.URL[https://aclanthology\.org/2022\.emnlp\-main\.731/](https://aclanthology.org/2022.emnlp-main.731/)\.
- Rafailov et al\. \(2023\)Rafailov, R\., Sharma, A\., Mitchell, E\., Manning, C\. D\., Ermon, S\., and Finn, C\.Direct preference optimization: Your language model is secretly a reward model\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=HPuSIXJaa9](https://openreview.net/forum?id=HPuSIXJaa9)\.
- Raykar et al\. \(2010\)Raykar, V\. C\., Yu, S\., Zhao, L\. H\., Valadez, G\. H\., Florin, C\., Bogoni, L\., and Moy, L\.Learning from crowds\.*Journal of machine learning research*, 11\(4\), 2010\.
- Santurkar et al\. \(2023\)Santurkar, S\., Durmus, E\., Ladhak, F\., Lee, C\., Liang, P\., and Hashimoto, T\.Whose opinions do language models reflect?In*International Conference on Machine Learning*, pp\. 29971–30004\. PMLR, 2023\.
- Sap et al\. \(2022\)Sap, M\., Swayamdipta, S\., Vianna, L\., Zhou, X\., Choi, Y\., and Smith, N\. A\.Annotators with attitudes: How annotator beliefs and identities bias toxic language detection\.In*Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies*, pp\. 5884–5906, 2022\.
- Schulman et al\. \(2017\)Schulman, J\., Wolski, F\., Dhariwal, P\., Radford, A\., and Klimov, O\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Sharma et al\. \(2024\)Sharma, M\., Tong, M\., Korbak, T\., Duvenaud, D\., Askell, A\., Bowman, S\. R\., DURMUS, E\., Hatfield\-Dodds, Z\., Johnston, S\. R\., Kravec, S\. M\., Maxwell, T\., McCandlish, S\., Ndousse, K\., Rausch, O\., Schiefer, N\., Yan, D\., Zhang, M\., and Perez, E\.Towards understanding sycophancy in language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=tvhaxkMKAn](https://openreview.net/forum?id=tvhaxkMKAn)\.
- Song et al\. \(2024\)Song, F\., Yu, B\., Li, M\., Yu, H\., Huang, F\., Li, Y\., and Wang, H\.Preference ranking optimization for human alignment\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp\. 18990–18998, 2024\.
- Stiennon et al\. \(2020\)Stiennon, N\., Ouyang, L\., Wu, J\., Ziegler, D\., Lowe, R\., Voss, C\., Radford, A\., Amodei, D\., and Christiano, P\. F\.Learning to summarize with human feedback\.*Advances in neural information processing systems*, 33:3008–3021, 2020\.
- Swamy et al\. \(2024\)Swamy, G\., Dann, C\., Kidambi, R\., Wu, Z\. S\., and Agarwal, A\.A minimaximalist approach to reinforcement learning from human feedback\.In*Proceedings of the 41st International Conference on Machine Learning*, pp\. 47345–47377, 2024\.
- Uma et al\. \(2021\)Uma, A\. N\., Fornaciari, T\., Hovy, D\., Paun, S\., Plank, B\., and Poesio, M\.Learning from disagreement: A survey\.*Journal of Artificial Intelligence Research*, 72:1385–1470, 2021\.
- Wang et al\. \(2024\)Wang, Z\., Dong, Y\., Delalleau, O\., Zeng, J\., Shen, G\., Egert, D\., Zhang, J\., Sreedhar, M\. N\., and Kuchaiev, O\.Helpsteer 2: Open\-source dataset for training top\-performing reward models\.*Advances in Neural Information Processing Systems*, 37:1474–1501, 2024\.
- Wang et al\. \(2025\)Wang, Z\., Bukharin, A\., Delalleau, O\., Egert, D\., Shen, G\., Zeng, J\., Kuchaiev, O\., and Dong, Y\.Helpsteer2\-preference: Complementing ratings with preferences\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=MnfHxPP5gs](https://openreview.net/forum?id=MnfHxPP5gs)\.
- Whitehill et al\. \(2009\)Whitehill, J\., Wu, T\.\-f\., Bergsma, J\., Movellan, J\., and Ruvolo, P\.Whose vote should count more: Optimal integration of labels from labelers of unknown expertise\.*Advances in neural information processing systems*, 22, 2009\.
- Wu et al\. \(2023\)Wu, Z\., Hu, Y\., Shi, W\., Dziri, N\., Suhr, A\., Ammanabrolu, P\., Smith, N\. A\., Ostendorf, M\., and Hajishirzi, H\.Fine\-grained human feedback gives better rewards for language model training\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=CSbGXyCswu](https://openreview.net/forum?id=CSbGXyCswu)\.
- Xu et al\. \(2024\)Xu, H\., Sharaf, A\., Chen, Y\., Tan, W\., Shen, L\., Durme, B\. V\., Murray, K\., and Kim, Y\. J\.Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=51iwkioZpn](https://openreview.net/forum?id=51iwkioZpn)\.
- Yuan et al\. \(2023\)Yuan, H\., Yuan, Z\., Tan, C\., Wang, W\., Huang, S\., and Huang, F\.Rrhf: Rank responses to align language models with human feedback\.*Advances in Neural Information Processing Systems*, 36:10935–10950, 2023\.
- Zhao et al\. \(2023\)Zhao, Y\., Joshi, R\., Liu, T\., Khalman, M\., Saleh, M\., and Liu, P\. J\.Slic\-hf: Sequence likelihood calibration with human feedback\.*CoRR*, 2023\.
- Zhu et al\. \(2023\)Zhu, B\., Jordan, M\., and Jiao, J\.Principled reinforcement learning with human feedback from pairwise or k\-wise comparisons\.In*International Conference on Machine Learning*, pp\. 43037–43067\. PMLR, 2023\.

## Appendix AImplementation

#### Training hyperparameters\.

Table[6](https://arxiv.org/html/2607.20515#A1.T6)summarizes all training hyperparameters used in our experiments\. We filter out all samples marked as ties for baselines and RGPO\-enhanced models to strictly establish the preference direction\. To rigorously evaluate the effectiveness of the RGPO advancement, we employ a constant learning rate scheduler throughout training, ensuring that performance gains are attributable to the method rather than scheduler decay\. We adapt the KL penalty coefficientβ\\betaacross a range of\{0\.05,0\.1,1,2\.5,10\}\\\{0\.05,0\.1,1,2\.5,10\\\}to align with the standard implementations of different baseline algorithms and datasets\. Specifically, for SimPO and SimPO\+RGPO training across both datasets, hyperparameters vary by base model: for Llama\-3\-8B\-Instruct, we useβ=2\.5\\beta=2\.5withγ/β=0\.55\\gamma/\\beta=0\.55; for Qwen2\.5\-7B\-Instruct, we setβ=10\\beta=10withγ/β=0\.3\\gamma/\\beta=0\.3\. For the consistency weight scaling coefficient, we useλ=1\\lambda=1as the default setting and normalize the output to the\[0,1\]\[0,1\]range\. This moderate scaling preserves the relative ordering of consistency weights while preventing extreme values from dominating the gradient updates\. We apply LoRA adapters to all linear projection layers to provide sufficient expressiveness while maintaining parameter efficiency\. Training is distributed across 3 GPUs using DeepSpeed ZeRO Stage\-2 optimization with gradient checkpointing enabled to reduce required memory\. We used AI assistance for language polishing during the writing process\.

Table 6:Training hyperparameter settings\.HyperparameterValueModel ConfigurationBase ModelMeta\-Llama\-3\-8B\-Instruct / Qwen2\.5\-7B\-InstructPrecisionBF16LoRA ConfigurationLoRA Rank \(rr\)64LoRA Alpha \(α\\alpha\)128LoRA Dropout0\.05Target ModulesAll LinearTraining ConfigurationLearning Rate5×10−65\\times 10^\{\-6\}LR SchedulerConstantWeight Decay0\.01Epochs2 / 3Effective Batch Size96Algorithm\-Specific Parametersβ\\beta\{0\.05, 0\.1, 1, 2\.5, 10\}SimPOγ/β\\gamma/\\beta0\.55 / 0\.3Max Sequence Length2048Max Prompt Length1024RGPO Scaling MethodTanhRGPO Scaling Coefficientλ\\lambda1\.0InfrastructureDistributed TrainingDeepSpeed ZeRO\-2Random Seed42
#### Inference hyperparamters\.

We evaluate our trained models using two established benchmarks: AlpacaEval 2 and Arena\-Hard\. Table[7](https://arxiv.org/html/2607.20515#A1.T7)summarizes the inference and evaluation configurations used in our experiments\. For AlpacaEval 2, which consists of 805 open\-ended user instructions designed to assess real\-world instruction\-following capabilities, we use a temperature of0\.70\.7to encourage diverse responses while maintaining coherence\. The evaluation compares model outputs against thegpt\-4\-1106\-previewreference model using a weighted scoring scheme that accounts for length bias\. For Arena\-Hard, we employ the default greedy decoding \(temperature0\.00\.0\) to ensure deterministic and reproducible outputs across its 500 challenging user queries, withgpt\-4\-0314as the reference model\. Notably, we extend the generation limit to 4096 new tokens for Arena\-Hard to fully accommodate the complex reasoning chains required by these difficult prompts, preventing truncating detailed responses\. We utilize vLLM as the inference backend for Arena\-Hard to enable efficient batched generation with tensor parallelism across multiple GPUs\(Kwon et al\.,[2023](https://arxiv.org/html/2607.20515#bib.bib20)\)\.

Table 7:Inference and evaluation hyperparameter settings\.HyperparameterAlpacaEval 2Arena\-HardInference ConfigurationMax New Tokens20484096Temperature0\.70\.0Evaluation ConfigurationNumber of Samples805500Reference ModelGPT\-4\-1106\-previewGPT\-4\-0314
#### MultiPref confidence label distribution\.

Table[8](https://arxiv.org/html/2607.20515#A1.T8)reports the distribution of annotator\-provided confidence labels in MultiPref\. The dataset contains four confidence levels: absolutely\-confident, fairly\-confident, not\-confident, and random\-guess\. For the confidence\-filtering baseline, we directly use these labels and remove annotations marked as “not\-confident” or “random\-guess”, which filters out 1,712 annotations out of 41,844 in total\. In contrast, RGPO does not rely on explicit confidence labels for filtering\. Instead, it retains the annotations and models their quality through reliability\-aware preference estimation and sample\-level consistency weighting\. This design addresses a limitation of confidence filtering: self\-reported confidence can be noisy or incomplete\. By modeling consistency with respect to both annotator agreement and estimated reliability, RGPO can identify ambiguous or unreliable comparisons beyond what confidence labels capture, while adaptively downweighting low\-consistency samples during optimization\.

Table 8:Distribution of annotator\-provided confidence labels in MultiPref\.Confidence LabelCount%absolutely\-confident28,84968\.94%fairly\-confident11,28326\.96%not\-confident1,4193\.39%random\-guess2930\.70%

## Appendix BTraining Efficiency

Beyond improvements in alignment performance, our experimental results also demonstrate that RGPO significantly enhances training efficiency\. As shown in Table[9](https://arxiv.org/html/2607.20515#A2.T9), training enhanced with the RGPO framework consistently reduces the wall\-clock training time per epoch across different base models and optimization methods\. By mitigating the noise and inconsistency inherent in the training datasets, and utilizing consistency\-weighted adaptive optimization, the alignment process is streamlined to focus more effectively on high\-quality supervision signals\.

Table 9:Wall\-clock training time per epoch \(in seconds\) on MultiPref\.MethodsLlama\-3\-8B\-InstructQwen2\.5\-7B\-InstructDPO4125\.26463981\.4421w/ RGPO3354\.6344 \(↓\\downarrow770\.6302\)3240\.2112 \(↓\\downarrow741\.2309\)SimPO4129\.16783990\.4065w/ RGPO3355\.2889 \(↓\\downarrow773\.8789\)3232\.7181 \(↓\\downarrow757\.6884\)IPO4117\.78773990\.2490w/ RGPO3352\.7373 \(↓\\downarrow765\.0504\)3247\.7613 \(↓\\downarrow742\.4877\)
## Appendix CAdditional Results

### C\.1Sensitivity Analysis of Weight Scaling Coefficientλ\\lambda

To understand the influence of the scaling coefficientλ\\lambda, we analyze the consistency weight evolution and resulting model performance\. As illustrated in Figure[6](https://arxiv.org/html/2607.20515#A3.F6), increasingλ\\lambdaresults in a decrease in the overall magnitude of the consistency weights and reduced step\-wise deviation\. This indicates that larger coefficients induce a more conservative and stable weighting mechanism\. On the other hand, Table[10](https://arxiv.org/html/2607.20515#A3.T10)shows a divergence between benchmarks: \(i\) on AlpacaEval 2, the model achieves the highest Length\-Controlled \(LC\) Win Rate \(39\.08%39\.08\\%\) with a high coefficient ofλ=3\.0\\lambda=3\.0, suggesting that this metric favors the rigorous stability provided by aggressive scaling; \(ii\) conversely, performance on Arena\-Hard peaks atλ=1\.0\\lambda=1\.0\(46\.60%46\.60\\%\) and degrades with larger coefficients\. This is likely due to the distinct nature of the evaluation sets\. While AlpacaEval 2 benefits from the noise reduction of a highly regularized objective, Arena\-Hard requires a more balanced configuration for complex reasoning tasks that retains sufficient sensitivity to the supervision signal without over\-penalizing potential outliers\.

Figure 6:Comparison of consistency weight by Tanh scaling with differentλ\\lambda\.Table 10:Evaluation results of different weight scaling coefficientsλ\\lambda\.Scaling Coefficientλ\\lambda0\.51\.01\.52\.02\.53\.0AlpacaEval 2Raw Win Rate\(%\)39\.0540\.7439\.6939\.4639\.0141\.85LC Win Rate\(%\)36\.0738\.3037\.6536\.8836\.4039\.08Arena\-HardWin Rate\(%\)44\.4046\.6044\.3046\.2045\.3045\.20
### C\.2Additional Analysis of Consistency Weight Scaling

To further investigate the impact of consistency weight scaling, we analyze the training trend presented in Figure[7](https://arxiv.org/html/2607.20515#A3.F7)with a focus on training accuracy and the reward margin\. As shown in Figure[7\(a\)](https://arxiv.org/html/2607.20515#A3.F7.sf1), the Tanh scaling consistently maintains the highest training accuracy throughout the optimization process\. This sustained accuracy aligns with the best evaluation performance reported in Table[5](https://arxiv.org/html/2607.20515#S4.T5), which confirms that the Tanh scaling most effectively preserves the preference signal\. Figure[7\(b\)](https://arxiv.org/html/2607.20515#A3.F7.sf2)reveals that despite achieving the highest accuracy, the Tanh scaling shows a comparatively lower training reward margin than the unscaled baseline or other scaling methods\. This suggests that Tanh scaling ensures the model correctly distinguishes between chosen and rejected responses without forcing an excessively large log\-probability gap, leading to a more robust and generalizable policy\.

\(a\)Training Accuracy\(b\)Training Reward Margin
Figure 7:Effectiveness of different consistency weight scaling methods on training performance\.

## Appendix DQualitative Analysis

To provide a concrete understanding of how RGPO distinguishes between robust and inconsistent supervision signals, we present a qualitative analysis of samples from the training datasets\. As demonstrated in Tables[11](https://arxiv.org/html/2607.20515#A4.T11),[12](https://arxiv.org/html/2607.20515#A4.T12),[13](https://arxiv.org/html/2607.20515#A4.T13),[14](https://arxiv.org/html/2607.20515#A4.T14), we categorize them based on their estimated consistency levels\. By examining the relationship between the nature of the instruction, the diversity of annotator feedback, and the calculated consistency weights, we can validate whether the model correctly identifies reliable training signals\.

In scenarios with objective facts, such as mathematical reasoning or well\-defined logical constraints, human annotators typically demonstrate a high degree of consensus\. As observed in the high\-consistency examples, the responses are judged based on verifiable accuracy or completeness, showing unanimous or near\-unanimous voting patterns\. Under these conditions, RGPO assigns a substantial consistency weight to the samples, signaling that the estimated preference label represents a reliable ground truth\. This high weighting ensures that the optimization process prioritizes these clear signals, and allows the model to confidently learn from data where the distinction between the chosen and rejected response is indisputable\.

In contrast, tasks with open\-ended creative writing or subjective advice often lead to significant disagreement among annotators, reflecting the inherent ambiguity of human preference\. In such low\-consistency examples, voting patterns are frequently fragmented and without a single decisive response\. Existing methods would typically force a binary label from this inconsistency, potentially confusing the model\. However, RGPO detects the high variance in annotator reliability and assigns a minimal consistency weight to these pairs\. By reducing the influence of these contentious samples, the framework prevents models from overfitting to inconsistency, which ensures that the alignment process is driven by clear signals\.

Table 11:High consistency task sample on MultiPref\.DatasetMultiPrefInstructionWhat is a Sierpinski triangle?Response AA Sierpinski triangle is a fractal that is formed by iteratively removing triangles from an equilateral triangle\. The fractal is named after the Polish mathematician Wacław Sierpiński, who studied it in the 1910s\. To create a Sierpinski triangle, you start with an equilateral triangle and then remove the smallest equilateral triangle that can be inscribed within it\. You then repeat the process with each of the remaining smaller triangles\. This process can be continued indefinitely, resulting in a self\-similar pattern of smaller and smaller triangles that make up the fractal\. The Sierpinski triangle is an example of a self\-similar fractal, meaning that it has the same pattern at different scales\. It has a fractal dimension oflog⁡\(3\)/log⁡\(2\)\\log\(3\)/\\log\(2\), which is approximately 1\.585\. This means that the Sierpinski triangle has a fractional dimension that is between that of a line \(dimension 1\) and a plane \(dimension 2\)\.Response BA Sierpinski triangle is a fractal and an attractive fixed set named after the Polish mathematician Wacław Sierpiński who described its distinctive properties in 1915\. However, its design dates back to the 13th century and has been used in various cultures’ art\. The Sierpinski triangle exhibits a recursive, self\-similar structure at different scales\. It’s created using a simple method of subtraction: 1\. Start with an equilateral triangle\. 2\. Subdivide the triangle into four smaller congruent equilateral triangles by connecting the midpoints of each side\. 3\. Remove the central triangle, leaving three equilateral triangles\. 4\. Repeat the process indefinitely\. The Sierpinski triangle demonstrates several interesting properties: It has a fractal dimension oflog⁡\(3\)/log⁡\(2\)\\log\(3\)/\\log\(2\), approximately 1\.585\. It has zero area because with each iteration, the total area of the triangles removed accumulates to the area of the original triangle\. It is self\-similar as the entire figure is made up of smaller versions of itself\. Fractals like the Sierpinski triangle have applications in computer graphics, antenna theory, and nature, where similar patterns can be observed\.AnnotationsAnnotator 0: Response B
Annotator 1: Response B
Annotator 2: Response B
Annotator 3: Response BRGPO Estimated PreferenceResponse B winsScaled Consistency Weight0\.761594Table 12:Low consistency task sample on MultiPref\.DatasetMultiPrefInstructionWhat are good ways to keep my food warm?Response AThere are several ways to keep food warm without using a microwave or oven: 1\. Keep food hot in an insulated container: Fill a container with hot water, place your food in the container, and then seal it\. This should keep the food warm for several hours\. 2\. Use a food warmer: If you have a large group of people, you can use a food warmer to keep the food hot\. 3\. Use a hot pot: Place the pot on a stove or heater and add water\. Once the water is hot, you can place your food in the pot and cover it\. 4\. Use a chafing dish: These are designed to keep food warm for a longer period of time\. 5\. Keep food warm in a slow cooker: Once the food is cooked, transfer it to the slow cooker and keep it warm on low heat\. 6\. Use a warming drawer: Simply place the food in the drawer and keep it on low heat\. Remember to never leave food sitting out for more than 2 hours\.Response B1\. Use a slow cooker or crockpot to keep dishes warm for longer periods of time\. 2\. Invest in an insulated food carrier or thermal bag to transport and keep food warm\. 3\. Keep food warm in the oven on low heat \(around 200\-250 degrees Fahrenheit\) until ready to serve\. 4\. Use a chafing dish or buffet server with a warming tray to keep multiple dishes warm at once\. 5\. Wrap dishes in aluminum foil to help retain heat and keep food warm\. 6\. Place a towel or cloth over dishes to trap heat and keep food warm\. 7\. Serve food in a preheated dish or platter to help retain warmth\. 8\. Use a portable food warmer or hot plate to keep food warm on the go\. 9\. Keep food covered to prevent heat from escaping and retain warmth\. 10\. Consider using a microwave or stovetop to quickly reheat food before serving\.AnnotationsAnnotator 0: Response A
Annotator 1: Response A
Annotator 2: Response B
Annotator 3: Response BRGPO Estimated PreferenceResponse A winsScaled Consistency Weight0\.024345Table 13:High consistency task sample on Helpsteer2\.DatasetHelpsteer2\-DisagreementInstructionUniform Distribution:The idea extends to higher dimensions, with relative lengths replaced by relative areas or relative volumes\. For example, a random point\(X,Y\)\(X,Y\)in the plane has a uniform distribution onDD, whereDDis a region of the plane with finite area, if:\(i\)\(X,Y\)\(X,Y\)is certain to lie inDD;\(ii\) The chance that\(X,Y\)\(X,Y\)falls in a subregionCCofDDis proportional to the area ofCC:P​\(\(X,Y\)∈C\)=area​\(C\)area​\(D\)for​C⊂DP\(\(X,Y\)\\in C\)=\\frac\{\\text\{area\}\(C\)\}\{\\text\{area\}\(D\)\}\\quad\\text\{for \}C\\subset DExplain this in detail\.Response AThe uniform distribution is a probability distribution where each outcome has an equal probability of occurring\. In the case of a random point\(X,Y\)\(X,Y\)in the plane, the uniform distribution means that the chance of the point falling in any particular region of the plane is proportional to the area of that region\.To see this in action, let’s consider a simple example\. Suppose we have a regionDDthat is a square with sides of length 1 unit\. If we want to randomly choose a point\(X,Y\)\(X,Y\)in this region, we can do so by picking a random number between 0 and 1 for bothXXandYY\.For example, if we pick 0\.5 forXXand 0\.75 forYY, then the point\(X,Y\)\(X,Y\)will be in the center of the square\. If we pick 0\.1 forXXand 0\.9 forYY, then the point\(X,Y\)\(X,Y\)will be in the top right corner of the square\.Now, …Response BThe uniform distribution is a probability distribution that is constant over a specified range of values and zero outside of that range\. In other words, it represents a situation where all values within the range are equally likely to occur\.The idea of uniform distribution extends to higher dimensions, such as two or three dimensions, where instead of relative lengths, we use relative areas or relative volumes\.For example, consider a random point\(X,Y\)\(X,Y\)in the plane\. If the regionDDin the plane has a finite area, then the point\(X,Y\)\(X,Y\)has a uniform distribution onDDif:\(i\) The point\(X,Y\)\(X,Y\)is certain to lie inDD\.\(ii\) The chance that\(X,Y\)\(X,Y\)falls in a subregionCCofDDis proportional to the area ofCC\. In other words, the probability of\(X,Y\)\(X,Y\)falling inCCis equal to the ratio of th…AnnotationsAnnotator 0: Response A
Annotator 1: Response A
Annotator 2: Response A
Annotator 3: Response A
Annotator 4: Response A
Annotator 5: Response ARGPO Estimated PreferenceResponse A WinsScaled Consistency Weight0\.760612Table 14:Low consistency task sample on Helpsteer2\.DatasetHelpsteer2\-DisagreementInstructionwrite a grand description of a huge battle between the duchy of moon and the duchy of sunThe massive armies of the duchies of Moon and Sun faced each other across a vast battlefield, tens of thousands of troops arrayed in neat formations\. The sun shone brightly, glinting off the polished armor of the Sun troops, while the Moon troops were dressed in dark armor, their helmets shaped like crescent moons\. The two sides were completely evenly matched, and the battle was…Response AThe diplomatic meeting between the Yang and Yin kingdoms was a tense affair, with both sides carefully considering every word they spoke\. When the topic of the recent battle between the Duchies of Moon and Sun came up, the representatives from the Yang kingdom were quick to offer their congratulations to the Moon Duchy for their victory\.“We offer our congratulations on your victory in the recent battle,” said the representative from the Yang kingdom\. “It must have been a hard\-fought battle, and we are pleased to see that the Moon Duchy has emerged victorious\.”

The representative from the Moon Duchy thanked the Yang kingdom for their kind words, but was quick to point out that the battle had been a difficult one, with many sacrifices made on both sides…

Response BThe Yang Kingdom’s representative, Lord Yang, sat across from the Yin Kingdom’s representative, Lady Yin, in the diplomatic meeting room\. The two kingdoms had been at odds for years, and the recent battle between the duchies of Moon and Sun had only heightened tensions\.Lord Yang spoke first, his voice cold and hard\. “The Moon duchy’s victory was a decisive one\. It seems the Sun duchy was no match for them\.”

Lady Yin narrowed her eyes\. “And what of it? The Moon duchy is not part of the Yin Kingdom\. Their victory is of no concern to us\.”

Lord Yang leaned forward, his voice dropping to a whisper\. “But what if I told you that the Moon duchy has been receiving aid from the Yang Kingdom? That our weapons and supplies helped them defeat the Sun duchy?”

Lady Yin’s eyes widened in …

AnnotationsAnnotator 0: Response B
Annotator 1: Response A
Annotator 2: Response A
Annotator 3: Response B
Annotator 4: TieRGPO Estimated PreferenceResponse B WinsScaled Consistency Weight0\.033595