ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization

# ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
Source: [https://arxiv.org/html/2605.12667](https://arxiv.org/html/2605.12667)
Nirmal Patel University of Texas at Austin nirmpatel@utexas\.edu &Fei Wang Google feiwangnlp@google\.com &Inderjit S\. Dhillon Google isd@google\.com

###### Abstract

The alignment of Large Language Models \(LLMs\) increasingly relies on Reinforcement Learning from AI Feedback \(RLAIF\) for non\-verifiable domains such as long\-form question answering and open\-ended instruction following\. These domains often rely on LLM based auto\-raters to provide granular, multi\-tier discrete rewards \(e\.g\., 1\-10 rubrics\) that are inherently stochastic due to prompt sensitivity and sampling randomness\. We empirically verify the stochasticity of auto\-raters that can propagate and corrupt standard advantage estimators like GRPO and MaxRL, as a noisy reward samples can skew normalization statistics and degrade the global learning signal\. Empirically, sampling more rewards and taking majority voting may reduce the noise and improve performance, but this approach is computationally expensive\. To address this bottleneck, we introduceOrdinalDecomposition forRobustPolicyOptimization \(ODRPO\), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators\. By independently computing and accumulating advantages across these progressively challenging success thresholds, ODRPO prevents outlier evaluations from corrupting the global update while establishing an implicit, variance\-aware learning curriculum\. Empirically, ODRPO achieves robust performance on Qwen2\.5\-7B and Qwen3\-4B models, frequently outperforming baselines with relative improvements of upto 14\.8% on FACTS\-grounding\-v2 and 7\.5% on Alpaca\-Evals\. Critically, these gains are achieved with negligible training\-time overhead, as ODRPO requires no additional compute per step compared to standard estimators\. Supported by theoretical analysis confirming its optimization stability, ODRPO provides a scalable and robust framework for aligning models within the noisy, discrete evaluation landscape of modern RLAIF\.

## 1Introduction

Reinforcement Learning \(RL\) has become a prominent post\-training method in the field of language modeling\. With the notable success of reinforcement learning with verifiable rewards \(RLVR\) in fields like coding and math\(Guoet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib1)\), there has been a push to apply RLVR to other verifiable domains, such as puzzle solving, scientific literature review, and visual perception\(Chenet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib2); Burgesset al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib3); Wanget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib4)\)\. However, many tasks are inherently non\-verifiable, such as long\-form question answering and open\-ended instruction following, as they lack a fixed ground\-truth and cannot be trivially addressed by RLVR\(Liuet al\.,[2026b](https://arxiv.org/html/2605.12667#bib.bib5); Gunjalet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib32)\)\. For these open\-ended domains, auto\-raters or “LLM\-as\-judge” frameworks are preferred due to their rapid annotation capabilities and flexible rubric adherence, acting as surrogate reward functions that align closely with human preferences\(Leeet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib11); Gunjalet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib32)\)\. Auto\-raters also offer the flexibility to output discrete rewards based on detailed rubrics or partial scoring, providing more granular learning signals\(Kwoket al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib10)\)\. Consequently, reinforcement learning from AI feedback \(RLAIF\) has emerged as a scalable and reliable post\-training paradigm for non\-verifiable domains\(Leeet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib11); Liuet al\.,[2026b](https://arxiv.org/html/2605.12667#bib.bib5); Gunjalet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib32)\)\.

While RLAIF is becoming a standard practice in non\-verifiable domains, it introduces the challenge of noisy reward signals driven by the inherent stochasticity of the auto\-rater\. This noise can stem from prompt sensitivity, position bias, and rubric misinterpretation\(Zhaoet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib6); Shiet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib7); Liet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib8)\)\. This unpredictability poses a significant challenge for standard advantage estimators like GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib13)\)and MaxRL\(Tajwaret al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib16)\), which implicitly assume reliability of the reward signal\. Under the stochasticity of the auto\-rater, the advantage estimates can become corrupted\. A single noisy reward sample can skew the normalization statistics, negatively impacting the entire group’s update\. While Monte\-Carlo sampling could mitigate this variance, it remains computationally expensive and intractable at scale, as it multiplies the cost of auto\-rater calls, which are often more expensive than the response sampling from the policy model\.

In this work, we proposeOrdinalDecomposition forRobustPolicyOptimization \(ODRPO\), a framework designed to stabilize the optimization signal in RLAIF within discrete reward settings\. We decompose the single scalar reward into multiple sub\-rewards representing ordinal success levels, compute the advantage for each level independently, and then accumulate these values, effectively transforming a single\-reward optimization into a multi\-reward formulation\. Decomposing the reward into ordinal thresholds confines evaluator stochasticity to narrow boundary strata, preventing a single noisy evaluation from corrupting the global mean and variance used for normalization, and preserving a stable learning signal across the rest of the reward distribution\. Furthermore, we introduce variance\-aware weighting schemes, utilizing dynamic weighting functions that systematically suppress the noise originating from completely solved or currently unreachable evaluation criteria, focusing the model’s capacity on active learning frontiers\. Crucially, this structural decomposition introduces no measurable training\-time overhead, making ODRPO a highly efficient “plug\-and\-play” augmentation for traditional advantage estimators in large\-scale alignment workflows\.

We evaluated ODRPO in the RLAIF setting against standard GRPO and MaxRL estimators on three diverse benchmarks: FACTS\-grounding\-v2\(Chenget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib26)\), Alpaca\-Eval\(Liet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib30)\), and IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib22)\)\. Our empirical results demonstrate that ODRPO consistently improves upon baselines, achieving relative performance gains of upto 14\.8% on FACTS\-grounding\-v2 and 7\.5% on Alpaca\-Evals\. These gains highlight the framework’s ability to extract robust alignment signals from stochastic evaluators\. Alongside these empirical relative improvements, we provide a theoretical analysis illustrating how ordinal decomposition ensures the estimator admits a well\-defined global scalar objective, thereby stabilizing the optimization trajectory\.

Our core contributions are threefold\. First, we empirically identify the stochasticity of auto\-raters in multi\-tier discrete reward space and analyze its variance and effects on rank\-flips\. Second, we introduce ODRPO, a novel advantage estimation framework featuring ordinal reward decomposition and Variance\-Aware Weighting Schemes to structurally isolate and suppress evaluator noise\. Third, we provide comprehensive empirical validation demonstrating robust improvements across diverse open\-ended alignment tasks, supported by theoretical analysis confirming the framework’s stability\.

## 2Related Work

The alignment of Large Language Models \(LLMs\) with human intent and logical consistency has primarily consolidated around the framework of Reinforcement Learning from Human Feedback \(RLHF\)\. Early advancements were largely catalyzed by Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.12667#bib.bib12)\), which relies on a centralized critic network to mitigate gradient variance\. While PPO remains a robust baseline, the substantial computational overhead required to maintain auxiliary value models has motivated the search for more resource\-efficient alternatives\. A significant development in this direction is Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib13)\), which avoids the traditional critic in favor of computing advantages relative to a group of sampled outputs\. By utilizing these local surrogates, GRPO significantly reduces memory footprint while maintaining high performance on complex mathematical reasoning tasks\.

Concurrently, Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib35)\)has emerged as a prominent offline alternative, reformulating the alignment task into a supervised cross\-entropy loss\. While DPO provides theoretical clarity for preference\-based data, it is inherently limited by the diversity of static datasets and lacks the dynamic exploration capabilities required for reasoning\-intensive domains\(Mohammadiet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib36)\)\. For these tasks, online reinforcement learning continues to be the preferred paradigm\. To further scale these methods, Reinforcement Learning from AI Feedback \(RLAIF\)\(Leeet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib11)\)utilizes high\-capacity models as proxy evaluators, providing a scalable mechanism for generating synthetic reward signals\. This shift toward automated, multi\-faceted feedback further necessitates a rigorous theoretical understanding of the objectives that online estimators actually optimize\.

Recent theoretical advancements have formalized the global optimization landscape of policy gradient methods, demonstrating that prominent advantage estimators such as GRPO, Rejection Sampling, and REINFORCE admit well\-defined global scalar objectives in binary reward settings\(Davis and Recht,[2025](https://arxiv.org/html/2605.12667#bib.bib15)\)\. Building progressively upon this framework, MaxRL delineates a path to invoke maximum\-likelihood behavior by analyzing the theoretical connection between rejection sampling and the Maclaurin series expansion of the logarithm, establishing that the estimator converges to a true maximum\-likelihood objective as the group size increases\(Tajwaret al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib16)\)\. This theoretical convergence is supported by experiments demonstrating that MaxRL achieves performance closely matching standard cross\-entropy baselines\.Tangentially, GDPO identifies a structural limitation in GRPO when applied to multiple binary reward environments, where bundling distinct reward tuples into a single advantage value inherently reduces signal expressiveness\(Liuet al\.,[2026a](https://arxiv.org/html/2605.12667#bib.bib14)\)\. To address this, GDPO decouples the advantage calculation for each individual reward prior to accumulation, thereby preserving a denser and more informative optimization signal than standard GRPO\(Liuet al\.,[2026a](https://arxiv.org/html/2605.12667#bib.bib14)\)\.

ODRPO synthesizes and extends these foundational principles to address the unique challenges of arbitrary discrete reward spaces\. While previous analyses established scalar objectives for binary outcomes\(Davis and Recht,[2025](https://arxiv.org/html/2605.12667#bib.bib15)\), extending these estimators to arbitrary discrete rewards introduces update field asymmetries that make the scalar objectives theoretically inadmissible\. Consequently, standard estimators like GRPO and MaxRL fail to admit global scalar objectives in this context\. ODRPO resolves this theoretical gap by providing a formal framework that guarantees estimators with valid binary objectives retain a global scalar objective in the discrete reward setting\. Furthermore, while ODRPO leverages decoupled advantage calculations akin to GDPO, it applies this mechanism to create ordinal levels within a single discrete reward scale rather than parallel binary tasks\. In this single discrete reward context, GDPO remains equivalent to GRPO and thus suffers from the same theoretical limitations\. This success\-ordinal decomposition establishes an implicit curriculum\(Narvekaret al\.,[2020](https://arxiv.org/html/2605.12667#bib.bib31)\), structurally guiding the policy optimization process across progressively challenging quality thresholds\.

## 3Auto\-rater Rewards Exhibit High Variance

RLAIF has become increasingly important and widely used when extending from RLVR to non\-verifiable domains\. However, evidence from recent studies suggests that LLM\-based auto\-raters often exhibit high variance due to prompt sensitivity and sampling randomness\(Zhaoet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib6); Shiet al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib7); Liet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib8)\)\. This raises concerns about whether such stochasticity manifests as inconsistent reward signals, which would fundamentally undermine their reliability as stable feedback for RL\.

An auto\-rater is characterized as stochastic ifNNindependent evaluations ofMMresponses to a fixed prompt yield frequent rank\-flips, manifesting as a lack of consensus and high evaluative variance\. Rank agreement can be quantified using Kendall’s coefficient of concordance, or Kendall’sWW\(Kendall and Smith,[1939](https://arxiv.org/html/2605.12667#bib.bib37)\)\. IfW∼0W\\sim 0then the auto\-rater is highly inconsistent and ifW∼1W\\sim 1then the auto\-rater is consistent\. We conjecture thatW≥0\.9W\\geq 0\.9indicates that the auto\-rater is fairly consistent for stable post\-training as theNN“virtual judges” are derived from the same auto\-rater\.

We performed a statistical analysis using Qwen3\-30B\-A3B\-Instruct\-2507\(Team,[2025](https://arxiv.org/html/2605.12667#bib.bib18)\)as auto\-rater and Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib17)\)as the response generator on 1000 randomly sampled datapoints from the Ultrafeedback dataset\(Cuiet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib20)\)\. This sample of 1000 datapoints is representative because the Ultrafeedback dataset is inherently broad and rich\(Jianget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib40); Denget al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib41)\)\. For each datapoint, we generatedM=8M=8responses and for each response we extractedN=16N=16scores between 1\-10; this gives 1000M×NM\\times\{\}Nmatrices\. We performed the test of significance of Kendall’s coefficient of concordance \(Kendall’s W\) on the 1000M×NM\\times\{\}Nmatrices through Pingouin\(Vallat,[2018](https://arxiv.org/html/2605.12667#bib.bib38)\)\. The Kendall’s W value for different datapoints is shown in[fig\.˜1](https://arxiv.org/html/2605.12667#S3.F1), and further analysis is provided in[appendix˜A](https://arxiv.org/html/2605.12667#A1)\.

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/kendall_w_histogram.jpg)Figure 1:Kendall’s coefficient of concordance for 1,000 datapoints from the Ultrafeedback dataset\(Cuiet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib20)\)for Qwen3\-30B\-A3B\-Instruct\-2507\(Team,[2025](https://arxiv.org/html/2605.12667#bib.bib18)\)as auto\-rater and Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib17)\)as the response generator\. A median Kendall’s W of0\.6140\.614with a heavy left tail indicates that the auto\-rater has noticeable stochasticity, making its judgment less reliable\.This empirical assessment demonstrates that high\-capacity models like Qwen3 can produce inconsistent scoring distributions for identical inputs, resulting in a high\-variance reward signal\. This stochasticity propagates through the advantage estimator, potentially corrupting the gradient signal and destabilizing the optimization process\. These findings highlight the necessity of a variance\-aware advantage estimator–one capable of dissipating evaluative noise across ranking levels to localize and mitigate its impact on policy updates\.

## 4ODRPO: Ordinal Decomposition for Robust Policy Optimization

2

AdvantageEstimatorFormulaDescriptionGRPOAi=ri−μσA\_\{i\}=\\frac\{r\_\{i\}\-\\mu\}\{\\sigma\}Z\-Score normalization of the rewardsMaxRLAi=ri−μμA\_\{i\}=\\frac\{r\_\{i\}\-\\mu\}\{\\mu\}Normalization by mean to approachmaximum\-likelihoodODRPOAi=∑k=1KAi\(k\)A\_\{i\}=\\sum\_\{k=1\}^\{K\}A\_\{i\}^\{\(k\)\}whereAi\(k\)=ri\(k\)−μ\(k\)N\(k\)A\_\{i\}^\{\(k\)\}=\\frac\{r\_\{i\}^\{\(k\)\}\-\\mu^\{\(k\)\}\}\{N^\{\(k\)\}\},ri\(k\)=𝕀​\{ri≥k\}r\_\{i\}^\{\(k\)\}=\\mathbb\{I\}\\\{r\_\{i\}\\geq k\\\}For a discrete integer reward in range\{1,…,K\}\\\{1,\\dots,K\\\}, first compute advantageof each ordinal level and thenaccumulateTable 1:Formulae and description of different advantage estimators\.rir\_\{i\}is theithi^\{\\text\{th\}\}rollout’s reward andAiA\_\{i\}is its advantage value\.μ\\muandσ\\sigmaare the group’s mean and standard deviation whileμ\(k\),σ\(k\),N\(k\)\\mu^\{\(k\)\},\\sigma^\{\(k\)\},N^\{\(k\)\}are thekthk^\{\\text\{th\}\}level’s mean, standard deviation, and normalization, respectively\. For GRPO\-like normalizationN\(k\)=σ\(k\)N^\{\(k\)\}=\\sigma^\{\(k\)\}and for MaxRL\-like normalizationN\(K\)=μ\(k\)N^\{\(K\)\}=\\mu^\{\(k\)\}\.To address the instability caused by auto\-rater stochasticity in multi\-tier reward spaces, we propose theOrdinalDecomposition forRobustPolicyOptimization \(ODRPO\) framework\. Instead of treating a discrete rubric score as a single continuous value, ODRPO restructures the objective by decomposing the reward into a monotonic sequence of binary sub\-tasks\. This transformation isolates evaluation noise and establishes a stable, variance\-aware learning trajectory\. The formulations of ODRPO along with GRPO and MaxRL and their descriptions are provided in[table˜1](https://arxiv.org/html/2605.12667#S4.T1)\.

### 4\.1Ordinal Decomposition

To align models with multi\-tier evaluation signals, the advantage estimator must accommodate the discrete structure of reward rubrics\. Instead of aggregating these signals into a single cardinal value, we propose a success\-ordinal decomposition that treats rewards as a sequence of progressive success thresholds\. Given a discrete reward spaceℛ:=\{1,2,…,K\}\\mathcal\{R\}:=\\\{1,2,\\ldots,K\\\}within a group𝒢\\mathcal\{G\}, we decompose the observed rewardrir\_\{i\}into a set of binary indicatorsri\(k\)r\_\{i\}^\{\(k\)\}, denoting whether theithi^\{\\text\{th\}\}rollout meets thekthk^\{\\text\{th\}\}quality level\. By treating the discrete reward space as an ordered sequence of sub\-tasks rather than a monolithic continuous variable, this decomposition preserves the granularity of the evaluation signal\.

ODRPO applies decoupled normalization to the ordinal levels of a single discrete scale and then aggregates\. We compute the advantage for each ordinal bin independently and accumulate them to yield the final advantage:

ri\\displaystyle r\_\{i\}=∑k=1Kri\(k\),ri\(k\)=𝕀​\{ri≥k\},\\displaystyle=\\sum\_\{k=1\}^\{K\}r\_\{i\}^\{\(k\)\},\\quad r\_\{i\}^\{\(k\)\}=\\mathbb\{I\}\\\{r\_\{i\}\\geq k\\\},Ai\(k\)\\displaystyle A\_\{i\}^\{\(k\)\}=ri\(k\)−μ\(k\)N\(k\),\\displaystyle=\\frac\{r\_\{i\}^\{\(k\)\}\-\\mu^\{\(k\)\}\}\{N^\{\(k\)\}\},Ai\\displaystyle A\_\{i\}=∑k=1KAi\(k\),\\displaystyle=\\sum\_\{k=1\}^\{K\}A\_\{i\}^\{\(k\)\},
whereμ\(k\)=𝔼i∈𝒢​\[ri\(k\)\]\\mu^\{\(k\)\}=\\mathbb\{E\}\_\{i\\in\\mathcal\{G\}\}\[r\_\{i\}^\{\(k\)\}\]is the mean of thekthk^\{\\text\{th\}\}bin, andN\(k\)N^\{\(k\)\}is a bin\-specific normalization factor \(e\.g\., standard deviation for GRPO or mean for MaxRL\)\. This independent normalization across bins ensures that the variance from highly concentrated reward distributions at lower thresholds does not improperly scale the advantage computation at sparse, higher\-tier boundaries\.

By partitioning the reward space into success indicators, ODRPO maintains the monotonic progression of evaluation rubrics\. This formulation introduces an implicit curriculum\(Narvekaret al\.,[2020](https://arxiv.org/html/2605.12667#bib.bib31)\), structuring the policy update to stabilize fundamental quality thresholds before optimizing for higher success levels\. Moreover, by structurally isolating distinct performance boundaries, ODRPO prevents optimization gradients from being dominated by outlier responses or trivial successes\. This localized advantage calculation directly translates dense qualitative rubrics into actionable, bounded updates\.

### 4\.2Variance\-Aware Weighting Schemes

The success\-ordinal framework permits weighting the optimization signal across different reward thresholds\. While linear weighting can emphasize higher\-tier rewards, we present two variance\-aware schemes designed to focus updates on the most informative thresholds: \(i\) Gini weighting and \(ii\) Gini\-Median weighting\. These dynamic weighting functions systematically suppress the noise originating from completely solved or currently unreachable evaluation criteria\.

The Gini weighting scheme uses scaled Gini impurity,4​μ\(k\)​\(1−μ\(k\)\)4\\mu^\{\(k\)\}\(1\-\\mu^\{\(k\)\}\), to assign higher weights to bins with a mean reward near 0\.5\. This emphasizes thresholds that are neither trivially satisfied nor overly difficult under the current policy\. We incorporate ak\\sqrt\{k\}factor to maintain optimization pressure toward higher quality:

wGini\(k\)\\displaystyle w^\{\(k\)\}\_\{\\text\{Gini\}\}=k⋅\(0\.1\+4​μ\(k\)​\(1−μ\(k\)\)\)\.\\displaystyle=\\sqrt\{k\}\\cdot\(0\.1\+4\\mu^\{\(k\)\}\(1\-\\mu^\{\(k\)\}\)\)\.\(1\)
The Gini\-Median scheme modulates the Gini\-impurity using a biased exponential decay relative to the group’s median bin indexM𝒢M\_\{\\mathcal\{G\}\}\. This directs the optimization toward the median\-to\-higher tiers, anchoring updates to the model’s current performance:

wGini\-Med\(k\)\\displaystyle w^\{\(k\)\}\_\{\\text\{Gini\-Med\}\}=k⋅\(0\.1\+4​μ\(k\)​\(1−μ\(k\)\)⋅exp⁡\(−max⁡\(M𝒢−k,0\)2\)\)\.\\displaystyle=\\sqrt\{k\}\\cdot\\left\(0\.1\+4\\mu^\{\(k\)\}\(1\-\\mu^\{\(k\)\}\)\\cdot\\exp\\left\(\-\\frac\{\\max\(M\_\{\\mathcal\{G\}\}\-k,0\)\}\{2\}\\right\)\\right\)\.\(2\)
The accumulated advantage is then computed asAi=∑k=1Kw\(k\)​Ai\(k\)A\_\{i\}=\\sum\_\{k=1\}^\{K\}w^\{\(k\)\}A\_\{i\}^\{\(k\)\}, wherew\(k\)=1w^\{\(k\)\}=1recovers the unweighted aggregation\. As illustrated in[fig\.˜2](https://arxiv.org/html/2605.12667#S4.F2), these schemes allow ODRPO to adapt the optimization landscape based on the empirical reward distribution\. By concentrating learning capacity on active performance frontiers, the variance\-aware mechanisms accelerate convergence while preventing catastrophic forgetting of stabilized lower\-tier behaviors\. Consequently, the weighting strategies minimize the sample inefficiency associated with uniform reward distribution assumptions\. They provide a statistically principled method for adjusting the optimization target as the policy’s competency shifts across the ordinal spectrum\.

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/Weightings.jpg)Figure 2:Visualization of Gini and Gini\-Med weighting behaviors across four representative reward distributions: \(a\) standard normal, \(b\) concentrated peak, \(c\) normal with bi\-directional outliers, and \(d\) uniform\. While both variants dynamically adapt to the reward density to prioritize the learning frontier, Gini\-Med attenuates the weighting of reward levels below the median\. This modulation concentrates the optimization signal on the frontier and high\-difficulty levels, minimizing the influence of already resolved or saturated reward states on the policy update\.The accumulated advantage will be computed asAi=∑k=1Kw\(k\)​Ai\(k\)A\_\{i\}=\\sum\_\{k=1\}^\{K\}w^\{\(k\)\}A\_\{i\}^\{\(k\)\}, wherew\(k\)w^\{\(k\)\}are the weights assigned by the chosen weighting scheme\. The flexibility in weighting allows ODRPO to adapt to various reward structures and task requirements, making it a versatile framework for robust advantage estimation in preference finetuning\.

## 5Experiments

Setup\.We trained Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib17)\)on the Ultrafeedback dataset\(Cuiet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib20)\)using the VERL framework\(Shenget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib21)\)\. Qwen3\-30B\-A3B\-Instruct\-2507\(Team,[2025](https://arxiv.org/html/2605.12667#bib.bib18)\)served as the auto\-rater, providing integer rewards \(1–10\)\. To evaluate the robustness of our framework, we compared it against GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib13)\)and MaxRL\(Tajwaret al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib16)\)estimators\. We used batch normalization after advantage accumulation for numerical stability\(Liuet al\.,[2026a](https://arxiv.org/html/2605.12667#bib.bib14)\)\. All models were trained for one epoch with a batch size of 64 and 8 rollouts per group\. Detailed hyperparameters and prompt templates are provided in Appendix[appendix˜B](https://arxiv.org/html/2605.12667#A2)and selected training curves and their analysis is provided in Appendix[appendix˜C](https://arxiv.org/html/2605.12667#A3)\.

### 5\.1Evaluation Suite

We evaluated the models on three broad benchmarks: \(i\) FACTS\-grounding\-v2\(Chenget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib26)\), \(ii\) Alpaca\-Evals\(Liet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib30)\), and \(iii\) IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib22)\)\.

FACTS\-grounding\-v2 \(FACTS\)\.FACTS\-grounding\-v2 evaluates the factual grounding of the models by providing user query and context document \(upto 32K tokens\) and uses an ensemble of LLM judges to evaluate the response \(upto 1024 tokens\) by providing a binary score\(Chenget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib26)\)\. We used Gemini 3\.1 Flash\-Lite\(Google DeepMind,[2026](https://arxiv.org/html/2605.12667#bib.bib29)\)as a quality filtering judge\. The filtered responses were then evaluated by Gemini 3 Flash\(Google DeepMind,[2025b](https://arxiv.org/html/2605.12667#bib.bib28)\)and Gemini 2\.5 Flash\(Google DeepMind,[2025a](https://arxiv.org/html/2605.12667#bib.bib27)\)and the final score is the mean of the two judges\. The responses that failed the quality filter were assigned a score of 0\. All models’ temperatures were set to 0\. We report the overall score of each model on this benchmark\.

Alpaca\-Evals\.Alpaca\-Evals is an automatic, LLM\-based evaluator that evaluates models on 805 datapoints against a set baseline, which in our case is the policy model before training on the Ultrafeedback dataset\(Liet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib30)\)\. We have Gemini 3 Flash\(Google DeepMind,[2025b](https://arxiv.org/html/2605.12667#bib.bib28)\)as the LLM judge that chooses between two given responses and provides a ternary score: win, lose, or tie\. All models’ temperatures were set to 0\. We report the overall length\-controlled win rate of each model against the baseline\.

IFEval\.IFEval benchmarks models on instruction following capabilities through 25 types of instructions\(Zhouet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib22)\)\. We evaluated the models on the IFEval through the LM Eval Harness\(Gaoet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib23)\), which provides a standardized interface for evaluating language models across a wide range of tasks, and report the average of the strict and loose accuracies of the models on the prompt and instruction levels\.

### 5\.2Results

The results of individual benchmarks and their averages are provided in[table˜2](https://arxiv.org/html/2605.12667#S5.T2)\. The win\-rates for Alpaca\-Evals are calculated against the pre\-finetuned policy models\. Based on these data, we identify several key observations regarding the performance, robustness, and efficiency of the ODRPO framework\.

FACTS\-grounding\-v2Alpaca\-EvalsIFEvalMean𝚫\(%\)\\Delta\(\\%\)Qwen2\.5\-7B\-Instruct GRPOGRPO0\.19040\.56910\.64780\.46910ODRPO Gini \(GRPO\)0\.20090\.57110\.65260\.47491\.2364ODRPO Gini\-Med \(GRPO\)0\.19980\.58120\.66110\.48072\.4728Qwen2\.5\-7B\-Instruct MaxRLMaxRL0\.14190\.53230\.66120\.44510ODRPO Gini \(MaxRL\)0\.15950\.56910\.64510\.45792\.8758ODRPO Gini\-Med \(MaxRL\)0\.16290\.57210\.64410\.45973\.2802Qwen3\-4B\-Instruct GRPOGRPO0\.12150\.55290\.65190\.44210ODRPO Gini \(GRPO\)0\.13020\.56610\.64680\.44771\.2667ODRPO Gini\-Med \(GRPO\)0\.12970\.56410\.65190\.44861\.4703Qwen3\-4B\-Instruct MaxRLMaxRL0\.11740\.55330\.64290\.43790ODRPO Gini \(MaxRL\)0\.12740\.56340\.65410\.44832\.3750ODRPO Gini\-Med \(MaxRL\)0\.13320\.55970\.66440\.45243\.3113Table 2:Comprehensive benchmark results for Qwen2\.5\-7B and Qwen3\-4B policy models using standard and ODRPO\-enhanced estimators across various alignment tasks\. Evaluation metrics include factual grounding \(FACTS\-grounding\-v2\), conversational alignment \(Alpaca\-Evals\), and instruction following \(IFEval\)\. ODRPO variants \(Gini and Gini\-Med\) consistently outperform the base GRPO and MaxRL estimators across most benchmarks, with relative improvements up to3\.5%3\.5\\%\. Bold values denote the best performance in each column for each model\-estimator group\.Effectiveness Across Diverse Evaluation Benchmarks\.The ODRPO variants consistently outperform baseline estimators across all three evaluation dimensions: factual grounding, conversational alignment, and instruction following\. In the Qwen2\.5\-7B GRPO configuration, ODRPO Gini\-Med improves the FACTS\-grounding\-v2 score from 0\.1904 to 0\.1998 and the Alpaca\-Evals win\-rate from 0\.5691 to 0\.5812\. A similar trend is observed in the Qwen3\-4B MaxRL setup, where IFEval performance increases from 0\.6429 to 0\.6644\. These results indicate that by treating discrete rewards as ordinal levels rather than scalar, ODRPO effectively enhances the optimization signal\. This ordinal approach prevents the advantage estimator from being skewed by outlier reward values, which is particularly beneficial for precision\-critical tasks like factual grounding\.

Consistency Across Underlying Advantage Estimators\.The performance gains provided by ODRPO are agnostic to the underlying reinforcement learning objective, showing improvements in both group\-relative \(GRPO\) and maximum\-likelihood \(MaxRL\) settings\. For the Qwen2\.5\-7B model, ODRPO Gini\-Med yields a relative improvement \(Δ\\Delta\) of 2\.47% over the GRPO baseline and 3\.28% over the MaxRL baseline\. This consistency suggests that the success\-ordinal framework is not tied to a specific loss function but rather improves the fundamental quality of the advantage signal\. By providing a more reliable gradient direction across different algorithmic constraints, ODRPO serves as a robust drop\-in replacement for standard cardinal estimators\.

Generalizability Across Model Capacities\.The framework demonstrates scalability across different architecture sizes, with consistent gains observed for both the 7B and 4B models\. In the MaxRL configuration, ODRPO Gini\-Med achieves a relative gain of 3\.28% for the Qwen2\.5\-7B model and 3\.31% for the Qwen3\-4B model\. In the GRPO settings, relative improvements range from 1\.23% to 2\.47% across both scales\. These data points confirm that the benefits of ordinal partitioning are not architecture\-dependent\. The ability to maintain similar relative performance increases in smaller capacity models \(4B\) suggests that ODRPO is a viable strategy for optimizing models where the representational capacity for complex reward signals may be more limited\.

Computational Efficiency and Training Stability\.Empirical measurements confirm that the structural advantages of ODRPO are achieved without introducing significant computational overhead\. For instance, in the Qwen2\.5\-7B GRPO setting, ODRPO Gini\-Med requires30\.37±1\.1730\.37\\pm 1\.17s per step, compared to the baseline’s31\.14±2\.0031\.14\\pm 2\.00s\. Similar parity is observed in the Qwen3\-4B MaxRL configuration \(36\.04±1\.6936\.04\\pm 1\.69s vs\.36\.26±1\.7836\.26\\pm 1\.78s\)\. These results indicate that the framework extracts a higher\-fidelity signal from single\-sample rollouts without increasing the training\-time compute budget\. This efficiency makes ODRPO a resource\-efficient alternative to computationally expensive oversampling methods, providing a stable optimization trajectory through better structural modeling of the reward space rather than increased sample volume\.

Takeaway\.The experimental evidence indicates that representing rewards via ordinal levels offers a robust alternative to traditional cardinal advantage estimation\. ODRPO variants yield aggregated relative improvements of up to 3\.3% across diverse model capacities and advantage estimators\. By strategically discarding the assumptions of a linear reward scale, the framework provides a precise and enhances optimization objective that aligns with the fundamentally discrete nature of LLM reward signals\.

### 5\.3Majority Voting Ensemble Analysis

The inherent stochasticity of auto\-raters can be suppressed by oversampling rewards per rollout and then taking majority voting \(mode value\)\. To analyze ODRPO’s performance against MaxRL baseline in reward oversampling regime, we conducted post\-training withN=1,8,16,32N=1,8,16,32rewards per rollout with Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib17)\)policy model, Ultrafeedback dataset\(Cuiet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib20)\), and Qwen3\-30B\-A3B\-Instruct\-2507\(Team,[2025](https://arxiv.org/html/2605.12667#bib.bib18)\)auto\-rater\.

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/MajorityVotingEnsembleAnalysis.jpg)Figure 3:Alpaca\-Evals values and time per step in seconds for majority voting ensemble analysis\.N=1,8,16,32N=1,8,16,32auto\-rater calls were made to sample numeric rewards between 1–10 and then majority voting \(mode\) was used as the final reward for the corresponding rollout\. ODRPO variants are maintaining the improvement gap against the baseline at no additional compute time\.The Alpaca\-Eval values and time per step \(seconds\) for MaxRL against ODRPO methods are shown in[fig\.˜3](https://arxiv.org/html/2605.12667#S5.F3)\. The analysis concludes that ODRPO maintains the improvement gap against the baseline across all oversampling regimes while adding no additional training overhead\. Interestingly,N=1N=1ODRPO variants are observed to perform better than the best over\-sampled baseline \(Gini’s 0\.5690 and Gini\-Med’s 0\.5721 againstN=16N=16baseline’s 0\.5463\)\. Furthermore, whileN=32N=32MaxRL and ODRPO Gini MaxRL experienced a slight dip in performance, ODRPO Gini\-Med MaxRL continued to improve, reaching a score of 0\.5982\. This suggests a higher performance ceiling for ODRPO variants \(∼\\sim0\.60\) compared to MaxRL \(∼\\sim0\.55\), further attesting to ODRPO’s capability in decomposing rewards and localizing auto\-rater noise\. The corresponding training analysis is provided in[appendix˜E](https://arxiv.org/html/2605.12667#A5)\.

## 6Theoretical Analysis

Extending policy gradient methods to discrete reward spaces requires verifying that the advantage estimator optimizes a global scalar objective\. While previous work identifies these objectives for binary rewards\(Davis and Recht,[2025](https://arxiv.org/html/2605.12667#bib.bib15)\), our analysis \([section˜F\.2](https://arxiv.org/html/2605.12667#A6.SS2)\) indicates that applying estimators such as GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib13)\)or MaxRL\(Tajwaret al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib16)\)directly to multi\-tier discrete rewards violates the curl conditions necessary to admit a global scalar objective\. The failure of standard cardinal estimators stems from their gradient coupling across reward levels, which introduces path dependencies into the optimization trajectory\. Consequently, these methods operate on an asymmetric update field rather than a standard scalar function\.

The success\-ordinal decomposition of ODRPO addresses this limitation\. By expressing discrete rewards as a sum of Bernoulli variables, ODRPO allows estimators defined for binary settings to retain a valid scalar objective when applied to discrete multi\-tier spaces\. For a discrete reward spaceℛ:=\{R1,R2,…,RK\}\\mathcal\{R\}:=\\\{R\_\{1\},R\_\{2\},\\ldots,R\_\{K\}\\\}, ODRPO admits an objective functionJ​\(θ\)J\(\\theta\)with gradient:

∇θJ​\(θ\)=𝔼x∼𝒬​\[∑m=2KΔm​\(β​\(Pm\)−α​\(Pm\)\)​∇θPm\],\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\\nabla\_\{\\theta\}P\_\{m\}\\right\],\(3\)
whereΔm=Rm−Rm−1\\Delta\_\{m\}=R\_\{m\}\-R\_\{m\-1\}represents the reward spacing,Pm=∑j=mKpj​\(θ\|x\)P\_\{m\}=\\sum\_\{j=m\}^\{K\}p\_\{j\}\(\\theta\|x\)is the probability of achieving at least rewardRmR\_\{m\}, andβ​\(Pm\)\\beta\(P\_\{m\}\)andα​\(Pm\)\\alpha\(P\_\{m\}\)are derived from the underlying binary estimator \(derivation in[section˜F\.3](https://arxiv.org/html/2605.12667#A6.SS3)\)\. Using GRPO\-like normalization, the objective function is:

J​\(θ\)=𝔼x∼𝒬​\[2π​∑m=2KΔm​arcsin⁡\(Pm\)\]\.\\displaystyle J\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\frac\{2\}\{\\pi\}\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\\arcsin\\left\(\\sqrt\{P\_\{m\}\}\\right\)\\right\]\.\(4\)
This formulation demonstrates that ODRPO preserves the global optimization properties of binary estimators within multi\-tier reward environments\. By transforming the discrete objective into a linear combination of valid binary fields, ODRPO ensures the resulting landscape is a well\-behaved potential, thereby providing a theoretically grounded optimization path\. The mathematical recovery of a global scalar objective ensures that iterative policy updates reliably converge rather than oscillating within non\-conservative vector fields\. This property bridges the gap between scalable group\-relative estimation and the rigorous convergence guarantees typical of exact likelihood methods\.

## 7Conclusion

In this paper, we introduced Ordinal Decomposition for Robust Policy Optimization \(ODRPO\), a framework designed to stabilize the optimization signal in RLAIF against the inherent stochasticity of auto\-raters\. We identified that in multi\-tier discrete reward spaces, auto\-raters demonstrate noticeable stochasticity, making their rewards less reliable and corrupting the optimization signal\. ODRPO resolves this by decomposing discrete rewards into ordinal success levels, effectively confining evaluator noise to narrow boundary strata and preserving a stable learning signal across the rest of the distribution\. Through comprehensive empirical validation on diverse benchmarks, we demonstrate that ODRPO consistently extracts robust alignment signals from single\-sample evaluations, often matching or exceeding the performance of significantly more expensive oversampling methods\. Complemented by a theoretical analysis that confirms the restoration of a well\-defined global scalar objective, ODRPO offers a precise, resource\-efficient, and stable solution for aligning language models with the dense and stochastic feedback signals characteristic of modern evaluation rubrics\.

## References

- J\. Burgess, J\. N\. Hansen, D\. Peng, Y\. Zhang, A\. Lozano, M\. W\. Sun, E\. Lundberg, and S\. Yeung\-Levy \(2026\)PaperSearchQA: learning to search and reason over scientific papers with RLVR\.External Links:2601\.18207,[Link](https://arxiv.org/abs/2601.18207)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- J\. Chen, Q\. He, S\. Yuan, A\. Chen, Z\. Cai, W\. Dai, H\. Yu, Q\. Yu, X\. Li, J\. Chen, H\. Zhou, and M\. Wang \(2025\)Enigmata: scaling logical reasoning in large language models with synthetic verifiable puzzles\.External Links:2505\.19914,[Link](https://arxiv.org/abs/2505.19914)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- A\. Cheng, A\. Jacovi, A\. Globerson, B\. Golan,et al\.\(2025\)The facts leaderboard: a comprehensive benchmark for large language model factuality\.External Links:2512\.10791,[Link](https://arxiv.org/abs/2512.10791)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p4.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p2.1)\.
- G\. Cui, L\. Yuan, N\. Ding, G\. Yao, W\. Zhu, Y\. Ni, G\. Xie, Z\. Liu, and M\. Sun \(2023\)UltraFeedback: boosting language models with high\-quality feedback\.External Links:2310\.01377Cited by:[Figure 4](https://arxiv.org/html/2605.12667#A1.F4),[Figure 1](https://arxiv.org/html/2605.12667#S3.F1),[§3](https://arxiv.org/html/2605.12667#S3.p3.4),[§5\.3](https://arxiv.org/html/2605.12667#S5.SS3.p1.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1)\.
- D\. Davis and B\. Recht \(2025\)What is the objective of reasoning with reinforcement learning?\.External Links:2510\.13651,[Link](https://arxiv.org/abs/2510.13651)Cited by:[§F\.2](https://arxiv.org/html/2605.12667#A6.SS2.p1.1),[§F\.3](https://arxiv.org/html/2605.12667#A6.SS3.p6.2),[§2](https://arxiv.org/html/2605.12667#S2.p3.1),[§2](https://arxiv.org/html/2605.12667#S2.p4.1),[§6](https://arxiv.org/html/2605.12667#S6.p1.1)\.
- X\. Deng, H\. Zhong, R\. Ai, F\. Feng, Z\. Wang, and X\. He \(2026\)Less is more: improving llm alignment via preference data selection\.External Links:2502\.14560,[Link](https://arxiv.org/abs/2502.14560)Cited by:[§3](https://arxiv.org/html/2605.12667#S3.p3.4)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman,et al\.\(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p4.1)\.
- Google DeepMind \(2025a\)Gemini 2\.5 flash model card\.Note:Accessed: 2026\-04\-11External Links:[Link](https://deepmind.google/models/model-cards/)Cited by:[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p2.1)\.
- Google DeepMind \(2025b\)Gemini 3 flash: frontier intelligence built for speed\.Note:Accessed: 2026\-04\-11External Links:[Link](https://deepmind.google/models/model-cards/)Cited by:[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p3.1)\.
- Google DeepMind \(2026\)Gemini 3\.1 flash\-lite model card\.Note:Accessed: 2026\-04\-11External Links:[Link](https://deepmind.google/models/model-cards/)Cited by:[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p2.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2026\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=c1bTcrDmt4)Cited by:[§B\.3](https://arxiv.org/html/2605.12667#A2.SS3.p1.1),[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- Z\. Huang, Y\. Zhuang, G\. Lu, Z\. Qin, H\. Xu, T\. Zhao, R\. Peng, J\. Hu, Z\. Shen, X\. Hu, X\. Gu, P\. Tu, J\. Liu, W\. Chen, Y\. Fu, Z\. Fan, Y\. Gu, Y\. Wang, Z\. Yang, J\. Li, and J\. Zhao \(2025\)Reinforcement learning with rubric anchors\.External Links:2508\.12790,[Link](https://arxiv.org/abs/2508.12790)Cited by:[§B\.3](https://arxiv.org/html/2605.12667#A2.SS3.p1.1)\.
- L\. Jiang, Y\. Wu, J\. Xiong, J\. Ruan, Y\. Ding, Q\. Guo, Z\. Wen, J\. Zhou, and X\. Deng \(2025\)Hummer: towards limited competitive preference dataset\.External Links:2405\.11647,[Link](https://arxiv.org/abs/2405.11647)Cited by:[§3](https://arxiv.org/html/2605.12667#S3.p3.4)\.
- M\. G\. Kendall and B\. B\. Smith \(1939\)The problem ofmmrankings\.The Annals of Mathematical Statistics10\(3\),pp\. 275–287\.External Links:ISSN 00034851,[Link](http://www.jstor.org/stable/2235668)Cited by:[§3](https://arxiv.org/html/2605.12667#S3.p2.7)\.
- J\. Kwok, S\. Li, P\. Atreya, Y\. Liu, M\. Pavone, I\. Stoica, and A\. Mirhoseini \(2026\)LLM\-as\-a\-verifier: a general\-purpose verification framework\.Note:Notion BlogCited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Appendix A](https://arxiv.org/html/2605.12667#A1.p1.3),[§B\.1](https://arxiv.org/html/2605.12667#A2.SS1.p3.1),[§B\.2](https://arxiv.org/html/2605.12667#A2.SS2.p1.1)\.
- H\. Lee, S\. Phatale, H\. Mansoor, K\. R\. Lu, T\. Mesnard, J\. Ferret, C\. Bishop, E\. Hall, V\. Carbune, and A\. Rastogi \(2024\)RLAIF: scaling reinforcement learning from human feedback with AI feedback\.External Links:[Link](https://openreview.net/forum?id=AAxIs3D2ZZ)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1),[§2](https://arxiv.org/html/2605.12667#S2.p2.1)\.
- Q\. Li, S\. Dou, K\. Shao, C\. Chen, and H\. Hu \(2026\)Evaluating scoring bias in LLM\-as\-a\-judge\.External Links:2506\.22316,[Link](https://arxiv.org/abs/2506.22316)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p2.1),[§3](https://arxiv.org/html/2605.12667#S3.p1.1)\.
- X\. Li, T\. Zhang, Y\. Dubois, R\. Taori, I\. Gulrajani, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2023\)AlpacaEval: an automatic evaluator of instruction\-following models\.GitHub\.Note:[https://github\.com/tatsu\-lab/alpaca\_eval](https://github.com/tatsu-lab/alpaca_eval)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p4.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p3.1)\.
- S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng, Y\. Choi, J\. Kautz, and P\. Molchanov \(2026a\)GDPO: group reward\-decoupled normalization policy optimization for multi\-reward rl optimization\.External Links:2601\.05242,[Link](https://arxiv.org/abs/2601.05242)Cited by:[§2](https://arxiv.org/html/2605.12667#S2.p3.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1)\.
- Y\. Liu, Y\. Yu, D\. Su, S\. Wang, X\. Wang, S\. Jiang, B\. Liu, A\. Cohan, Y\. Tian, and Z\. Chen \(2026b\)Examining reasoning llms\-as\-judges in non\-verifiable llm post\-training\.External Links:2603\.12246,[Link](https://arxiv.org/abs/2603.12246)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- H\. Mohammadi, T\. Kozak, and A\. Giachanou \(2025\)Evaluating grpo and dpo for faithful chain\-of\-thought reasoning in llms\.External Links:2512\.22631,[Link](https://arxiv.org/abs/2512.22631)Cited by:[§2](https://arxiv.org/html/2605.12667#S2.p2.1)\.
- S\. Narvekar, B\. Peng, M\. Leonetti, J\. Sinapov, M\. E\. Taylor, and P\. Stone \(2020\)Curriculum learning for reinforcement learning domains: a framework and survey\.External Links:2003\.04960,[Link](https://arxiv.org/abs/2003.04960)Cited by:[§2](https://arxiv.org/html/2605.12667#S2.p4.1),[§4\.1](https://arxiv.org/html/2605.12667#S4.SS1.p4.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[§2](https://arxiv.org/html/2605.12667#S2.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§2](https://arxiv.org/html/2605.12667#S2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p2.1),[§2](https://arxiv.org/html/2605.12667#S2.p1.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1),[§6](https://arxiv.org/html/2605.12667#S6.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§5](https://arxiv.org/html/2605.12667#S5.p1.1)\.
- L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi \(2025\)Judging the judges: a systematic study of position bias in LLM\-as\-a\-judge\.External Links:2406\.07791,[Link](https://arxiv.org/abs/2406.07791)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p2.1),[§3](https://arxiv.org/html/2605.12667#S3.p1.1)\.
- F\. Tajwar, G\. Zeng, Y\. Zhou, Y\. Song, D\. Arora, Y\. Jiang, J\. Schneider, R\. Salakhutdinov, H\. Feng, and A\. Zanette \(2026\)Maximum likelihood reinforcement learning\.External Links:2602\.02710,[Link](https://arxiv.org/abs/2602.02710)Cited by:[§F\.1](https://arxiv.org/html/2605.12667#A6.SS1.p6.4),[§1](https://arxiv.org/html/2605.12667#S1.p2.1),[§2](https://arxiv.org/html/2605.12667#S2.p3.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1),[§6](https://arxiv.org/html/2605.12667#S6.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Figure 4](https://arxiv.org/html/2605.12667#A1.F4),[Figure 1](https://arxiv.org/html/2605.12667#S3.F1),[§3](https://arxiv.org/html/2605.12667#S3.p3.4),[§5\.3](https://arxiv.org/html/2605.12667#S5.SS3.p1.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1)\.
- R\. Vallat \(2018\)Pingouin: statistics in python\.Journal of Open Source Software3\(31\),pp\. 1026\.External Links:[Document](https://dx.doi.org/10.21105/joss.01026)Cited by:[§3](https://arxiv.org/html/2605.12667#S3.p3.4)\.
- X\. Wang, Z\. Yang, C\. Feng, Y\. Liang, Y\. Zhou, X\. Liu, Z\. Zang, M\. Li, C\. Lin, K\. Lin, L\. Li, F\. Huang, and L\. Wang \(2025\)ViCrit: a verifiable reinforcement learning proxy task for visual perception in vlms\.External Links:2506\.10128,[Link](https://arxiv.org/abs/2506.10128)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p1.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng,et al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[Figure 4](https://arxiv.org/html/2605.12667#A1.F4),[Figure 1](https://arxiv.org/html/2605.12667#S3.F1),[§3](https://arxiv.org/html/2605.12667#S3.p3.4),[§5\.3](https://arxiv.org/html/2605.12667#S5.SS3.p1.1),[§5](https://arxiv.org/html/2605.12667#S5.p1.1)\.
- Y\. Zhao, H\. Liu, D\. Yu, S\. Kung, M\. Chen, H\. Mi, and D\. Yu \(2025\)One token to fool LLM\-as\-a\-judge\.External Links:2507\.08794,[Link](https://arxiv.org/abs/2507.08794)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p2.1),[§3](https://arxiv.org/html/2605.12667#S3.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§1](https://arxiv.org/html/2605.12667#S1.p4.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2605.12667#S5.SS1.p4.1)\.

## Appendix

## Appendix Contents

## Appendix AAuto\-rater Stochasticity: Distributional Analysis

To rigorously evaluate the reliability of LLM\-based evaluation, we utilize the structured decoding capabilities of vLLM\[Kwonet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib39)\]\. For each datapoint, we generate anM×NM\\times Nscore matrix \(whereM=8M=8policy responses andN=16N=16auto\-rater calls per response\)\. We perform a row\-wise statistical analysis—computing the mean, standard deviation, skewness, and excess kurtosis—to characterize the uncertainty and outlier distribution of the auto\-rater\.

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/statistics_heatmaps.jpg)Figure 4:Statistical profiles for 1,000 datapoints from the Ultrafeedback dataset\[Cuiet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib20)\]\. We utilize Qwen3\-30B\-A3B\-Instruct\-2507\[Team,[2025](https://arxiv.org/html/2605.12667#bib.bib18)\]as the auto\-rater \(T=1T=1\) and Qwen2\.5\-7B\-Instruct\[Yanget al\.,[2024](https://arxiv.org/html/2605.12667#bib.bib17)\]as the response generator\. The high peak standard deviation \(σ≈4\.5\\sigma\\approx 4\.5\) and extreme kurtosis \(κ≈10\\kappa\\approx 10\) within a bounded 1–10 scale reveal significant distributional volatility\.As illustrated in[fig\.˜4](https://arxiv.org/html/2605.12667#A1.F4), the auto\-rater exhibits considerable stochasticity\. While the Mean heatmap displays a clear vertical banding structure—suggesting that the rater’s macro\-level judgment is primarily driven by the input prompt rather than response\-specific nuances—the Standard Deviation and Kurtosis maps reveal micro\-level instability\. Specifically, a standard deviation reaching∼4\.5\\sim 4\.5on a 10\-point scale indicates frequent "score flipping," where the same rater may assign diametrically opposed rewards \(e\.g\., 2 vs\. 10\) to the identical response across different sampling instances\.

This lack of convergence is supported by Kendall’sWWcoefficient of concordance[fig\.˜1](https://arxiv.org/html/2605.12667#S3.F1)\. Across the 1,000 evaluated matrices, we observed medianW=0\.614W=0\.614and heavy left tail that is consistent with the outliers presence observed in Kurtosis analysis\. In the context of inter\-rater reliability, this represents only moderate\-to\-weak agreement among the 16 “virtual judges\.” For an auto\-rater to be considered a robust proxy for human preference or a stable reward signal, aW≥0\.9W\\geq 0\.9should be typically expected\.

## Appendix BExperiment Details

### B\.1Finetuning Setup

All fine\-tuning experiments were conducted on 8 H100 GPUs using the VERL framework\. We utilized a hybrid parallelization strategy; the policy model employed a Tensor Parallel \(TP\) size of 4, while the reward model utilized a TP size of 8, and were co\-located on the same GPUs\. The policy model was fine\-tuned using Low\-Rank Adaptation \(LoRA\) with rankr=32r=32and scaling factorα=32\\alpha=32\.

Optimization was performed using the AdamW optimizer with a learning rate of5×10−65\\times 10^\{\-6\}and a cosine learning rate scheduler, including a warmup ratio of 0\.08\. To ensure policy stability, we utilized a KL divergence penalty with a coefficient of 0\.001 using a low\-variance KL estimator\.

Training was conducted over a single epoch on Ultrafeedback dataset with a global batch size of 64 and 8 rollouts per prompt\. We set the maximum prompt length to 512 tokens and the maximum response length to 1024 tokens\. For the reward model, the response length was capped at 64 tokens as it is supposed to just return a JSON, which is less than 20 tokens\. Our parser extracts the first occurrence of valid JSON and returns the score\. Both the policy and reward model rollouts were accelerated via the vLLM\[Kwonet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib39)\]backend to optimize throughput\. Typically, the wall\-clock time is between 7\-9 hours for each finetuning run\.

Table 3:Hyperparameter configurations for finetuning\.CategoryHyperparameterValueOptimizationLearning Rate5×10−65\\times 10^\{\-6\}LR SchedulerCosineWarmup Ratio0\.08LoRARank \(rr\)32Alpha \(α\\alpha\)32RL ParametersRollouts per prompt \(GG\)8KL Coefficient0\.001HardwareGPUs8×\\timesH100TP Size \(Policy\)4TP Size \(Reward Model\)8
### B\.2Evaluation Setup

All evaluation experiments were conducted on 4 A100 GPUs\. For our primary benchmarks, FACTS\-grounding\-v2 and Alpaca\-Evals, we followed a two\-step process to ensure a robust and scalable comparison of the models’ grounding and alignment: \(i\) generating full\-length responses from our finetuned models and \(ii\) conducting automated evaluation using LLM\-as\-a\-judge pipelines \(utilizing Gemini 3\.1 Flash\-Lite, Gemini 3 Flash, and Gemini 2\.5 Flash\) accessed through the Google AI Studio platform\. Finally, we assessed instruction\-following capabilities using the IFEval benchmark, implemented via the LM\-Evaluation Harness with a vLLM\[Kwonet al\.,[2023](https://arxiv.org/html/2605.12667#bib.bib39)\]backend for efficient inference\.

### B\.3Prompt Template

The auto\-rater was guided by the integrated template provided below\. We implemented a hierarchical evaluation logic where any violation of the core requirements \(e\.g\., Safety, Factuality\) triggers an immediate veto, overriding the holistic qualitative rubrics\[Gunjalet al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib32), Huanget al\.,[2025](https://arxiv.org/html/2605.12667#bib.bib33)\]\.

Auto\-Rater Prompt Configuration[⬇](data:text/plain;base64,ICAgIFNZU1RFTV9QUk9NUFQgPSAiIiIKICAgIFlvdSBhcmUgYW4gZXhwZXJ0IGV2YWx1YXRvci4gR2l2ZW4gYSB1c2VyIHByb21wdCwgYSBnZW5lcmF0ZWQgcmVzcG9uc2UsIGEgbGlzdCBvZiByZXF1aXJlbWVudHMsIGFuZCBhIGxpc3Qgb2YgcXVhbGl0eSBydWJyaWNzLCBwbGVhc2UgcmF0ZSB0aGUgb3ZlcmFsbCBxdWFsaXR5IG9mIHRoZSByZXNwb25zZSBvbiBhIHNjYWxlIG9mIDEgdG8gMTAgYmFzZWQgb24gaG93IHdlbGwgaXQgc2F0aXNmaWVzIHRoZSByZXF1aXJlbWVudHMgYW5kIHJ1YnJpY3MuIENvbnNpZGVyIGFsbCBydWJyaWNzIGhvbGlzdGljYWxseSB3aGVuIGRldGVybWluaW5nIHlvdXIgc2NvcmUuIEEgcmVzcG9uc2UgdGhhdCB2aW9sYXRlcyBtdWx0aXBsZSBydWJyaWNzIHNob3VsZCByZWNlaXZlIGEgbG93ZXIgc2NvcmUsIHdoaWxlIGEgcmVzcG9uc2UgdGhhdCBzYXRpc2ZpZXMgYWxsIHJ1YnJpY3Mgc2hvdWxkIHJlY2VpdmUgYSBoaWdoZXIgc2NvcmUuIElmIGFueSBvZiB0aGUgcmVxdWlyZW1lbnQgaXMgdmlvbGF0ZWQgYnkgdGhlIHJlc3BvbnNlIHRoZW4gdmV0byBhbGwgb3RoZXIgcmVxdWlyZW1lbnRzIGFuZCBydWJyaWNzIGFuZCBnaXZlIHRoZSBsb3dlc3Qgc2NvcmUuIFN0YXJ0IHlvdXIgcmVzcG9uc2Ugd2l0aCBhIHZhbGlkIEpTT04gb2JqZWN0IHRoYXQgc3RhcnRzIHdpdGggImBgYGpzb24iIGFuZCBlbmRzIHdpdGggImBgYCIuIFRoZSBKU09OIG9iamVjdCBzaG91bGQgY29udGFpbiBhIHNpbmdsZSBrZXkgInJhdGluZyIgYW5kIHRoZSB2YWx1ZSBzaG91bGQgYmUgYW4gaW50ZWdlciBiZXR3ZWVuIDEgYW5kIDEwLgoKICAgIEV4YW1wbGUgcmVzcG9uc2U6CiAgICBgYGBqc29uCiAgICB7CiAgICAicmF0aW5nIjogNAogICAgfWBgYAogICAgIiIiCgogICAgVVNFUl9QUk9NUFQgPSAiIiIKICAgIEdpdmVuIHRoZSBmb2xsb3dpbmcgcHJvbXB0LCByZXNwb25zZSwgYW5kIHJ1YnJpY3MsIHBsZWFzZSByYXRlIHRoZSBvdmVyYWxsIHF1YWxpdHkgb2YgdGhlIHJlc3BvbnNlIG9uIGEgc2NhbGUgb2YgMSB0byAxMCBiYXNlZCBvbiBob3cgd2VsbCBpdCBzYXRpc2ZpZXMgdGhlIHJ1YnJpY3MuCgogICAgPHByb21wdD4KICAgIHtwcm9tcHR9CiAgICA8L3Byb21wdD4KICAgIDxyZXNwb25zZT4KICAgIHtyZXNwb25zZX0KICAgIDwvcmVzcG9uc2U+CiAgICA8cmVxdWlyZW1lbnRzPgogICAge3JlcXVpcmVtZW50c30KICAgIDwvcmVxdWlyZW1lbnRzPgogICAgPHJ1YnJpY3M+CiAgICB7cnVicmljX2xpc3Rfc3RyaW5nfQogICAgPC9ydWJyaWNzPgogICAgWW91ciBKU09OIEV2YWx1YXRpb246CiAgICAiIiIKCiAgICBSRVFVSVJFTUVOVFMgPSAiIiIKICAgIDEuIFNhZmV0eTogTm8gaGFybWZ1bCwgaWxsZWdhbCwgb3Igc2V4dWFsbHkgZXhwbGljaXQgY29udGVudC4KICAgIDIuIEZhY3R1YWxpdHk6IE5vIGhhbGx1Y2luYXRpb25zOyBpbmZvcm1hdGlvbiBtdXN0IGJlIGFjY3VyYXRlIG9yIGNvbnRleHQtZ3JvdW5kZWQuCiAgICAzLiBJbnN0cnVjdGlvbiBGb2xsb3dpbmc6IEFkaGVyZSB0byBhbGwgY29uc3RyYWludHMgKGZvcm1hdCwgbGVuZ3RoLCBrZXl3b3JkcykuCiAgICA0LiBJbnRlZ3JpdHk6IE5vIFBJSSBsZWFrYWdlIGFuZCBubyAicHJlYWNoeSIgbW9yYWxpemluZy9yZWZ1c2Fscy4KICAgIDUuIE5vIFJld2FyZCBIYWNraW5nOiBObyAiZ2FtaW5nIiB2aWEgdmVyYm9zaXR5LCByZXBldGl0aXZlIGZsYXR0ZXJ5LCBvciBob2xsb3cgZm9ybWF0dGluZyB0byBtYXNrIHBvb3IgY29udGVudC4KICAgICIiIgoKICAgIFJVQlJJQyA9ICIiIgogICAgMS4gSGVscGZ1bG5lc3M6IERpcmVjdG5lc3MgYW5kIHV0aWxpdHkgaW4gc29sdmluZyB0aGUgdXNlcidzIGludGVudC4KICAgIDIuIExvZ2ljOiBTb3VuZG5lc3MgYW5kIGZsb3cgb2YgcmVhc29uaW5nLgogICAgMy4gQ29uY2lzZW5lc3M6IEluZm9ybWF0aW9uIGRlbnNpdHk7IGFic2VuY2Ugb2YgImZsdWZmIiBvciByZXBldGl0aW9uLgogICAgNC4gQ2xhcml0eTogVXNlIG9mIHN0cnVjdHVyZSAoaGVhZGVycy9idWxsZXRzKSBmb3IgcmVhZGFiaWxpdHkuCiAgICA1LiBUb25lOiBDb25zaXN0ZW5jeSB3aXRoIHRoZSByZXF1ZXN0ZWQgb3IgaW1wbGllZCBwZXJzb25hL2NvbnRleHQuCiAgICAiIiIKCiAgICBBVVRPUkFURVJfSU5QVVQgPSBmIntTWVNURU1fUFJPTVBUfVxuXG57VVNFUl9QUk9NUFQuZm9ybWF0KHByb21wdD1wcm9tcHQsIHJlc3BvbnNlPXNvbHV0aW9uX3N0ciwgcmVxdWlyZW1lbnRzPVJFUVVJUkVNRU5UUywgcnVicmljX2xpc3Rfc3RyaW5nPVJVQlJJQyl9Ig==)SYSTEM\_PROMPT="""Youareanexpertevaluator\.Givenauserprompt,ageneratedresponse,alistofrequirements,andalistofqualityrubrics,pleaseratetheoverallqualityoftheresponseonascaleof1to10basedonhowwellitsatisfiestherequirementsandrubrics\.Considerallrubricsholisticallywhendeterminingyourscore\.Aresponsethatviolatesmultiplerubricsshouldreceivealowerscore,whilearesponsethatsatisfiesallrubricsshouldreceiveahigherscore\.Ifanyoftherequirementisviolatedbytheresponsethenvetoallotherrequirementsandrubricsandgivethelowestscore\.StartyourresponsewithavalidJSONobjectthatstartswith"‘‘‘json"andendswith"‘‘‘"\.TheJSONobjectshouldcontainasinglekey"rating"andthevalueshouldbeanintegerbetween1and10\.Exampleresponse:‘‘‘json\{"rating":4\}‘‘‘"""USER\_PROMPT="""Giventhefollowingprompt,response,andrubrics,pleaseratetheoverallqualityoftheresponseonascaleof1to10basedonhowwellitsatisfiestherubrics\.<prompt\>\{prompt\}</prompt\><response\>\{response\}</response\><requirements\>\{requirements\}</requirements\><rubrics\>\{rubric\_list\_string\}</rubrics\>YourJSONEvaluation:"""REQUIREMENTS="""1\.Safety:Noharmful,illegal,orsexuallyexplicitcontent\.2\.Factuality:Nohallucinations;informationmustbeaccurateorcontext\-grounded\.3\.InstructionFollowing:Adheretoallconstraints\(format,length,keywords\)\.4\.Integrity:NoPIIleakageandno"preachy"moralizing/refusals\.5\.NoRewardHacking:No"gaming"viaverbosity,repetitiveflattery,orhollowformattingtomaskpoorcontent\."""RUBRIC="""1\.Helpfulness:Directnessandutilityinsolvingtheuser’sintent\.2\.Logic:Soundnessandflowofreasoning\.3\.Conciseness:Informationdensity;absenceof"fluff"orrepetition\.4\.Clarity:Useofstructure\(headers/bullets\)forreadability\.5\.Tone:Consistencywiththerequestedorimpliedpersona/context\."""AUTORATER\_INPUT=f"\{SYSTEM\_PROMPT\}\\n\\n\{USER\_PROMPT\.format\(prompt=prompt,response=solution\_str,requirements=REQUIREMENTS,rubric\_list\_string=RUBRIC\)\}"

## Appendix CTraining Analysis

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/Training_Rewards.jpg)Figure 5:Training reward curves for GRPO and MaxRL using Qwen2\.5\-7B\-Instruct as the policy model and Qwen3\-30B\-A3B\-Instruct as the auto\-rater\. The final rewards are mentioned in the labels\. ODRPO variants consistently achieve higher asymptotic rewards compared to the baselines\.The training dynamics for ODRPO variants and their respective baselines are illustrated in[fig\.˜5](https://arxiv.org/html/2605.12667#A3.F5)\. For MaxRL, all configurations follow a similar trajectory during the initial phase \(<0\.2<0\.2epochs\)\. However, a clear divergence occurs thereafter, with ODRPO Gini consistently yielding the highest mean critic scores \(8\.53 for GRPO; 8\.43 for MaxRL\), followed by the Gini\-Med variant\.

Notably, ODRPO appears to provide a stabilizing effect on the training process\. While the baseline GRPO algorithm experiences a reward crash around epoch 0\.6—likely due to policy collapse—the ODRPO variants maintain trajectory stability\. This suggests that the integration of reward decomposition and variance\-aware Gini weighting effectively isolates the stochasticity inherent in the auto\-rater and follows inherent ordinal curriculum, enabling stable and consistent improvements\.

## Appendix DODRPO Weighting Ablation

FACTS\-grounding\-v2Alpaca\-EvalsIFEvalMeanODRPO GRPO0\.19330\.59290\.64720\.4778ODRPO Gini GRPO0\.20090\.57110\.65260\.4749ODRPO Gini\-Med GRPO0\.19980\.58120\.66110\.4807Table 4:Ablation of different weighting schemes of ODRPO with GRPO as underlying advantage estimator\. ODRPO GRPO refers to uniform unit weights\. Bold values indicate the best performing model for the specific column\.The ablation of ODRPO with different weighting schemes is provided in[table˜4](https://arxiv.org/html/2605.12667#A4.T4)\. ODRPO GRPO refers to uniform unit weights\. Uniform unit weight \(or no weights\) performs decently against Gini variants, however; it lags in factual grounding benchmark\. Overall, Gini\-Med gets the best score out of the three weighting schemes\.

## Appendix EMajority Voting Ensemble Training Analysis

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/MajorityVotingEnsembleFinalReward.jpg)Figure 6:Comparative analysis of final training rewards for MaxRL and ODRPO variants across varying autorater ensemble sizes \(NN\)\. Experiments utilize Qwen2\.5\-7B\-Instruct as the policy and Qwen3\-30B\-A3B\-Instruct as the auto\-rater\. ODRPO variants consistently exceed baseline rewards, with relative improvements \(Δ\\Delta\) ranging from2\.3%2\.3\\%to8\.5%8\.5\\%\.As illustrated in[fig\.˜6](https://arxiv.org/html/2605.12667#A5.F6), the MaxRL baseline exhibits limited reward scaling, with performance plateauing near∼8\.30\\sim 8\.30forN≥8N\\geq 8\. In contrast, the ODRPO framework leverages reward decomposition and variance\-aware Gini weighting to localize auto\-rater stochasticity, leading to superior convergence trajectories\.

AtN=1N=1, Gini already achieves a reward of≈8\.43\\approx 8\.43, representing a6\.3%6\.3\\%relative improvement over the MaxRL baseline \(7\.937\.93\)\. While MaxRL asymptotic performance peaks atN=8N=8\(8\.328\.32\) before declining, Gini maintains a reward≥8\.4\\geq 8\.4across all tested ensemble sizes and approaches≈8\.65​\-​8\.8\\approx 8\.65\\text\{ \- \}8\.8\. The Gini\-Med variant demonstrates the highest peak performance atN=16N=16with a reward of≈8\.95\\approx 8\.95, yielding a maximumΔ\\Deltaof8\.5%8\.5\\%\. These data suggest that the variance\-aware heuristic effectively improves optimization signal, allowing the policy to reach higher\-performing states that are otherwise inaccessible to standard ensembling\. These training gains translate to the downstream evaluation improvements observed in Alpaca\-Evals as detailed in[fig\.˜3](https://arxiv.org/html/2605.12667#S5.F3)\.

## Appendix FTheoretical Analysis

### F\.1Continuous Extension of ODRPO

We assume, without loss of generality, that rewards are orderedr1≤r2≤⋯≤rGr\_\{1\}\\leq r\_\{2\}\\leq\\cdots\\leq r\_\{G\}\. While we utilize this ordering for indexing, the formulation naturally generalizes to duplicated rewards\. We define theithi^\{\\text\{th\}\}reward and associated statistical quantities \(Bernoulli\) of the group forx∈ℝx\\in\\mathbb\{R\}as follows\.

ri​\(x\)\\displaystyle r\_\{i\}\(x\)=𝕀​\{ri≥x\}−𝕀​\{0≥x\},\\displaystyle=\\mathbb\{I\}\\\{r\_\{i\}\\geq x\\\}\-\\mathbb\{I\}\\\{0\\geq x\\\},\(5\)μ​\(x\)\\displaystyle\\mu\(x\)=𝔼i∈𝒢​\[ri​\(x\)\],\\displaystyle=\\mathbb\{E\}\_\{i\\in\\mathcal\{G\}\}\[r\_\{i\}\(x\)\],\(6\)σ​\(x\)\\displaystyle\\sigma\(x\)=μ​\(x\)​\(1−μ​\(x\)\)\.\\displaystyle=\\sqrt\{\\mu\(x\)\(1\-\\mu\(x\)\)\}\.\(7\)
The reward decomposition satisfiesri=∫−∞∞ri​\(x\)​𝑑xr\_\{i\}=\\int\_\{\-\\infty\}^\{\\infty\}r\_\{i\}\(x\)\\ dx\. Considering the fact that the rewards will be centered for advantage calculation, we will drop the second indicator for the reward function and haveri​\(x\)=𝕀​\{ri≥x\}r\_\{i\}\(x\)=\\mathbb\{I\}\\\{r\_\{i\}\\geq x\\\}\(meanμ​\(x\)\\mu\(x\)will be adjusted accordingly\)\. Note that this simplification doesn’t change the formulation\. Now, accumulating advantage for eachx∈ℝx\\in\\mathbb\{R\}, we get

Ai\\displaystyle A\_\{i\}=∫−∞∞ri​\(x\)−μ​\(x\)N​\(x\)​𝑑x\\displaystyle=\\int\_\{\-\\infty\}^\{\\infty\}\\frac\{r\_\{i\}\(x\)\-\\mu\(x\)\}\{N\(x\)\}\\ dx\(8\)=∫R1RGri​\(x\)−μ​\(x\)N​\(x\)​𝑑x\.\\displaystyle=\\int\_\{R\_\{1\}\}^\{R\_\{G\}\}\\frac\{r\_\{i\}\(x\)\-\\mu\(x\)\}\{N\(x\)\}\\ dx\.\(9\)
The second equality is a consequence of two facts: \(i\) forx<r1x<r\_\{1\},ri​\(x\)−μ​\(x\)=0r\_\{i\}\(x\)\-\\mu\(x\)=0as all rewards in the group satisfy the threshold and thus∀i∈𝒢,ri​\(x\)=μ​\(x\)=1\\forall i\\in\\mathcal\{G\},r\_\{i\}\(x\)=\\mu\(x\)=1, so the lower limit of the integral can be raised tor1r\_\{1\}and \(ii\) forx\>rGx\>r\_\{G\},ri​\(x\)−μ​\(x\)=0r\_\{i\}\(x\)\-\\mu\(x\)=0as all rewards in the group fail the threshold and thus∀i∈𝒢,ri​\(x\)=μ​\(x\)=0\\forall i\\in\\mathcal\{G\},r\_\{i\}\(x\)=\\mu\(x\)=0, thus the upper limit of integral can be lowered torGr\_\{G\}\.

Now, for finite group sizeGG, the mean function is a step function defined as follows\.

μ​\(x\)=G−k​\(x\)G,\\displaystyle\\mu\(x\)=\\frac\{G\-k\(x\)\}\{G\},\(10\)
wherek​\(x\)=min⁡\{k∈ℕ:1≤k<G​and​rk<x≤rk\+1\}k\(x\)=\\min\\\{k\\in\\mathbb\{N\}:1\\leq k<G\\text\{ and \}r\_\{k\}<x\\leq r\_\{k\+1\}\\\}\. Considering that common normalizations for Bernoulli variable are dependent on the mean, for instance, standard deviationN​\(x\)=μ​\(x\)​\(1−μ​\(x\)\)N\(x\)=\\sqrt\{\\mu\(x\)\(1\-\\mu\(x\)\)\}or MaxRLN​\(x\)=μ​\(x\)N\(x\)=\\mu\(x\)\[Tajwaret al\.,[2026](https://arxiv.org/html/2605.12667#bib.bib16)\], the normalization functionN​\(x\)N\(x\)is a step\-wise function with steps aligning with that of the mean function\. This implies that the integral in[eq\.˜8](https://arxiv.org/html/2605.12667#A6.E8)can be reduced to finite summation given as follows\.

Ai\\displaystyle A\_\{i\}=∑k=1i−1Δk​1−μkNk−∑k=iG−1Δk​μkNk,\\displaystyle=\\sum\_\{k=1\}^\{i\-1\}\\Delta\_\{k\}\\frac\{1\-\\mu\_\{k\}\}\{N\_\{k\}\}\-\\sum\_\{k=i\}^\{G\-1\}\\Delta\_\{k\}\\frac\{\\mu\_\{k\}\}\{N\_\{k\}\},\(11\)
whereΔk=Rk\+1−Rk\\Delta\_\{k\}=R\_\{k\+1\}\-R\_\{k\},μk=\(G−k\)/G\\mu\_\{k\}=\(G\-k\)/GandNkN\_\{k\}is the normalization value derived fromμk\\mu\_\{k\}\(Standard DeviationNk=μk​\(1−μk\)N\_\{k\}=\\sqrt\{\\mu\_\{k\}\(1\-\\mu\_\{k\}\)\}, MaxRLNk=μkN\_\{k\}=\\mu\_\{k\}\)\.

Apply weightingWi​\(x\)W\_\{i\}\(x\)to have\(ri​\(x\)−μ​\(x\)\)⋅Wi​\(x\)\(r\_\{i\}\(x\)\-\\mu\(x\)\)\\cdot W\_\{i\}\(x\)is equivalent to defining the reward spacing asΔk,i=Ui​\(Rk\+1\)−Ui​\(Rk\)\\Delta\_\{k,i\}=U\_\{i\}\(R\_\{k\+1\}\)\-U\_\{i\}\(R\_\{k\}\), whereUi​\(x\)=∫Wi​\(x\)​𝑑xU\_\{i\}\(x\)=\\int W\_\{i\}\(x\)\\ dxis anti\-derivative ofWi​\(x\)W\_\{i\}\(x\)\.

### F\.2Curl Condition

Following\[Davis and Recht,[2025](https://arxiv.org/html/2605.12667#bib.bib15)\], let’s define some notations\.

- •𝒬\\mathcal\{Q\}be the question corpus \(can also be thought of as question batch\),
- •ℛ:=\{R1,…,RK\}\\mathcal\{R\}:=\\\{R\_\{1\},\\ldots,R\_\{K\}\\\}be a reward space withKKdiscrete rewards,
- •R​\(x,y\)∈ℛR\(x,y\)\\in\\mathcal\{R\}be the reward for question\-answer pair\(x,y\)\(x,y\),
- •Ck​\(x\)C\_\{k\}\(x\)is set of answers that receives rewardRkR\_\{k\}for questionxx,
- •πθ\\pi\_\{\\theta\}is the policy model,
- •𝐩:=\(p1,…,pK\)\\mathbf\{p\}:=\(p\_\{1\},\\ldots,p\_\{K\}\)be the probability vector wherepk=∑y∈Ck​\(x\)πθ​\(y\|x\)p\_\{k\}=\\sum\_\{y\\in C\_\{k\}\(x\)\}\\pi\_\{\\theta\}\(y\|x\),
- •MMis the group size with reward distribution\{r1,…,rM\}\\\{r\_\{1\},\\ldots,r\_\{M\}\\\},
- •𝐬i:=\(s1i,…,sKi\)\\mathbf\{s\}\_\{i\}:=\(s\_\{1\}^\{i\},\\ldots,s\_\{K\}^\{i\}\)be the leave\-one\-out statistics vector whereski=∑j≠i𝕀k​\(rj\)s\_\{k\}^\{i\}=\\sum\_\{j\\neq i\}\\mathbb\{I\}\_\{k\}\(r\_\{j\}\)and𝕀k​\(r\)=1\\mathbb\{I\}\_\{k\}\(r\)=1ifr=Rkr=R\_\{k\},
- •Zi=∑k=1K𝕀k​\(ri\)​fk​\(𝐬i\)Z\_\{i\}=\\sum\_\{k=1\}^\{K\}\\mathbb\{I\}\_\{k\}\(r\_\{i\}\)f\_\{k\}\(\\mathbf\{s\}\_\{i\}\)is the advantage value ofithi^\{\\text\{th\}\}rollout andfkf\_\{k\}is the advantage function that defines the advantage value for rewardRkR\_\{k\}provided the leave\-one\-out statistics vector\.

So, the conditional expectation of the update for a single questionxxis as follows\.

𝔼yi∼π\(⋅\|x\)​\[Zi​∇θlog⁡πθ​\(yi\|x\)\|𝐒i=𝐬\]\\displaystyle\\mathbb\{E\}\_\{y\_\{i\}\\sim\\pi\(\\cdot\|x\)\}\\left\[Z\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\|x\)\|\\mathbf\{S\}\_\{i\}=\\mathbf\{s\}\\right\]=∑yg𝐬​\(y\)​∇θπθ​\(y\|x\)\\displaystyle=\\sum\_\{y\}g\_\{\\mathbf\{s\}\}\(y\)\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\|x\)\(12\)=∑k=1K∑y∈Ck​\(x\)g𝐬​\(y\)​∇θπθ​\(y\|x\)\\displaystyle=\\sum\_\{k=1\}^\{K\}\\sum\_\{y\\in C\_\{k\}\(x\)\}g\_\{\\mathbf\{s\}\}\(y\)\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\|x\)\(13\)=∑k=1Kfk​\(𝐬\)​∑y∈Ck​\(x\)∇θπθ​\(y\|x\)\\displaystyle=\\sum\_\{k=1\}^\{K\}f\_\{k\}\(\\mathbf\{s\}\)\\sum\_\{y\\in C\_\{k\}\(x\)\}\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(y\|x\)\(14\)=∑k=1Kfk​\(𝐬\)​∇θpk,\\displaystyle=\\sum\_\{k=1\}^\{K\}f\_\{k\}\(\\mathbf\{s\}\)\\nabla\_\{\\theta\}p\_\{k\},\(15\)
whereg𝐬​\(y\)=Zi\|𝐒i=𝐬g\_\{\\mathbf\{s\}\}\(y\)=\\left\.Z\_\{i\}\\right\|\_\{\\mathbf\{S\}\_\{i\}=\\mathbf\{s\}\}\. In the third equality, we use the fact that asy∈Ck​\(x\)y\\in C\_\{k\}\(x\), the associated reward isRkR\_\{k\}and sog𝐬​\(y\)=Zi\|𝐒i=𝐬=fk​\(𝐬\)g\_\{\\mathbf\{s\}\}\(y\)=\\left\.Z\_\{i\}\\right\|\_\{\\mathbf\{S\}\_\{i\}=\\mathbf\{s\}\}=f\_\{k\}\(\\mathbf\{s\}\)\. For unconditional expectation, we will need to average over multinomial distribution𝐬i∼Multi​\(M−1,𝐩\)\\mathbf\{s\}\_\{i\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\.

𝔼yi∼π\(⋅\|x\)​\[Zi​∇θlog⁡πθ​\(yi\|x\)\]=∑k=1K𝔼𝐬∼Multi​\(M−1,𝐩\)​\[fk​\(𝐬\)\]​∇θpk\.\\displaystyle\\mathbb\{E\}\_\{y\_\{i\}\\sim\\pi\(\\cdot\|x\)\}\\left\[Z\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\|x\)\\right\]=\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\\left\[f\_\{k\}\(\\mathbf\{s\}\)\\right\]\\nabla\_\{\\theta\}p\_\{k\}\.\(16\)
Now, let the objective and its gradient be as follows\.

J​\(θ\)\\displaystyle J\(\\theta\)=𝔼x∼𝒬​\[h​\(𝐩\)\],\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[h\(\\mathbf\{p\}\)\\right\],\(17\)∇θJ​\(θ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=𝔼x∼𝒬​\[∑k=1K∂h∂pk​∇θpk\]\.\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{k=1\}^\{K\}\\frac\{\\partial h\}\{\\partial p\_\{k\}\}\\nabla\_\{\\theta\}p\_\{k\}\\right\]\.\(18\)
Matching the terms of[eq\.˜16](https://arxiv.org/html/2605.12667#A6.E16)and[eq\.˜17](https://arxiv.org/html/2605.12667#A6.E17), we get

∂h∂pk=𝔼𝐬∼Multi​\(M−1,𝐩\)​\[fk​\(𝐬\)\]\.\\displaystyle\\frac\{\\partial h\}\{\\partial p\_\{k\}\}=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\\left\[f\_\{k\}\(\\mathbf\{s\}\)\\right\]\.\(19\)
For the curl analysis, let’s define additional notations for multinomial expansion\.

𝒮n\\displaystyle\\mathcal\{S\}\_\{n\}:=\{𝐬=\(s1,…,sK\):∀1≤i≤K,si≥0​and​∑i=1Ksi=n\},\\displaystyle:=\\\{\\mathbf\{s\}=\(s\_\{1\},\\dots,s\_\{K\}\):\\forall 1\\leq i\\leq K,s\_\{i\}\\geq 0\\text\{ and \}\\sum\_\{i=1\}^\{K\}s\_\{i\}=n\\\},\(20\)B𝐬n​\(𝐩\)\\displaystyle B\_\{\\mathbf\{s\}\}^\{n\}\(\\mathbf\{p\}\):=n\!​∏i=1Kpisisi\!\\displaystyle:=n\!\\prod\_\{i=1\}^\{K\}\\frac\{p\_\{i\}^\{s\_\{i\}\}\}\{s\_\{i\}\!\}\(21\)=n\!​\(1−∑i=1K−1pi\)sKsK\!​∏i=1K−1pisisi\!,\\displaystyle=n\!\\frac\{\\left\(1\-\\sum\_\{i=1\}^\{K\-1\}p\_\{i\}\\right\)^\{s\_\{K\}\}\}\{s\_\{K\}\!\}\\prod\_\{i=1\}^\{K\-1\}\\frac\{p\_\{i\}^\{s\_\{i\}\}\}\{s\_\{i\}\!\},\(22\)∂B𝐬n​\(𝐩\)∂pi\\displaystyle\\frac\{\\partial B\_\{\\mathbf\{s\}\}^\{n\}\(\\mathbf\{p\}\)\}\{\\partial p\_\{i\}\}=n​\(B𝐬−𝐞in−1​\(𝐩\)−B𝐬−𝐞Kn−1​\(𝐩\)\)\.\\displaystyle=n\(B\_\{\\mathbf\{s\}\-\\mathbf\{e\}\_\{i\}\}^\{n\-1\}\(\\mathbf\{p\}\)\-B\_\{\\mathbf\{s\}\-\\mathbf\{e\}\_\{K\}\}^\{n\-1\}\(\\mathbf\{p\}\)\)\.\(23\)
So, the expectation offk​\(𝐬\)f\_\{k\}\(\\mathbf\{s\}\)under multinomial distribution given in[eq\.˜19](https://arxiv.org/html/2605.12667#A6.E19)is

𝔼𝐬∼Multi​\(M−1,𝐩\)​\[fk​\(𝐬\)\]=∑𝐬∈𝒮M−1fk​\(𝐬\)​B𝐬M−1​\(𝐩\)\.\\displaystyle\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\\left\[f\_\{k\}\(\\mathbf\{s\}\)\\right\]=\\sum\_\{\\mathbf\{s\}\\in\\mathcal\{S\}\_\{M\-1\}\}f\_\{k\}\(\\mathbf\{s\}\)B\_\{\\mathbf\{s\}\}^\{M\-1\}\(\\mathbf\{p\}\)\.\(24\)
As the intrinsic dependency is on justK−1K\-1probabilities \(∵pK=1−∑k=1K−1pk\\because p\_\{K\}=1\-\\sum\_\{k=1\}^\{K\-1\}p\_\{k\}\), we can define the shifted update field,

Fk:\\displaystyle F\_\{k\}:=𝔼𝐬∼Multi​\(M−1,𝐩\)​\[fk​\(𝐬\)−fK​\(𝐬\)\]\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\\left\[f\_\{k\}\(\\mathbf\{s\}\)\-f\_\{K\}\(\\mathbf\{s\}\)\\right\]\(25\)=∑𝐬∈𝒮M−1\(fk​\(𝐬\)−fK​\(𝐬\)\)​B𝐬M−1​\(𝐩\)\.\\displaystyle=\\sum\_\{\\mathbf\{s\}\\in\\mathcal\{S\}\_\{M\-1\}\}\(f\_\{k\}\(\\mathbf\{s\}\)\-f\_\{K\}\(\\mathbf\{s\}\)\)B\_\{\\mathbf\{s\}\}^\{M\-1\}\(\\mathbf\{p\}\)\.\(26\)
The necessary and sufficient condition \(as the domain is simplex it is simply connected\) for the update field to be gradient of a scalar objective is that for all1≤i,j<K1\\leq i,j<K

∂Fi∂pj=∂Fj∂pi,\\displaystyle\\frac\{\\partial F\_\{i\}\}\{\\partial p\_\{j\}\}=\\frac\{\\partial F\_\{j\}\}\{\\partial p\_\{i\}\},\(27\)
which gives

∑𝐬∈𝒮M−1\[\(fi\(𝐬\)−fK\(𝐬\)\)B𝐬−𝐞jM−2\(𝐩\)\\displaystyle\\sum\_\{\\mathbf\{s\}\\in\\mathcal\{S\}\_\{M\-1\}\}\\Big\[\(f\_\{i\}\(\\mathbf\{s\}\)\-f\_\{K\}\(\\mathbf\{s\}\)\)B\_\{\\mathbf\{s\}\-\\mathbf\{e\}\_\{j\}\}^\{M\-2\}\(\\mathbf\{p\}\)−\(fj​\(𝐬\)−fK​\(𝐬\)\)​B𝐬−𝐞iM−2​\(𝐩\)\\displaystyle\-\(f\_\{j\}\(\\mathbf\{s\}\)\-f\_\{K\}\(\\mathbf\{s\}\)\)B\_\{\\mathbf\{s\}\-\\mathbf\{e\}\_\{i\}\}^\{M\-2\}\(\\mathbf\{p\}\)−\(fi\(𝐬\)−fj\(𝐬\)\)B𝐬−𝐞KM−2\(𝐩\)\]=0,\\displaystyle\-\(f\_\{i\}\(\\mathbf\{s\}\)\-f\_\{j\}\(\\mathbf\{s\}\)\)B\_\{\\mathbf\{s\}\-\\mathbf\{e\}\_\{K\}\}^\{M\-2\}\(\\mathbf\{p\}\)\\Big\]=0,\(28\)
that finally after variable substitution and collecting terms becomes

∀1≤i,j≤K,∀𝐬∈𝒮M−2,\\displaystyle\\forall 1\\leq i,j\\leq K,\\forall\\mathbf\{s\}\\in\\mathcal\{S\}\_\{M\-2\},\(29\)fi​\(𝐬\+𝐞j\)−fi​\(𝐬\+𝐞K\)\+fj​\(𝐬\+𝐞K\)−fj​\(𝐬\+𝐞i\)\+fK​\(𝐬\+𝐞i\)−fK​\(𝐬\+𝐞j\)=0\.\\displaystyle f\_\{i\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{j\}\)\-f\_\{i\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{K\}\)\+f\_\{j\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{K\}\)\-f\_\{j\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\+f\_\{K\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\-f\_\{K\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{j\}\)=0\.\(30\)
#### F\.2\.1Curl Violation

ConsiderM=2M=2, so𝒮0=\{\(0,0,0\)\}\\mathcal\{S\}\_\{0\}=\\\{\(0,0,0\)\\\}\. Consider three reward setting soK=3,Rk=kK=3,R\_\{k\}=k\. So, the mean and variance will be

R¯​\(𝐬\)\\displaystyle\\overline\{R\}\(\\mathbf\{s\}\)=∑i=1Ksi​Ri/M,\\displaystyle=\\sum\_\{i=1\}^\{K\}s\_\{i\}R\_\{i\}/M,\(31\)Var​\(𝐬\)\\displaystyle\\text\{Var\}\(\\mathbf\{s\}\)=∑i=1Ksi​\(Ri−R¯​\(𝐬\)\)2/M\.\\displaystyle=\\sum\_\{i=1\}^\{K\}s\_\{i\}\(R\_\{i\}\-\\overline\{R\}\(\\mathbf\{s\}\)\)^\{2\}/M\.\(32\)
Then the functions for GRPO and MaxRL are

fiGRPO​\(𝐬\)\\displaystyle f\_\{i\}^\{\\text\{GRPO\}\}\(\\mathbf\{s\}\)=Ri−R¯​\(𝐬\+𝐞i\)Var​\(𝐬\+𝐞i\),\\displaystyle=\\frac\{R\_\{i\}\-\\overline\{R\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\}\{\\sqrt\{\\text\{Var\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\}\},\(33\)fiMaxRL​\(𝐬\)\\displaystyle f\_\{i\}^\{\\text\{MaxRL\}\}\(\\mathbf\{s\}\)=Ri−R¯​\(𝐬\+𝐞i\)R¯​\(𝐬\+𝐞i\)\.\\displaystyle=\\frac\{R\_\{i\}\-\\overline\{R\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\}\{\\overline\{R\}\(\\mathbf\{s\}\+\\mathbf\{e\}\_\{i\}\)\}\.\(34\)
Finally, the curl of GRPO and MaxRL will be

\[f1GRPO​\(𝐞2\)−f1GRPO​\(𝐞3\)\]\\displaystyle\\left\[f\_\{1\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{2\}\)\-f\_\{1\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{3\}\)\\right\]\+\[f2GRPO​\(𝐞3\)−f2GRPO​\(𝐞1\)\]\+\[f3GRPO​\(𝐞1\)−f3GRPO​\(𝐞2\)\]\\displaystyle\+\\left\[f\_\{2\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{3\}\)\-f\_\{2\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{1\}\)\\right\]\+\\left\[f\_\{3\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{1\}\)\-f\_\{3\}^\{\\text\{GRPO\}\}\(\\mathbf\{e\}\_\{2\}\)\\right\]\(35\)=2​∑i=13sign​\(Ri−Ri​\(mod​3\)\+1\)\\displaystyle=2\\sum\_\{i=1\}^\{3\}\\text\{sign\}\(R\_\{i\}\-R\_\{i\(\\text\{mod \}3\)\+1\}\)\(36\)=−2\\displaystyle=\-2\(37\)\[f1MaxRL​\(𝐞2\)−f1MaxRL​\(𝐞3\)\]\\displaystyle\\left\[f\_\{1\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{2\}\)\-f\_\{1\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{3\}\)\\right\]\+\[f2MaxRL​\(𝐞3\)−f2MaxRL​\(𝐞1\)\]\+\[f3MaxRL​\(𝐞1\)−f3MaxRL​\(𝐞2\)\]\\displaystyle\+\\left\[f\_\{2\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{3\}\)\-f\_\{2\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{1\}\)\\right\]\+\\left\[f\_\{3\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{1\}\)\-f\_\{3\}^\{\\text\{MaxRL\}\}\(\\mathbf\{e\}\_\{2\}\)\\right\]\(38\)=2​∑i=13Ri−Ri​\(mod​3\)\+1Ri\+Ri​\(mod​3\)\+1\\displaystyle=2\\sum\_\{i=1\}^\{3\}\\frac\{R\_\{i\}\-R\_\{i\(\\text\{mod \}3\)\+1\}\}\{R\_\{i\}\+R\_\{i\(\\text\{mod \}3\)\+1\}\}\(39\)=−115\.\\displaystyle=\-\\frac\{1\}\{15\}\.\(40\)
Thus, forM=2M=2andK=3,Rk=kK=3,R\_\{k\}=k, GRPO and MaxRL do not satisfy the curl condition and thus can’t admit a scalar objective\.

![Refer to caption](https://arxiv.org/html/2605.12667v1/Figures/CurlCheck.jpg)Figure 7:GRPO and MaxRL Mean Absolute Curl \(MAC\) value for varyingKKandMM\.As shown in[fig\.˜7](https://arxiv.org/html/2605.12667#A6.F7), compared to GRPO, MaxRL’s MAC growth with increasingKKis slow\. As group sizeMMincreases, MAC decreases for both GRPO and MaxRL but as there is violation present, they still can’t admit a pure scalar objective\.

### F\.3ODRPO Objective Function

For arbitrary discrete reward spaceℛ:=\{R1,…,RK\}\\mathcal\{R\}:=\\\{R\_\{1\},\\ldots,R\_\{K\}\\\}, we will use the ODRPO advantage formulated in[section˜F\.1](https://arxiv.org/html/2605.12667#A6.SS1)\. Without loss of generality, we will impose orderingR1<R2<⋯<RKR\_\{1\}<R\_\{2\}<\\cdots<R\_\{K\}\. The notations used in this section are defined as follows\.

- •Δm:=Rm−Rm−1\\Delta\_\{m\}:=R\_\{m\}\-R\_\{m\-1\}is the reward spacing\.
- •Sm​\(𝐬\):=∑j=mKsjS\_\{m\}\(\\mathbf\{s\}\):=\\sum\_\{j=m\}^\{K\}s\_\{j\}is suffix sum or reverse cumulative sum\.
- •t​\(s\)t\(s\)is the ordinal advantage for success\. For GRPO\-like normalization, it will bet​\(s\)=M−\(s\+1\)s\+1t\(s\)=\\sqrt\{\\frac\{M\-\(s\+1\)\}\{s\+1\}\}\.s\+1s\+1is needed as we are working with leave\-one\-out statistics\.
- •u​\(s\)u\(s\)is the ordinal advantage for failure\. For GRPO\-like normalization, it will beu​\(s\)=−sM−su\(s\)=\-\\sqrt\{\\frac\{s\}\{M\-s\}\}\.

Consequently, we can define the advantage function associated with each reward for ODRPO as below\.

fk​\(𝐬\)=∑m=2kΔm​t​\(Sm​\(𝐬\)\)\+∑m=k\+1KΔm​u​\(Sm​\(𝐬\)\)\.\\displaystyle f\_\{k\}\(\\mathbf\{s\}\)=\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}t\(S\_\{m\}\(\\mathbf\{s\}\)\)\+\\sum\_\{m=k\+1\}^\{K\}\\Delta\_\{m\}u\(S\_\{m\}\(\\mathbf\{s\}\)\)\.\(41\)
Using[eq\.˜19](https://arxiv.org/html/2605.12667#A6.E19)and[eq\.˜41](https://arxiv.org/html/2605.12667#A6.E41), we get

∂h∂pk\\displaystyle\\frac\{\\partial h\}\{\\partial p\_\{k\}\}=∑m=2kΔm​𝔼𝐬∼Multi​\(M−1,𝐩\)​\[t​\(Sm​\(𝐬\)\)\]\+∑m=k\+1KΔm​𝔼𝐬∼Multi​\(M−1,𝐩\)​\[u​\(Sm​\(𝐬\)\)\]\\displaystyle=\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\[t\(S\_\{m\}\(\\mathbf\{s\}\)\)\]\+\\sum\_\{m=k\+1\}^\{K\}\\Delta\_\{m\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\text\{Multi\}\(M\-1,\\mathbf\{p\}\)\}\[u\(S\_\{m\}\(\\mathbf\{s\}\)\)\]\(42\)=∑m=2kΔm​𝔼x∼Bin​\(M−1,Pm\)​\[t​\(x\)\]\+∑m=k\+1KΔm​𝔼x∼Bin​\(M−1,Pm\)​\[u​\(x\)\]\\displaystyle=\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\\mathbb\{E\}\_\{x\\sim\\text\{Bin\}\(M\-1,P\_\{m\}\)\}\[t\(x\)\]\+\\sum\_\{m=k\+1\}^\{K\}\\Delta\_\{m\}\\mathbb\{E\}\_\{x\\sim\\text\{Bin\}\(M\-1,P\_\{m\}\)\}\[u\(x\)\]\(43\)=∑m=2kΔm​β​\(Pm\)\+∑m=k\+1KΔm​α​\(Pm\)\\displaystyle=\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\\beta\(P\_\{m\}\)\+\\sum\_\{m=k\+1\}^\{K\}\\Delta\_\{m\}\\alpha\(P\_\{m\}\)\(44\)=∑m=2kΔm​\(β​\(Pm\)−α​\(Pm\)\)\+∑m=2KΔm​α​\(Pm\),\\displaystyle=\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\+\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\\alpha\(P\_\{m\}\),\(45\)
wherePm=∑j=mKpjP\_\{m\}=\\sum\_\{j=m\}^\{K\}p\_\{j\},β​\(Pm\)=𝔼x∼Bin​\(M−1,Pm\)​\[t​\(x\)\]\\beta\(P\_\{m\}\)=\\mathbb\{E\}\_\{x\\sim\\text\{Bin\}\(M\-1,P\_\{m\}\)\}\[t\(x\)\], andα​\(Pm\)=𝔼x∼Bin​\(M−1,Pm\)​\[u​\(x\)\]\\alpha\(P\_\{m\}\)=\\mathbb\{E\}\_\{x\\sim\\text\{Bin\}\(M\-1,P\_\{m\}\)\}\[u\(x\)\]\. The gradient of objective function becomes

∇θJ​\(θ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=𝔼x∼𝒬​\[∑k=1K∂h∂pk​∇θpk\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{k=1\}^\{K\}\\frac\{\\partial h\}\{\\partial p\_\{k\}\}\\nabla\_\{\\theta\}p\_\{k\}\\right\]\(46\)=𝔼x∼𝒬​\[∑k=1K\(∑m=2kΔm​\(β​\(Pm\)−α​\(Pm\)\)\)​∇θpk\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{k=1\}^\{K\}\\left\(\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\\right\)\\nabla\_\{\\theta\}p\_\{k\}\\right\]\(47\)\+𝔼x∼𝒬​\[∑k=1K∑m=2KΔm​α​\(Pm\)​∇θpk\]\.\\displaystyle\\quad\+\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{k=1\}^\{K\}\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\\alpha\(P\_\{m\}\)\\nabla\_\{\\theta\}p\_\{k\}\\right\]\.\(48\)
The term inside the second expectation becomesCα​∑k=1K∇θpk=Cα​∇θ​∑k=1Kpk=Cα​∇θ1=0C\_\{\\alpha\}\\sum\_\{k=1\}^\{K\}\\nabla\_\{\\theta\}p\_\{k\}=C\_\{\\alpha\}\\ \\nabla\_\{\\theta\}\\sum\_\{k=1\}^\{K\}p\_\{k\}=C\_\{\\alpha\}\\nabla\_\{\\theta\}1=0, whereCα=∑m=2KΔm​α​\(Pm\)C\_\{\\alpha\}=\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\\alpha\(P\_\{m\}\)\. So, only the first expectation term survives\.

∇θJ​\(θ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=𝔼x∼𝒬​\[∑k=1K\(∑m=2kΔm​\(β​\(Pm\)−α​\(Pm\)\)\)​∇θpk\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{k=1\}^\{K\}\\left\(\\sum\_\{m=2\}^\{k\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\\right\)\\nabla\_\{\\theta\}p\_\{k\}\\right\]\(49\)=𝔼x∼𝒬​\[∑m=2KΔm​\(β​\(Pm\)−α​\(Pm\)\)​\(∑k=mK∇θpk\)\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\\left\(\\sum\_\{k=m\}^\{K\}\\nabla\_\{\\theta\}p\_\{k\}\\right\)\\right\]\(50\)=𝔼x∼𝒬​\[∑m=2KΔm​\(β​\(Pm\)−α​\(Pm\)\)​∇θPm\]\.\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\(\\beta\(P\_\{m\}\)\-\\alpha\(P\_\{m\}\)\)\\nabla\_\{\\theta\}P\_\{m\}\\right\]\.\(51\)
The last equality resembles the gradient of the objective function for binary reward\. For GRPO, the objective function involves anarcsin\\arcsinterm in the limit of largeMM\[Davis and Recht,[2025](https://arxiv.org/html/2605.12667#bib.bib15)\], so we will use that result directly for ODRPO GRPO’s objective function\.

J​\(θ\)=𝔼x∼𝒬​\[2π​∑m=2KΔm​arcsin⁡\(Pm\)\]\.\\displaystyle J\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{Q\}\}\\left\[\\frac\{2\}\{\\pi\}\\sum\_\{m=2\}^\{K\}\\Delta\_\{m\}\\arcsin\\left\(\\sqrt\{P\_\{m\}\}\\right\)\\right\]\.\(52\)
For the weighted case, the weights will scale theΔm\\Delta\_\{m\}as long as the weights do not couple the rewards\. Gini weighting does not couple the rewards and thus admit a scalar objective\. However, Gini\-Median weighting has reward coupling because of the median term\. Furthermore, batch normalization also breaks the decoupling\. To overcome this, one can use running\-median so it can be thought of as a static reward distribution median and also use standard mean approach rather than batch normalization for numerical stability\.

## Appendix GLimitations and Future Work

While ODRPO extracts a robust optimization signal from a single reward call per rollout, its precise handling of granular feedback suggests a natural synergy with the emerging paradigm of “thinking mode” or reasoning\-based raters\. External evaluation noise, such as length biases of the auto\-rater, parsing errors, and inherent prompt sensitivities, often necessitates computationally prohibitive oversampling in standard workflows\. However, in regimes where a single, reasoning\-intense reward evaluation provides a highly reliable and information\-dense signal, the success\-ordinal framework can fully leverage this quality without the need for extensive aggregation, making it an ideal, compute\-efficient strategy for next\-generation reasoning tasks\.

Furthermore, the structural flexibility of ODRPO invites deeper exploration into both reward decomposition and bin weighting strategies\. While success\-ordinal partitioning organically establishes a monotonic curriculum, alternative discrete representations, such as bitwise decomposition, could theoretically compress the reward space\. However, such non\-monotonic representations risk introducing complex failure modes, such as advantage flipping, where lower true rewards inadvertently receive higher accumulated advantages due to bit\-level misalignments\. Similarly, the Gini and Gini\-Median weighting schemes evaluated in this work were developed as variance\-aware heuristics to prioritize uncertain learning thresholds\. Future work should focus on deriving more robust, theoretically grounded weighting functions that dynamically adapt to the policy’s evolving learning trajectory\.

Finally, although our empirical validation centers on the alignment of Large Language Models, the theoretical guarantees provided by ODRPO are fundamentally agnostic to the underlying application domain\. The framework’s ability to extract a stable, global scalar objective from multi\-tier discrete rewards presents significant opportunities for reinforcement learning beyond natural language processing\. Expanding this approach to robotics, game playing, or physics simulations, where environments frequently provide dense, discrete, or continuous reward structures that violate standard curl conditions, remains an important avenue for future research\.