Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

# Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
Source: [https://arxiv.org/html/2607.29185](https://arxiv.org/html/2607.29185)
Sanwoo Lee1,2,3\\internship, Clive Bai3, Hsiu\-Yuan Huang1,2,3\\internship, Kun Liang1,2,3\\internship, Weijie Liu3, Yunfang Wu1,2\\corresponding

###### Abstract

Reward models \(RMs\) are central to aligning large language models with human preferences via reinforcement learning\. Although traditional scalar RMs enable efficient and probabilistic reward modeling, they rely on superficial cues that fail to generalize to complex or out\-of\-distribution \(OOD\) tasks\. Conversely, generative RMs leverage extensive reasoning to improve robustness on challenging tasks, but their natural language\-based scores lack the numerical flexibility and probabilistic interpretability that scalar RMs offer\. While recent approaches combine both paradigms through off\-policy multi\-task learning, such parallel optimization does not guarantee that generated reasoning traces actively align with or benefit downstream scalar reward prediction\. To address this mismatch, we proposeLatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize the likelihood of downstream scalar rewards\. Through on\-policy optimization of the latent reasoning space end\-to\-end, LatentRM tightly couples deep reasoning\-based evaluation with precise scoring\. Extensive validations on in\-distribution and OOD datasets and RLHF show that LatentRM outperforms scalar, generative, and hybrid RMs on preference modeling and policy alignment across tasks ranging from open\-ended conversation to complex reasoning\.

## Introduction

Reward models \(RMs\) have shown great promise in aligning large language models \(LLMs\) with human preference\(Ouyanget al\.[2022](https://arxiv.org/html/2607.29185#bib.bib9); Baiet al\.[2022](https://arxiv.org/html/2607.29185#bib.bib1)\)\. Central to the reinforcement learning from human feedback \(RLHF\) process, RMs offer an efficient and scalable proxy for human preference, improving LLMs on a wide range of tasks\(Wuet al\.[2023](https://arxiv.org/html/2607.29185#bib.bib3)\)beyond domains where rule\-based verifiable rewards are readily available\. Therefore, a key challenge in developing RMs lies in establishing an accurate and faithful proxy robust to potential distribution shifts in prompt and response\.

![Refer to caption](https://arxiv.org/html/2607.29185v1/x1.png)Figure 1:Rank correlation coefficient against groundtruth ranking of scalar RM and generative RM, measured over 1 epoch training on the training set\. Both models are initialized from Qwen3\-4B\-Instruct\-2507\(Team[2025](https://arxiv.org/html/2607.29185#bib.bib32)\), where acalar RM is trained to predict under the same prefix context as generative RM\.RMs are typically designed as scalar RM which places a linear head on top of an LLM backbone and learns from preference rankings, to assign a scalar score for a pair of prompt and response\(Ziegleret al\.[2019](https://arxiv.org/html/2607.29185#bib.bib2); Ouyanget al\.[2022](https://arxiv.org/html/2607.29185#bib.bib9)\)\. However, scalar RMs struggle with distribution shift\(Gaoet al\.[2023](https://arxiv.org/html/2607.29185#bib.bib4)\)and reasoning\-intensive prompts\(Honget al\.[2025](https://arxiv.org/html/2607.29185#bib.bib5)\)as they often overfit to superficial patterns in the training data\. Later works have mitigated these issues with generative RMs that follows LLM\-as\-Judge paradigm and invokes deep reasoning traces to substantiate the final rankings or scores\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6); Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\), yet the scores in natural language lack the flexibility and probabilistic interpretation that scalar RMs naturally offer\. As Figure[1](https://arxiv.org/html/2607.29185#Sx1.F1)illustrates, when optimizing a generative RM for Kendall’sτ\\tauagainst groundtruth ranking, a scalar RM which shares exactly same prefix context as generative RM for scoring achieves markedly superior performance\. This shows that the flexibility of scalar RM translates to more accurate preference modeling when backed up by the reasoning as additional context, suggesting the need for an effective interplay between these two paradigms\.

Despite recent efforts to enhance scalar RMs with reasoning, seamlessly combining the two remains challenging\. An initial work explored appending critiques generated by external teacher LLMs to the input of a scalar RM\(Yeet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib11)\), demonstrating consistent gains across teacher models\. Subsequent works resolved the reliance on teachers at inference time by jointly training the scalar RM on critique distillation and preference learning\(Ankneret al\.[2024](https://arxiv.org/html/2607.29185#bib.bib13)\), or removed external teachers altogether by leveraging self\-filtered critiques for distillation instead\(Yuet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib12)\)\. However, fine\-tuning from static critiques, even the self\-generated and filtered ones, incurs a mismatch between training and inference, since the reasoning the model generates becomes increasingly off\-policy as training progresses\. Most importantly, the enhancement of reasoning quality relies on ad\-hoc heuristics, such as calling strong LLMs and self\-refinement, leaving their contribution to the downstream reward prediction largely unclear\. This underscores the need for a task\-aligned objective that refines the reasoning trace explicitly for assisting the scalar RM\.

In this work, we proposeLatentRM, a scalar reward model with a generator that learns to shape its reasoning for aunifiedgoal of maximizing the likelihood of its downstream scalar reward\. Specifically, we treat the reasoning as a discrete latent variablezzbridging the inputxxand preference labelyyin a conditional generative model\. In this framework, we optimize for the data likelihoodlog⁡p​\(y\|z\)\\log p\(y\|z\)under Plackett\-Luce model\(Luce[1959](https://arxiv.org/html/2607.29185#bib.bib18)\), by maximizing its lower bound with a simple and affordable choice of variational posterior\. We transform this objective into an end\-to\-end joint training procedure where both scalar RM and generator are updated on the on\-policy critiques from the generator\. Through this design, LatentRM achieves a tight interplay between generator and scalar RM for the ultimate goal of accurate preference learning\. Experiments on in\-distribution \(ID\) and out\-of\-distribution \(OOD\) datasets, as well as RLHF alignment, demonstrate that LatentRM outperforms scalar RM, generative RM and a multi\-task hybrid RM where the generator has strong yet separate learning objective from the scalar RM\.

In essence, we articulate our contributions as follows:

- •We present LatentRM, an end\-to\-end framework that treats reasoning as a latent variable to directly optimize downstream scalar RM’s preference modeling, which replaces manually designed reward\.
- •We devise an on\-policy training procedure derived from the data likelihood ELBO, co\-optimizing the generator and scalar RM that removes training\-inference mismatch\.
- •LatentRM showcases superior ID and OOD performance over scalar, generative and hybrid RMs on diverse benchmarks without introducing hyperparameter overhead\.

## Related Work

#### Reward Models

Reward models have been central in aligning LLMs during on\-policy training\(Ouyanget al\.[2022](https://arxiv.org/html/2607.29185#bib.bib9)\)as well as test\-time generation\(Fricket al\.[2025](https://arxiv.org/html/2607.29185#bib.bib36)\)\. Traditional RMs are typically scalar RMs trained with Bradley\-Terry loss\(Bradley and Terry[1952](https://arxiv.org/html/2607.29185#bib.bib17)\)over pairwise preference data\(Christianoet al\.[2017](https://arxiv.org/html/2607.29185#bib.bib15); Baiet al\.[2022](https://arxiv.org/html/2607.29185#bib.bib1)\)\. Scalar RMs had often suffered from distribution shifts\(Miaoet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib10)\)and consume massive datasets to generalize\(Yeet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib11)\), which prompted active research on fine\-tuning generative RMs to elicit reasoning prior to rewards in textual format\(Kimet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib16); Liet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib14)\)\. This approach has been extended to leveraging reinforcement learning to scale reasoning towards deeper depth\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6); Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\)\. Concurrently, hybrid architectures generate a rationale and pass it to a scalar RM for reward prediction, fine\-tuning both on preference labels alongside rationales distillted from stronger LLMs\(Yeet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib11); Ankneret al\.[2024](https://arxiv.org/html/2607.29185#bib.bib13)\)or self\-refinement process\(Yuet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib12)\)\. Nevertheless, few is known about the link between refining reasoning and optimizing downstream scalar RM, which we aim to address by a unified objective\.

#### Latent Variable View of Reasoning

Chain\-of\-thought reasoning serves as a key driver for improving LLM performance on downstream applications\. Early foundational work byLeiet al\.\([2016](https://arxiv.org/html/2607.29185#bib.bib19)\)trained an extractive rationale generator by treating rationales as latent variables that maximize a downstream discriminator’s predictive accuracy\. Building on this perspective, STaR\(Zelikmanet al\.[2022](https://arxiv.org/html/2607.29185#bib.bib20)\)iteratively fine\-tuned models on self\-generated rationales filtered by binary verifiers, in effect approximating a policy gradient objective over latent reasoning traces\. This approach was later extended using Markov\-chain Monte Carlo sampling to generate rationales that align with both the generator and the ground\-truth answer\(Phanet al\.[2023](https://arxiv.org/html/2607.29185#bib.bib21)\)\. Most recently,Tanget al\.\([2025](https://arxiv.org/html/2607.29185#bib.bib22)\)introduced a fully on\-policy RL method to learn latent reasoning for semi\-verifiable tasks, such as mathematical proofs\. However, leveraging latent variable reasoning frameworks for reward modeling remains largely unexplored\. To our knowledge, LatentRM is the first work to formalize reasoning as a latent variable in a generator–scalar RM architecture\.

## Method

### Problem Statement

We consider a general reward modeling problem where the inputxxconsists of a promptxpx\_\{p\}andkkcandidate responses\{xri\}i=1k\\\{x\_\{r\_\{i\}\}\\\}\_\{i=1\}^\{k\}\. For the sake of generality, we letk≥1k\\geq 1and posit thatkkmay vary across prompts, which subsumes canonical pointwise and pairwise reward modeling setup as special cases\. The reward model then predictskkreal\-valued rewardss=\(s1,s2,…,sk\)∈ℝks=\(s\_\{1\},s\_\{2\},\.\.\.,s\_\{k\}\)\\in\\mathbb\{R\}^\{k\}for the responses\. The goal is to train a reward model whose predicted ranking induced byssis aligned with the human preference rankingyyover the responses\.

### Model Architecture

In this paper, we introduce a generator\-discriminator architecture for the reward model\. In particular, the generator LLM parametrized byθ∈ℝdθ\\theta\\in\\mathbb\{R\}^\{d\_\{\\theta\}\}samples a chain\-of\-thought reasoningz∼pθ​\(z\|x\)z\\sim p\_\{\\theta\}\(z\|x\)givenxx\. Then the discriminator scalar RM parametrized byφ∈ℝdφ\\varphi\\in\\mathbb\{R\}^\{d\_\{\\varphi\}\}outputs rewardss∈ℝks\\in\\mathbb\{R\}^\{k\}givenxxandzz\. This design allows the discriminator to leverage ample evidence supporting the final decision in the reasoning, while ensuring flexible scoring that generative RMs lack\. While the scalar RM may be of different architecture from the generator, we simply initialize the scalar RM as a copy of the generator with the language modeling head replaced by a scalar head randomly parametrized byw∈ℝhw\\in\\mathbb\{R\}^\{h\}\. To induce a distribution over the the listwise rankingspφp\_\{\\varphi\}, we use Plackett\-Luce model\(Luce[1959](https://arxiv.org/html/2607.29185#bib.bib18)\)\. For a strict total orderingy=\(y1≻y2≻…≻yk\)y=\(y\_\{1\}\\succ y\_\{2\}\\succ\.\.\.\\succ y\_\{k\}\), its likelihood is defined as:

pφ​\(y\|x,z\)=∏i=1kexp⁡\(syi\)∑j=ikexp⁡\(syj\)p\_\{\\varphi\}\(y\|x,z\)=\\prod\_\{i=1\}^\{k\}\\frac\{\\exp\(s\_\{y\_\{i\}\}\)\}\{\\sum\_\{j=i\}^\{k\}\\exp\(s\_\{y\_\{j\}\}\)\}\(1\)which reduces to the standard Bradley\-Terry model whenk=2k=2\. When ties are present inyy, we sum over the likelihoods of all strict total orderingsπ=\(π1≻π2≻…≻πk\)∈Ω​\(y\)\\pi=\(\\pi\_\{1\}\\succ\\pi\_\{2\}\\succ\.\.\.\\succ\\pi\_\{k\}\)\\in\\Omega\(y\)consistent with the weak orderingyycontaining ties\.

![Refer to caption](https://arxiv.org/html/2607.29185v1/x2.png)Figure 2:A condensed prompt template for our listwise generator\. Full template is available at Figure[6](https://arxiv.org/html/2607.29185#A1.F6)in Appendix\.To implement the generator\-discriminator inference flow, we organize our prompt template as in Figure[2](https://arxiv.org/html/2607.29185#Sx3.F2)that wraps the prompt and candidate responses as inputxxfor reasoning generation\. The reasoning trace complies with the format requirements of the template, yielding a rubric\-based verbal evaluation followed by an overall score for each response in format<score\_i\>x\.x</score\_i\>\(i=1,2,…,ki=1,2,\.\.\.,k\)\. We then feed the concatenated input and reasoning trace\(x,z\)\(x,z\)into the scalar RM, and extract the last\-layer hidden states at thekktoken positions immediately preceding the generated score values, namely the last tokens in<score\_i\>\. Stacking these hidden states column\-wise yields

H=\[h1,…,hk\]∈ℝh×k\.H=\[h\_\{1\},\\ldots,h\_\{k\}\]\\in\\mathbb\{R\}^\{h\\times k\}\.\(2\)We then apply a shared scalar headw∈ℝhw\\in\\mathbb\{R\}^\{h\}to obtain the rewards ass=w⊤​Hs=w^\{\\top\}H\.

![Refer to caption](https://arxiv.org/html/2607.29185v1/x3.png)Figure 3:Graphical models illustrating differences between MultitaskRM \(baseline\) and LatentRM\. Solid line denotes generative model and dashed line denotes inference models\. Circles are random variables and squares are parameters, where shaded circles represent observed variables\.
### Learning to Reason as Latent Variables

A straightforward approach for generator\-discriminator architectures is co\-training both components in parallel using each component’s own objective\. Generators are widely trained with binary correctness reward\(Honget al\.[2025](https://arxiv.org/html/2607.29185#bib.bib5); Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\)while discriminators are predominantly trained with Bradley\-Terry loss\(Baiet al\.[2022](https://arxiv.org/html/2607.29185#bib.bib1)\), which generalizes to Kendall’sτ\\taurank correlation reward and Plackett\-Luce loss under listwise setup\. However, these objectives were originally formulated for individual components in isolation\. Combining them via naive multi\-task learning, which we denote asMultitask RM, does not guarantee optimal accuracy of downstream reward, as maximizing surrogate rewards like Kendall’sτ\\tauserves only as an imperfect proxy for maximizing the true log\-likelihoodlog⁡pφ​\(y\|x,z\)\\log p\_\{\\varphi\}\(y\|x,z\)\(Eq\.[1](https://arxiv.org/html/2607.29185#Sx3.E1)\)\.

To address this issue, we establish an end\-to\-end objective by casting CoT reasoningzzas a discrete latent variable that conditions the scalar RM \(discriminator\) for better prediction\. This is equivalent to formalizing the architecture as a conditional generative model\(Sohnet al\.[2015](https://arxiv.org/html/2607.29185#bib.bib8)\), where givenxx, the latent variablezzis drawn from the priorpθ​\(z\|x\)p\_\{\\theta\}\(z\|x\)and the outputyyis drawn from the decoder networkpφ​\(y\|x,z\)p\_\{\\varphi\}\(y\|x,z\)\. Under the generative model, our goal is to jointly train the parameters\(θ,φ\)\(\\theta,\\varphi\)to maximize the conditional log\-likelihoodlog⁡p​\(y\|x\)\\log p\(y\|x\), marginalizing over the latent reasoning:

maxθ,φ⁡log⁡p​\(y\|x\)=maxθ,φ⁡log⁡𝔼z∼pθ​\(z\|x\)​\[pφ​\(y\|x,z\)\]\\max\_\{\\theta,\\varphi\}\\log p\(y\|x\)=\\max\_\{\\theta,\\varphi\}\\log\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\(z\|x\)\}\\left\[p\_\{\\varphi\}\(y\|x,z\)\\right\]\(3\)Directly optimizing Eq\.[3](https://arxiv.org/html/2607.29185#Sx3.E3)is intractable because its gradient requires expectations over the true posteriorp​\(z\|x,y\)p\(z\|x,y\)\. The standard variational approach optimizes its evidence lower bound \(ELBO\) by introducing an inference networkqϕ​\(z\|x,y\)q\_\{\\phi\}\(z\|x,y\)as an approximation to the posterior:

log⁡p​\(y\|x\)=𝔼qϕ​\(z\|x,y\)​\[log⁡pθ​\(z\|x\)​pφ​\(y\|x,z\)qϕ​\(z\|x,y\)\]\+DK​L\(qϕ\(z\|x,y\)∥p\(z\|x,y\)\)≥𝔼qϕ​\(z\|x,y\)​\[log⁡pφ​\(y\|x,z\)\]−DK​L\(qϕ\(z\|x,y\)∥pθ\(z\|x\)\)=ℒELBO\\begin\{split\}\\log p\(y\|x\)=\\,&\\mathbb\{E\}\_\{q\_\{\\phi\}\(z\|x,y\)\}\\left\[\\log\\frac\{p\_\{\\theta\}\(z\|x\)p\_\{\\varphi\}\(y\|x,z\)\}\{q\_\{\\phi\}\(z\|x,y\)\}\\right\]\\\\ &\+D\_\{KL\}\\left\(q\_\{\\phi\}\(z\|x,y\)\\parallel p\(z\|x,y\)\\right\)\\\\ \\geq\\,&\\mathbb\{E\}\_\{q\_\{\\phi\}\(z\|x,y\)\}\\left\[\\log p\_\{\\varphi\}\(y\|x,z\)\\right\]\\\\ &\-D\_\{KL\}\\left\(q\_\{\\phi\}\(z\|x,y\)\\parallel p\_\{\\theta\}\(z\|x\)\\right\)=\\mathcal\{L\}\_\{\\text\{ELBO\}\}\\end\{split\}\(4\)However, optimization of parameterizedqϕ​\(z\|x,y\)q\_\{\\phi\}\(z\|x,y\)requires training a second LLM conditioned onyy\. This adds substantial computational cost and is conceptually counter\-intuitive, asqϕ​\(z\|x,y\)q\_\{\\phi\}\(z\|x,y\)risks generating spurious justification solely to match any given label rather than discovering authentic task logic\. Therefore, we simply choose the inference network to be shared by the prior, i\.e\.,qϕ​\(z\|x,y\)=pθ​\(z\|x\)q\_\{\\phi\}\(z\|x,y\)=p\_\{\\theta\}\(z\|x\)\. By Jensen’s inequality, this choice still recovers a valid lower bound on the true marginal likelihood\. The resulting optimization objective thus simplifies to:

maxθ,φ⁡𝔼z∼pθ​\(z\|x\)​\[log⁡pφ​\(y\|x,z\)\]\\max\_\{\\theta,\\varphi\}\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\(z\|x\)\}\\left\[\\log p\_\{\\varphi\}\(y\|x,z\)\\right\]\(5\)Figure[3](https://arxiv.org/html/2607.29185#Sx3.F3)compares graphical models for MultitaskRM and LatentRM\. LatentRM leverage solelyxxandyyto jointly superviseθ\\thetaandφ\\varphi, whereas MultitaskRM introduces an auxiliary rewardrrto trainθ\\thetain parallel withφ\\varphi\.

### Gradient of the Lower Bound and its Estimator

To switch from the objective in Eq\.[5](https://arxiv.org/html/2607.29185#Sx3.E5)to practical training updates, we first derive the gradients ofℒELBO\\mathcal\{L\}\_\{\\text\{ELBO\}\}w\.r\.t\.θ\\thetaandφ\\varphias follows:

∇φℒELBO\\displaystyle\\nabla\_\{\\varphi\}\\mathcal\{L\}\_\{\\text\{ELBO\}\}=𝔼z∼pθ​\[∇φlog⁡pφ​\(y\|x,z\)\]\\displaystyle=\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\}\\left\[\\nabla\_\{\\varphi\}\\log p\_\{\\varphi\}\(y\|x,z\)\\right\]\(6\)=𝔼z∼pθ​\[∑i=1k\(1−∑j=1iexp⁡\(syi\)∑l=jkexp⁡\(syl\)\)​∇φsyi​\(x,z\)\]\\displaystyle=\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\}\\left\[\\sum\_\{i=1\}^\{k\}\\left\(1\-\\sum\_\{j=1\}^\{i\}\\frac\{\\exp\(s\_\{y\_\{i\}\}\)\}\{\\sum\_\{l=j\}^\{k\}\\exp\(s\_\{y\_\{l\}\}\)\}\\right\)\\nabla\_\{\\varphi\}s\_\{y\_\{i\}\}\(x,z\)\\right\]∇θℒELBO\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{ELBO\}\}=𝔼z∼pθ​\[log⁡pφ​\(y\|x,z\)​∇θlog⁡pθ​\(z\|x\)\]\\displaystyle=\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\}\\left\[\\log p\_\{\\varphi\}\(y\|x,z\)\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(z\|x\)\\right\]\(7\)Eq\.[6](https://arxiv.org/html/2607.29185#Sx3.E6)and[7](https://arxiv.org/html/2607.29185#Sx3.E7)reveal LatentRM’s optimization mechanism\. For a given\(x,y\)\(x,y\), the scalar RM is updated to increase the expected log\-likelihood ofyyunder the current reasoning distribution\. Under the Plackett\-Luce model, this amounts to correcting each scoresyis\_\{y\_\{i\}\}in proportion to the scalar RM’s remaining uncertainty aboutyiy\_\{i\}’s rank, a gap that vanishes once the it ranksyiy\_\{i\}correctly without ambiguitiy in scores\. Meanwhile, the generator is updated to favor reasoning traceszzthat increase the discriminator’s log\-likelihood of observingyy\.

We present the training procedure of LatentRM in Algorithm[1](https://arxiv.org/html/2607.29185#alg1)with the empirical estimators of the gradients\. The generator samplesmmrollouts \(i\.e\., reasoning\) per instance in a batch, then the scalar RM is updated with standard supervised learning\. Since sampling the rolloutzzis not differentiable, we update the generator using REINFORCE\(Suttonet al\.[1999](https://arxiv.org/html/2607.29185#bib.bib24)\), treating the log\-likelihoodlog⁡pφ​\(y\(i\)\|x\(i\),z\(i,j\)\)\\log p\_\{\\varphi\}\(y^\{\(i\)\}\|x^\{\(i\)\},z^\{\(i,j\)\}\)as the reward signal\. To reduce the variance of this policy gradient estimator, we compute a baselineb\(i\)b^\{\(i\)\}as the average reward across themmrollouts for a given prompt, subtracting it from each rollout’s reward to obtain the advantageA\(i,j\)A^\{\(i,j\)\}\. For rollouts violating format requirements, we compute advantages solely over valid rollouts and assign invalid ones an advantage ofminj∈𝒱​\(i\)⁡A\(i,j\)−1\\min\_\{j\\in\\mathcal\{V\}\(i\)\}A^\{\(i,j\)\}\-1where𝒱​\(i\)\\mathcal\{V\}\(i\)is the set of valid rollouts\. If𝒱​\(i\)=∅\\mathcal\{V\}\(i\)=\\varnothing, every rollout receives an advantage of−1\-1\. Overall,θ\\thetaandφ\\varphiare updated jointly at every iteration with their respective gradients\. We observe such joint learning is effective and stable, not requiring warmup steps for scalar RM\.

Algorithm 1LatentRM Training1:Training set

𝒟\\mathcal\{D\}, initial parameters

θ0,φ0\\theta\_\{0\},\\varphi\_\{0\}, number of rollouts per prompt

mm, batch size

BB, learning rates

ηθ,ηφ\\eta\_\{\\theta\},\\eta\_\{\\varphi\}
2:Initialize

θ←θ0\\theta\\leftarrow\\theta\_\{0\},

φ←φ0\\varphi\\leftarrow\\varphi\_\{0\}
3:foreach training stepdo

4:Sample a mini\-batch

\{\(x\(i\),y\(i\)\)\}i=1B∼𝒟\\\{\(x^\{\(i\)\},y^\{\(i\)\}\)\\\}\_\{i=1\}^\{B\}\\sim\\mathcal\{D\}
5:for

i=1,…,Bi=1,\\dots,Bdo

6:Sample

mmrollouts

\{z\(i,j\)\}j=1m∼pθ​\(z\|x\(i\)\)\\\{z^\{\(i,j\)\}\\\}\_\{j=1\}^\{m\}\\sim p\_\{\\theta\}\(z\|x^\{\(i\)\}\)
7:Compute

R\(i,j\)←log⁡pφ​\(y\(i\)\|x\(i\),z\(i,j\)\)R^\{\(i,j\)\}\\leftarrow\\log p\_\{\\varphi\}\(y^\{\(i\)\}\|x^\{\(i\)\},z^\{\(i,j\)\}\)
8:Compute baseline

b\(i\)←1m​∑j=1mR\(i,j\)b^\{\(i\)\}\\leftarrow\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}R^\{\(i,j\)\}
9:Compute advantage

A\(i,j\)←R\(i,j\)−b\(i\)A^\{\(i,j\)\}\\leftarrow R^\{\(i,j\)\}\-b^\{\(i\)\}
10:endfor

11:

φ←φ\+ηφ⋅1B​m​∑i,j∇φlog⁡pφ​\(y\(i\)\|x\(i\),z\(i,j\)\)\\varphi\\leftarrow\\varphi\+\\eta\_\{\\varphi\}\\cdot\\frac\{1\}\{Bm\}\\sum\_\{i,j\}\\nabla\_\{\\varphi\}\\log p\_\{\\varphi\}\(y^\{\(i\)\}\|x^\{\(i\)\},z^\{\(i,j\)\}\)
12:

θ←θ\+ηθ⋅1B​m​∑i,jA\(i,j\)​∇θlog⁡pθ​\(z\(i,j\)\|x\(i\)\)\\theta\\leftarrow\\theta\+\\eta\_\{\\theta\}\\cdot\\frac\{1\}\{Bm\}\\sum\_\{i,j\}A^\{\(i,j\)\}\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(z^\{\(i,j\)\}\|x^\{\(i\)\}\)
13:endfor

14:return

θ\\theta,

φ\\varphi

## Experiments

### Experimental Setup

We validate LatentRM’s effectiveness on ID test set and several OOD benchmarks for evaluating reward models\. In addition, we conduct RLHF experiments with LatentRM and baseline RMs to explore their utility on inducing an aligned LLM\. Throughout the experiments, we use Qwen3\-4B\-Instruct111https://huggingface\.co/Qwen/Qwen3\-4B\-Instruct\-2507\(Team[2025](https://arxiv.org/html/2607.29185#bib.bib32)\)as the backbone model across LatentRM and baselines\.

Table 1:Dataset statistics\.\# Resp\.: number of responses per prompt;\# Init\./\# Test\./\# Train: sizes of the initial pool, test, and filtered training set\. Dataset names are abbreviated by their initials\.DatasetDomain\# Resp\.\# Init\.\# Test\.\# TrainUFGeneral428,000168015,120OMRMathematics2–828,000167915,121HS3STEM&Code28,0004804,320WGSafety28,0004804,320OBAdversarial28,0004804,320Total––80,000479943,201#### Datasets

To cover diverse domains, we construct our training data pool by randomly sampling preference\-annotated data from UltraFeedback\(Cuiet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib25)\)\(28K\), OpenMathReasoning\(Moshkovet al\.[2025](https://arxiv.org/html/2607.29185#bib.bib26)\)\(28K\), Helpsteer3\(Wanget al\.[2026](https://arxiv.org/html/2607.29185#bib.bib27)\)\(STEM and Code subsets, 8K\), WildGuard\(Hanet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib28)\)\(adversarial subset, 8K\), and OffsetBias\(Parket al\.[2024](https://arxiv.org/html/2607.29185#bib.bib29)\)\(8K\)\. Dataset statistics are listed in Table[1](https://arxiv.org/html/2607.29185#Sx4.T1)\.

GeneralMathSTEM & CodeSafetyAdversarialAveragesMetricMethodUltraFeedbackOMRHelpsteer3WildguardOffsetBiasMicroMacroLog\-likelihoodScalarRM\-1\.986\-1\.154\-0\.436\-0\.109\-0\.221\-1\.175\-0\.781MultitaskRM\-1\.917\-0\.901\-0\.379\-0\.142\-0\.153\-1\.053\-0\.698LatentRM\-1\.899\-0\.840\-0\.373\-0\.199\-0\.149\-1\.031\-0\.692Kendall’sτ\\tauScalarRM0\.6490\.6060\.6000\.9290\.8070\.6730\.718GenRM w/o RL0\.5920\.3980\.5060\.4960\.6760\.4970\.534GenerativeRM0\.5860\.5410\.4700\.7580\.7020\.5880\.611MultitaskRM0\.6550\.6770\.7060\.8750\.8660\.7060\.756LatentRM0\.6620\.6850\.7120\.8500\.8870\.7120\.759Table 2:Evaluation results on ID test sets, measured by log\-likelihoodpφ​\(y\|x,z\)p\_\{\\varphi\}\(y\|x,z\)and Kendall’s tau\. Log\-likelihood is not applicable for GenerativeRMs as scores are in textual format\. The best result in each column per metric is bolded\.From this initial pool, we filter 40% of the samples which are potentially noisy and harmful for generalization\. In particular, we adapt from the split\-and\-filter protocol byGaoet al\.\([2025](https://arxiv.org/html/2607.29185#bib.bib30)\), where we split the data pool into two partitions, train an ensemble of44reward models separately on each partition, and evaluate the the ensemble’s per\-sample average validation losses on the other held\-out partition\. Samples in the two partitions are merged and the ones with smallest validation losses are discarded\. In our setup, we train lightweight MLP networks as the ensemble models on top of hand\-crafted features extracted from each \(prompt, response\) pair\. We used LFTK\(Lee and Lee[2023](https://arxiv.org/html/2607.29185#bib.bib31)\)for extraction of220220\-dimensional features from prompt and resposne each\. Discarding low\-loss samples according to the ensemble encodes our intuition that those samples potentially carry spurious patterns most easily captured by the shallow features\. The statistics of datasets are shown in Table[1](https://arxiv.org/html/2607.29185#Sx4.T1)\. We split the filtered set into training and test set by a ratio of9:19:1, which are shared by LatentRM and all baselines\. No hyperparameters are tuned on the test set for all methods, while LatentRM introduces no additional hyperparameter in itself\.

#### Evaluation Benchmarks

To evaluate OOD generalization, we test on RM\-Bench\(Liuet al\.[2025a](https://arxiv.org/html/2607.29185#bib.bib35)\)and PPE Correctness\(Fricket al\.[2025](https://arxiv.org/html/2607.29185#bib.bib36)\)\. RM\-Bench comprises44K preference pairs evaluating an RM’s sensitivity to subtle semantic differences and robustness against style bias\. PPE Correctness contains2\.52\.5K challenging prompts requiring deep reasoning, with55preference pairs per prompt\. To evaluate listwise candidates, we merge these pairs into deduplicated lists per prompt, score them directly, and convert the predicted rankings back to pairwise decisions222Pointwise scalar RM scores each \(prompt, response\) pair independently for both RM\-Bench and PPE Correctness\.\. Notably, both benchmarks construct response pairs from the same generating LLM per prompt, closely mirroring real\-world RLHF setup and demonstrating high correlation with downstream RLHF performance\(Liuet al\.[2025a](https://arxiv.org/html/2607.29185#bib.bib35); Fricket al\.[2025](https://arxiv.org/html/2607.29185#bib.bib36)\)\.

#### Baselines

We evaluate LatentRM against the following baselines:

- •ScalarRMis a standard pointwise reward model optimized over listwise preferences using Plackett\-Luce loss\.
- •GenerativeRMis a listwise generative reward model optimized via RL, using Kendall’sτb\\tau\_\{b\}as the reward\.
- •MultitaskRMis a generator\-discriminator method which uses Kendall’sτb\\tau\_\{b\}reward for the generator and Plackett\-Luce loss for scalar RM on reasoning\-augmented inputs\.

#### Implementation Details

We useVERL\(Shenget al\.[2025](https://arxiv.org/html/2607.29185#bib.bib38)\)withvLLMrollout engine\(Kwonet al\.[2023](https://arxiv.org/html/2607.29185#bib.bib39)\)throughout the experiments\. We set learning rate to2×10−62\\times 10^\{\-6\}for policy model and1×10−51\\times 10^\{\-5\}for scalar reward model, maximum prompt length to1638416384\. For each prompt,88rollouts are sampled with a temperature of1\.01\.0and a maximum response length of81928192\. We set batch size to128128and train for11epoch, and adopt pure on\-policy learning without PPO\-style mini\-batching\(Schulmanet al\.[2017](https://arxiv.org/html/2607.29185#bib.bib40)\)in case of RL\. Token\-level KL penalty coefficient is set to1×10−31\\times 10^\{\-3\}for baselines as in\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6)\)\. Rewards are transformed to advantages by mean\-centering as with LatentRM\. Due to cost\-intensive online RL training, all experiments are based on a single run with a fixed random seed of4242\.

RM\-benchPPE CorrectnessMethodChatMathCodeSafeEasyNormHardAvg\.MMLUMATHGPQAMBPP\+IFAvg\.ScalarRM71\.971\.165\.094\.886\.778\.761\.675\.771\.076\.158\.459\.564\.165\.8GenRM w/o RL69\.685\.065\.590\.986\.979\.766\.777\.875\.589\.061\.756\.953\.267\.2GenerativeRM75\.090\.966\.493\.186\.281\.975\.981\.373\.085\.659\.150\.052\.364\.0MultitaskRM78\.289\.165\.993\.382\.082\.980\.181\.780\.492\.269\.361\.356\.371\.9LatentRM78\.590\.966\.895\.183\.583\.681\.382\.880\.692\.770\.061\.655\.472\.1Table 3:Evaluation results on out\-of\-distribution RM\-Bench and PPE Correctness benchmarks, measured by accuracy\. The best value in each column is bolded\.

### Findings on Preference Modeling

#### Results on In\-distribution Test Sets

From ID test set evaluation results in Table[1](https://arxiv.org/html/2607.29185#Sx4.T1), we observe that LatentRM outperforms all baselines on general, math, STEM & Code and adversarial domains, achieving the best log\-likelihood and Kendall’s tau overall\. Importantly, LatentRM wins consistent gains in log\-likelihoodlog⁡pφ​\(y\|x,z\)\\log p\_\{\\varphi\}\(y\|x,z\)over MultitaskRM except for the safety domain, verifying that the kendall’s tau reward used to train state\-of\-the\-art generative RMs\(Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\)could still be suboptimal when its reasoning serves as an intermediate variable for assisting the scalar RM’s decision\. Similarly, optimizing the generator for kendall’s tau did not lead to the best kendall’s tau, compared to LatentRM\. LatentRM induces reward signal from the generator to maximize the data likelihood under latent variable model, thus features high fidelity to the scalar RM’s goal without any heuristic reward design for the generator\.

Compared to the vanilla ScalarRM, both MultitaskRM and LatentRM brings large performance gains which are most pronounced on reasoning\-intensive domains such as math, STEM & Code, and adversarial prompts\. This suggests that incorporating deep reasoning in scalar RM could visibly mitigate its reliance on superficial features and boost the performance\. On the other hand, generativeRM markedly underperforms scalarRM in Kendall’s tau, indicating that scalarRM more effectively fits to the training data distribution\. Yet scalar RM tends to overfit the data and suffer from limited generalization on OOD datasets, which we detail below\.

#### Results on Out\-of\-distribution Datasets

The results in Table[3](https://arxiv.org/html/2607.29185#Sx4.T3)show that LatentRM maintains the best overall OOD generalization performance, achieving an average of82\.8%82\.8\\%on RM\-Bench and72\.1%72\.1\\%on PPE Correctness\. Importantly, LatentRM wins consistent gains over MultitaskRM on diverse domains and difficulty levels, verifying that our end\-to\-end learning objective yields reasoning traces that are more likely to transfer to OOD distributions\. Meanwhile, scalarRM shows lowest or near\-lowest overall performance \(75\.775\.7on RM\-Bench and65\.865\.8on PPE\), suggesting that its advantage over generative RM in the ID setting no longer holds on OOD datasets\. It corroborates evidence from prior works\(Gaoet al\.[2023](https://arxiv.org/html/2607.29185#bib.bib4); Miaoet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib10)\)that scalarRMs easily suffer from distribution shifts\. Nevertheless, LatentRM could still outperform generativeRM, implying that conditioning scalar RM on latent chain\-of\-thought reasoning notably overcomes overfitting to superficial and in\-distribution cues\.

From a closer look into each benchmark and its subdomains, LatentRM’s improved performance on RM\-Bench suggests that LatentRM is capable of discerning subtle variations in content for preference decision, and is robust to stylistic biases that LLM judges are known to suffer from\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6)\), including tendency to favor long responses or the ones that use markdown format\. For instance, LatentRM’s gain is most pronounced onHardsubset \(81\.3%81\.3\\%\) which poses the hardest challenge on distinguishing stylic biases\. We observe such robustness is uniform across Chat, Math, Code and Safety domains\. As for PPE, we see that LatentRM shows markedly better accuracy than GenerativeRM and ScalarRM on reasoning\-intensive domains such as MATH \(92\.7%92\.7\\%\), GPQA \(70\.0%70\.0\\%\), and MBPP\+ \(61\.6%61\.6\\%\)\. This suggests that when augmented with reasoning, scalarRM could be much better in preference modeling on reasoning domains than generativeRMs\.

Table 4:Comparison of performance with external baselines\. Q and L inBackbonecolumn denotes Qwen and Llama, respectively; RM\-B: RM\-Bench; PPE: PPE Correctness\. The overall accuracies are referenced from the original papers\.†\\dagger: result reproduced byLiuet al\.\([2025b](https://arxiv.org/html/2607.29185#bib.bib48)\)\.MethodBackboneRM\-BPPEScalar RMsSkywork\-v0\.2\(Liuet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib43)\)L3\.1\-8B\-it64\.762\.5SteerLM\-RM\(Wanget al\.[2024](https://arxiv.org/html/2607.29185#bib.bib47)\)L3\.1\-70B\-it72\.263\.2Generative RMsRM\-R1\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6)\)Q2\.5\-7B\-it70\.2\-RM\-R1\(Chenet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib6)\)Q2\.5\-14B\-it76\.1\-J1\(Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\)L3\.1\-8B\-it73\.452\.8J1\(Whitehouseet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib7)\)L3\.3\-70B\-it82\.770\.2PaTaRM\(Jianet al\.[2026](https://arxiv.org/html/2607.29185#bib.bib46)\)Q3\-8B78\.7\-reasoning\-driven scalar RMsCLoud†\(Ankneret al\.[2024](https://arxiv.org/html/2607.29185#bib.bib13)\)Gemma2\-27B\-62\.4LatentRMQ3\-4B\-it82\.872\.1
#### Comparison with External Baseline Models

We report comparisons against external reward models in Table[4](https://arxiv.org/html/2607.29185#Sx4.T4)\. Since they employ different data curation strategies and pre\-trained model backbones, the reported numbers are not directly comparable to our baselines and serve primarily as a reference point\. In general, we find LatentRM achieves favorable performance that matches or outperforms much larger generative and scalar models \(e\.g\., 27B–70B parameters\) across both RM\-Bench and PPE Correctness benchmarks\. Closest to our approach, for example, CLoud\(Ankneret al\.[2024](https://arxiv.org/html/2607.29185#bib.bib13)\)achieves an overall accuracy of 62\.4 on PPE\.

![Refer to caption](https://arxiv.org/html/2607.29185v1/x4.png)Figure 4:Length\-controlled \(LC\) win rate and token length of post\-RLHF policy rewarded byLatentRMagainst post\-RLHF policy rewarded by baseline RMs, evaluated using Qwen3\.7\-plus\(Qwen Team[2026](https://arxiv.org/html/2607.29185#bib.bib45)\)as judge\. We usedDuboiset al\.\([2024](https://arxiv.org/html/2607.29185#bib.bib44)\)for the judge template and LC winrate computation formula which corrects the Judge’s length bias from the winrate by using a regression model\.

### Findings on RLHF

To evaluate LatentRM and baselines in online alignment, we perform100100steps of online RLHF using GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.29185#bib.bib41)\)on Qwen3\-4B\-Instruct as policy model\. We set KL\-loss coefficient of10−210^\{\-2\}, a batch size of128128,44rollouts per sample, and maximum token length of40964096for both prompt and response\. After training policy models under different RMs, we randomly sample200200samples from each subset of the test dataset, computing winrates of the policy by LatentRM against the policy by other RMs\. The results are shown in Figure[4](https://arxiv.org/html/2607.29185#Sx4.F4)\.

Figure[4](https://arxiv.org/html/2607.29185#Sx4.F4)shows that the RLHF policy guided by LatentRM yields average length\-controlled \(LC\) winrates exceeeding 50% against the base policy \(56\.9%56\.9\\%\), and RLHF policies by scalar RM \(58\.5%58\.5\\%\), Generative RM \(51\.5%51\.5\\%\) and MultitaskRM \(52\.0%52\.0\\%\)\. These gains demonstrate that LatentRM’s offline preference modeling strengths \(Table[2](https://arxiv.org/html/2607.29185#Sx4.T2),[3](https://arxiv.org/html/2607.29185#Sx4.T3)\) effectively generalize to dynamic, non\-stationary policy shifts\. Notably, LatentRM achieves these results without verbosity exploitation, generating the shortest average outputs \(1,289 tokens\) among all RLHF baselines\. Conversely, the Scalar RM policy suffers from reward overoptimization, yielding lower win rates against LatentRM than even the base policy, highlighting the necessity of explicit reasoning traces for stable online reward guidance\.

Across the tested domains, LatentRM shows consistent benefits over the baselines on general open\-ended conversation \(Ultrafeedback\), mathematics \(OpenMathReasoning\) and STEM & Code \(HelpSteer3\) domains\. The clear advances on reasoning domains signify that LatentRM develops genuinely informative reasoning traces\. Yet on safety \(WildGuard\) or adversarial \(OffsetBias\) domain, LatentRM lags behind MultitaskRM, which partially aligns with the ID results in Table[2](https://arxiv.org/html/2607.29185#Sx4.T2)\. We hypothesize that as UltraFeedback and OpenMathReasoning make up70%70\\%of the training dataset, LatentRM devotes more optimization efforts on these representative domains at certain cost of performance on underrepresented domains\.

![Refer to caption](https://arxiv.org/html/2607.29185v1/x5.png)Figure 5:Per\-sample score gapsΔ\\Delta\(best response−\-worst response\): generator’s natural language scoreΔ\\Deltabins \(y\-axis\) vs\. downstream Scalar RMΔ\\Delta\(x\-axis\)\. Red band denotes regions where natural language scores disagree withyy\. The distributions for the boxes are aggregated over our ID test sets\.
### Analysis of Score Gap Distribution

To draw deeper insights into the internal mechanism of LatentRM compared to MultitaskRM, in Figure[5](https://arxiv.org/html/2607.29185#Sx4.F5)we visualize the distribution of per\-sample score gaps between the best response and the worst response according toyy, jointly over the generator’s natural language scores \(y\-axis\) and the downstream scalar RM’s scores \(x\-axis\)\. When the natural langauge scores align with the groundtruth ranking, the corresponding scalarRM produces large positive score gaps, yet LatentRM’s scores provides clearer distinction than MultitaskRM\. Moreover, when the natural language scores are incorrect \(the red regions in the figure\), LatentRM exhibits exceptional recovery to correct decision by the downstream scalarRM\. In contrast, MultitaskRM tends to recover much less from such cases\. This suggests that LatentRM does not simply echo the verbalized scores for its decision, but actively leverages the intermediate reasoning traces as supporting evidence\. We hypothesize that this is a positive effect of end\-to\-end training, where the generator focuses more on developing informative reasoning trace without pressure to yield correct natural language scores\.

## Conclusion

In this paper, we introduce LatentRM, a scalar reward model enhanced by a reasoner that learns to shape its reasoning trace for a unified goal of maximizing the scalarRM’s likelihood of observing groundtruth preference ranking\. By casting reasoning trace as a latent variable in a conditional generative model, we derive a simple and natural end\-to\-end procedure for training LatentRM\. Extensive validations on ID and OOD datasets as well as RLHF experiments demonstrate that LatentRM achieves overall consistent improvements in preference modeling and policy alignment over a strong multi\-task learning baseline, and much more clearly over scalar and generative reward models in isolation\.

## References

- Critique\-out\-loud reward models\.arXiv preprint arXiv:2408\.11791\.Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p3.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Comparison with External Baseline Models](https://arxiv.org/html/2607.29185#Sx4.SSx2.SSS0.Px3.p1.1),[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p1.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Learning to Reason as Latent Variables](https://arxiv.org/html/2607.29185#Sx3.SSx3.p1.3)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.External Links:ISSN 00063444, 14643510,[Link](http://www.jstor.org/stable/2334029)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- X\. Chen, G\. Li, Z\. Wang, B\. Jin, C\. Qian, Y\. Wang, H\. WANG, Y\. Zhang, D\. Zhang, T\. Zhang, H\. Tong, and H\. Ji \(2026\)RM\-r1: reward modeling as reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1ZqJ6jj75q)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Implementation Details](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px4.p1.10),[Results on Out\-of\-distribution Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx2.SSS0.Px2.p2.4),[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.7.1),[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.8.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- G\. Cui, L\. Yuan, N\. Ding, G\. Yao, B\. He, W\. Zhu, Y\. Ni, G\. Xie, R\. Xie, Y\. Lin, Z\. Liu, and M\. Sun \(2024\)ULTRAFEEDBACK: boosting language models with scaled AI feedback\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 9722–9744\.External Links:[Link](https://proceedings.mlr.press/v235/cui24f.html)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Y\. Dubois, P\. Liang, and T\. Hashimoto \(2024\)Length\-controlled alpacaeval: a simple debiasing of automatic evaluators\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=CybBmzWBX0)Cited by:[Figure 4](https://arxiv.org/html/2607.29185#Sx4.F4)\.
- E\. Frick, T\. Li, C\. Chen, W\. Chiang, A\. N\. Angelopoulos, J\. Jiao, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2025\)How to evaluate reward models for RLHF\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cbttLtO94Q)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Evaluation Benchmarks](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px2.p1.3)\.
- C\. Gao, H\. Li, L\. Liu, Z\. Xie, P\. Zhao, and zhiqiang xu \(2025\)Principled data selection for alignment: the hidden risks of difficult examples\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=qut63YypaD)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p2.3)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 10835–10866\.External Links:[Link](https://proceedings.mlr.press/v202/gao23h.html)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1),[Results on Out\-of\-distribution Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx2.SSS0.Px2.p1.4)\.
- S\. Han, K\. Rao, A\. Ettinger, L\. Jiang, B\. Y\. Lin, N\. Lambert, Y\. Choi, and N\. Dziri \(2024\)WildGuard: open one\-stop moderation tools for safety risks, jailbreaks, and refusals of llms\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 8093–8131\.External Links:[Document](https://dx.doi.org/10.52202/079017-0261),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/0f69b4b96a46f284b726fbd70f74fb3b-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p1.1)\.
- I\. Hong, C\. Yu, L\. Qiu, W\. Yan, Z\. Xu, H\. Jiang, Q\. Zhang, Q\. Lu, X\. Liu, C\. Zhang, and T\. Zhao \(2025\)Think\-rm: enabling long\-horizon reasoning in generative reward models\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 174227–174251\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/fe9f0dcf515c331b124e4f49f760f85e-Paper-Conference.pdf)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1),[Learning to Reason as Latent Variables](https://arxiv.org/html/2607.29185#Sx3.SSx3.p1.3)\.
- A\. Jian, J\. Ruan, X\. Ma, X\. Zhang, D\. Li, W\. Zhang, K\. Zeng, and X\. Cai \(2026\)PaTaRM: bridging pairwise and pointwise signals via preference\-aware task\-adaptive reward modeling\.External Links:2510\.24235,[Link](https://arxiv.org/abs/2510.24235)Cited by:[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.11.1)\.
- S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. Seo \(2024\)Prometheus: inducing fine\-grained evaluation capability in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8euJaTveKw)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Implementation Details](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px4.p1.10)\.
- B\. W\. Lee and J\. Lee \(2023\)LFTK: handcrafted features in computational linguistics\.InProceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2023\),E\. Kochmar, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, N\. Madnani, A\. Tack, V\. Yaneva, Z\. Yuan, and T\. Zesch \(Eds\.\),Toronto, Canada,pp\. 1–19\.External Links:[Link](https://aclanthology.org/2023.bea-1.1/),[Document](https://dx.doi.org/10.18653/v1/2023.bea-1.1)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p2.3)\.
- T\. Lei, R\. Barzilay, and T\. Jaakkola \(2016\)Rationalizing neural predictions\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,J\. Su, K\. Duh, and X\. Carreras \(Eds\.\),Austin, Texas,pp\. 107–117\.External Links:[Link](https://aclanthology.org/D16-1011/),[Document](https://dx.doi.org/10.18653/v1/D16-1011)Cited by:[Latent Variable View of Reasoning](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px2.p1.1)\.
- J\. Li, S\. Sun, W\. Yuan, R\. Fan, h\. zhao, and P\. Liu \(2024\)Generative judge for evaluating alignment\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 27547–27574\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/747dc7c6566c74eb9a663bcd8d057c78-Paper-Conference.pdf)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- C\. Y\. Liu, L\. Zeng, J\. Liu, R\. Yan, J\. He, C\. Wang, S\. Yan, Y\. Liu, and Y\. Zhou \(2024\)Skywork\-reward: bag of tricks for reward modeling in llms\.External Links:2410\.18451,[Link](https://arxiv.org/abs/2410.18451)Cited by:[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.4.1)\.
- Y\. Liu, Z\. Yao, R\. Min, Y\. Cao, L\. Hou, and J\. Li \(2025a\)RM\-bench: benchmarking reward models of language models with subtlety and style\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QEHrmQPBdd)Cited by:[Evaluation Benchmarks](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px2.p1.3)\.
- Z\. Liu, P\. Wang, R\. Xu, S\. Ma, C\. Ruan, P\. Li, Y\. Liu, and Y\. Wu \(2025b\)Inference\-time scaling for generalist reward modeling\.External Links:2504\.02495,[Link](https://arxiv.org/abs/2504.02495)Cited by:[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4)\.
- R\.D\. Luce \(1959\)Individual choice behavior: a theoretical analysis\.Wiley\.External Links:LCCN 59009346,[Link](https://books.google.com/books?id=a80DAQAAIAAJ)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p4.4),[Model Architecture](https://arxiv.org/html/2607.29185#Sx3.SSx2.p1.10)\.
- Y\. Miao, S\. Zhang, L\. Ding, R\. Bao, L\. Zhang, and D\. Tao \(2024\)InfoRM: mitigating reward hacking in rlhf via information\-theoretic reward modeling\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 134387–134429\.External Links:[Document](https://dx.doi.org/10.52202/079017-4270),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/f25d75fc760aec0a6174f9f5d9da59b8-Paper-Conference.pdf)Cited by:[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Results on Out\-of\-distribution Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx2.SSS0.Px2.p1.4)\.
- I\. Moshkov, D\. Hanley, I\. Sorokin, S\. Toshniwal, C\. Henkel, B\. Schifferer, W\. Du, and I\. Gitman \(2025\)Aimo\-2 winning solution: building state\-of\-the\-art mathematical reasoning models with openmathreasoning dataset\.arXiv preprint arXiv:2504\.16891\.Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 27730–27744\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Park, S\. Jwa, R\. Meiying, D\. Kim, and S\. Choi \(2024\)OffsetBias: leveraging debiased data for tuning evaluators\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1043–1067\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.57/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.57)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p1.1)\.
- D\. Phan, M\. D\. Hoffman, D\. Dohan, S\. Douglas, T\. A\. Le, A\. Parisi, P\. Sountsov, C\. Sutton, S\. Vikram, and R\. A\. Saurous \(2023\)Training chain\-of\-thought via latent\-variable inference\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 72819–72841\.External Links:[Document](https://dx.doi.org/10.52202/075280-3184),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/e69a9560c450ca76584d9eb37e7f5ae8-Paper-Conference.pdf)Cited by:[Latent Variable View of Reasoning](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2026\)Qwen3\.7\-Plus: multimodal agent intelligence\.External Links:[Link](https://qwen.ai/blog?id=qwen3.7-plus)Cited by:[Figure 4](https://arxiv.org/html/2607.29185#Sx4.F4)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Implementation Details](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px4.p1.10)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Findings on RLHF](https://arxiv.org/html/2607.29185#Sx4.SSx3.p1.6)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[Implementation Details](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px4.p1.10)\.
- K\. Sohn, H\. Lee, and X\. Yan \(2015\)Learning structured output representation using deep conditional generative models\.InAdvances in Neural Information Processing Systems,C\. Cortes, N\. Lawrence, D\. Lee, M\. Sugiyama, and R\. Garnett \(Eds\.\),Vol\.28,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2015/file/8d55a249e6baa5c06772297520da2051-Paper.pdf)Cited by:[Learning to Reason as Latent Variables](https://arxiv.org/html/2607.29185#Sx3.SSx3.p2.8)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems,S\. Solla, T\. Leen, and K\. Müller \(Eds\.\),Vol\.12,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf)Cited by:[Gradient of the Lower Bound and its Estimator](https://arxiv.org/html/2607.29185#Sx3.SSx4.p2.12)\.
- Y\. Tang, S\. Wang, L\. Madaan, and R\. Munos \(2025\)Beyond verifiable rewards: scaling reinforcement learning in language models to unverifiable data\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 74421–74448\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/6bd67a424dc59481e1e5a5061ffc8dfe-Paper-Conference.pdf)Cited by:[Latent Variable View of Reasoning](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px2.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Figure 1](https://arxiv.org/html/2607.29185#Sx1.F1),[Experimental Setup](https://arxiv.org/html/2607.29185#Sx4.SSx1.p1.1)\.
- Z\. Wang, Y\. Dong, O\. Delalleau, J\. Zeng, G\. Shen, D\. Egert, J\. J\. Zhang, M\. N\. Sreedhar, and O\. Kuchaiev \(2024\)HelpSteer 2: open\-source dataset for training top\-performing reward models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 1474–1501\.External Links:[Document](https://dx.doi.org/10.52202/079017-0047),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/02fd91a387a6a5a5751e81b58a75af90-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.5.1)\.
- Z\. Wang, J\. Zeng, O\. Delalleau, H\. Shin, F\. Soares, A\. Bukharin, E\. Evans, Y\. Dong, and O\. Kuchaiev \(2026\)HelpSteer3\-preference: open human\-annotated preference data across diverse tasks and languages\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=lovsIkZLnI)Cited by:[Datasets](https://arxiv.org/html/2607.29185#Sx4.SSx1.SSS0.Px1.p1.1)\.
- C\. Whitehouse, T\. Wang, P\. Yu, X\. Li, J\. E\. Weston, I\. Kulikov, and S\. Saha \(2026\)J1: incentivizing thinking in LLM\-as\-a\-judge via reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dnJEHl6DI1)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1),[Learning to Reason as Latent Variables](https://arxiv.org/html/2607.29185#Sx3.SSx3.p1.3),[Results on In\-distribution Test Sets](https://arxiv.org/html/2607.29185#Sx4.SSx2.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.10.1),[Table 4](https://arxiv.org/html/2607.29185#Sx4.T4.3.9.1)\.
- Z\. Wu, Y\. Hu, W\. Shi, N\. Dziri, A\. Suhr, P\. Ammanabrolu, N\. Smith, M\. Ostendorf, and H\. Hajishirzi \(2023\)Fine\-grained human feedback gives better rewards for language model training\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 59008–59033\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/b8c90b65739ae8417e61eadb521f63d5-Paper-Conference.pdf)Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p1.1)\.
- Z\. Ye, F\. D\. Greenlee, M\. Bartolo, P\. Blunsom, J\. A\. Campos, and M\. Gallé \(2025\)Improving reward models with synthetic critiques\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4506–4520\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.254/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.254),ISBN 979\-8\-89176\-195\-7Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p3.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- Y\. Yu, Z\. Chen, A\. Zhang, L\. Tan, C\. Zhu, R\. Y\. Pang, Y\. Qian, X\. Wang, S\. Gururangan, C\. Zhang, M\. Kambadur, D\. Mahajan, and R\. Hou \(2025\)Self\-generated critiques boost reward modeling for language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 11499–11514\.External Links:[Link](https://aclanthology.org/2025.naacl-long.573/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.573),ISBN 979\-8\-89176\-189\-6Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p3.1),[Reward Models](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px1.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 15476–15488\.External Links:[Document](https://dx.doi.org/10.52202/068431-1126),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf)Cited by:[Latent Variable View of Reasoning](https://arxiv.org/html/2607.29185#Sx2.SS0.SSS0.Px2.p1.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[Introduction](https://arxiv.org/html/2607.29185#Sx1.p2.1)\.

## Appendix AFull Prompt Template of Listwise Generator

![Refer to caption](https://arxiv.org/html/2607.29185v1/x6.png)Figure 6:The complete prompt template of our listwise generator\. Brackets in blue denotes contents to be filled with conversation context or candidate responses\.We present the full template of our listwise generator in Figure[6](https://arxiv.org/html/2607.29185#A1.F6)\. This prompt template instructs an LLM generator to evaluate the candidate responses by guiding the generator through a structured three\-step process: \(1\) defining task\-specific rubrics with a priority hierarchy, \(2\) writing detailed qualitative analysis per response using pre\-defined tags, and \(3\) assigning a final numerical score from 0\.0 to 10\.0 that enforces strict total ordering across all candidates\.