重新审视奖励优化的缩放定律

# Revisiting scaling laws for reward optimization
Source: [https://arxiv.org/html/2609.38526](https://arxiv.org/html/2609.38526)
###### Abstract

Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort\-\-\-111All dashes in this manuscript are human generated\.measured by a KL\-divergence budget relative to a reference policy\. Beyond a certain budget, over\-optimization \(or reward hacking\) can arise: because we optimize against a proxy reward model \(distinct from true rewards\), performance can plateau or degrade\. Naturally, the proxy reward’s accuracy depends on how much preference data \(often in the form of pairwise comparisons\) was used to train it\. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget\. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context\. Performance roughly scales asΘ⁡\(min⁡\{log⁡\(M\),K\}\)\\Theta\(\\sqrt\{\\min\\\{\\log\(M\),K\\\}\}\), whereMMis the number of comparisons in training data andKKis the policy’s divergence budget\. We develop an information\-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure\. Informed by this, we conduct extensive empirical evaluations using a real\-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models \(0\.6B to 4B\)\. Our scaling law provides an excellent fit \(R2 from 97% to 99%\), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures \(best\-of\-NNor policy tilting\)\. Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback\.

## 1Introduction

Post\-training typically entails learning, either explicitly or implicitly, a proxy ‘reward’ model of user preferences\([Christiano et al\., 2017](https://arxiv.org/html/2609.38526#bib.bib12);[Ouyang et al\., 2022](https://arxiv.org/html/2609.38526#bib.bib4);[Rafailov et al\., 2023](https://arxiv.org/html/2609.38526#bib.bib3)\)\. Such a model remains an imperfect proxy of true user preferences, learned from limited data\. As such, it is natural that ‘over\-optimizing’ against such a proxy model may eventually lead to a decrease in the performance gained from post training\([Manheim and Garrabrant, 2018](https://arxiv.org/html/2609.38526#bib.bib8);[Skalse et al\., 2022](https://arxiv.org/html/2609.38526#bib.bib17)\)\. A seminal paper by[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)established empirical scaling laws for this phenomenon\. They showed that performance first increased like the square root of the divergence of the post\-trained policy from the pre\-trained policy, and then beyond some specific divergence budget, performance stalled or decreased\. With the advent of long running agents, one only expects such a problem to be exacerbated, potentially leading to catastrophic outcomes\([Marklund et al\., 2026](https://arxiv.org/html/2609.38526#bib.bib13)\)\. This has been the topic of much recent research on superalignment\. A consensus recipe that has emerged from the above research is that any post\-trained policy should be constrained to within some pre\-specifieddivergence budgetof a reference policy to prevent over\-optimization\. While previous literature extensively studies tradeoffs between achievable performance and divergence budget, its dependence on theamount of preference datais not well\-understood\. This raises the natural question:

*How should the policy divergence budget be set as a function of the amount of data used to train the proxy reward model?*

It is natural, in particular, that a proxy model trained on limited preference data is more susceptible to over\-optimization than one trained with a lot of such data\. As such, we seek scaling laws that capture the amount data used to train the proxy reward model\. In particular, we would like these scaling laws to inform the performance improvement this data affords us relative to a reference policy\. In addition, we wish to characterize how the aforementioned divergence budget ought to be scaled with the amount of preference data\. Answering these questions can provide important insights to clarify the fundamental limits of post\-training and guide algorithmic scaling recipes in AI alignment\.

#### Contributions\.

Our work provides a theoretical and empirical characterization of how post\-training performance jointly scales with the amount of preference data and the divergence budget\.

- •Simple theoretical derivation\.We develop an information\-theoretic model that guides our subsequent empirical evaluations\. The model yields a tight scaling law, roughly of the formΘ⁡\(min⁡\{log⁡\(M\),K\}\)\\Theta\(\\sqrt\{\\min\\\{\\log\(M\),K\\\}\}\), whereMMis the number of preference comparisons in the training dataset \(scale of feedback data\) andKKis the KL\-divergence budget \(optimization effort\)\. We give constructive procedures that near\-optimally attain this bound\.
- •Accurate empirical scaling law\.We provide extensive empirical evidence that our post\-training scaling law is highly accurate and provides a statistically better fit than alternative specifications, focusing on the novel predicted dependenceΘ⁡\(log⁡M\)\\Theta\(\\sqrt\{\\log M\}\)on the amount of preference data\. We use a real\-world annotation setup whereby a large 70B model \(serving as the gold reward model\) generates preference data while proxy reward models are trained on these data using smaller models\. The scaling law is robust across optimization procedures, noisy preferences, and model sizes\.
- •Conceptual insight\.Our evidence in support of this scaling law connects post\-training to a surprisingly simple selection problem: choosing among a sequence of IID Gaussian random variables from noisy preference data\.

## 2Related Literature

#### Alignment and overoptimization\.

Overoptimization of a misspecified objective function \(or reward hacking\) is a long\-standing concern in AI alignment\([Wiener, 1960](https://arxiv.org/html/2609.38526#bib.bib9);[Leike et al\., 2017](https://arxiv.org/html/2609.38526#bib.bib10)\)\. Overoptimization can be defined as a regime in which increasing proxy reward strictly decreases true reward—an extreme form of Goodhart’s Law whereby optimization against a certain metric erodes its efficacy\([Manheim and Garrabrant, 2018](https://arxiv.org/html/2609.38526#bib.bib8)\)\. To counter this,[Skalse et al\. \(2022\)](https://arxiv.org/html/2609.38526#bib.bib17)define a natural notion ofunhackable rewardsbut find that it implies a very restrictive condition: any nontrivial unhackable rewards must induce the same policy ordering\. In MDPs,[Karwowski et al\. \(2024\)](https://arxiv.org/html/2609.38526#bib.bib18)give a geometric explanation of how the proxy and true\-optimal policies diverge once optimization pushes the occupancy measure to a face of the polytope where the two rewards disagree\. While this strand of literature focuses on mechanisms leading to overoptimization, we instead focus on quantifying its effect on achievable performance\. Closer to us,[Marklund et al\. \(2026\)](https://arxiv.org/html/2609.38526#bib.bib13)study a selection model in which an agent optimizing a consequentialist objective over a large outcome space incurs catastrophic performance with high probability\. Our model shares the outcome selection set\-up and information\-theoretic approach, but we separate policy divergence and preference data budgets and construct policies that achieve matching lower bounds\. Our goal is eventually to develop accurate scaling laws\.

#### Post\-training scaling laws\.

[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)provide the closest empirical antecedent to this work: in a synthetic setup where a fixed gold reward model stands in for human annotators, they measure gold reward as a function of KL\-divergence from the base policy and find that it rises likeK\\sqrt\{K\}and then declines\. However, while they vary the size of the preference dataset, they do not identify an explicit scaling law in that dimension\.222Quoting the authors, “The scaling ofα\\alphaandβ\\betawith data size are not as cleanly described as for RM size scaling\.”An inverted\-U shape also uncovered for direct policy optimization setups without a reward model\([Rafailov et al\., 2024](https://arxiv.org/html/2609.38526#bib.bib14)\)\. Pessimistic adjustments and robust ensembling methods can mitigate the perverse effects of overoptimization\([Coste et al\., 2024](https://arxiv.org/html/2609.38526#bib.bib15);[Eisenstein et al\., 2023](https://arxiv.org/html/2609.38526#bib.bib16);[Huang et al\., 2025](https://arxiv.org/html/2609.38526#bib.bib19)\)\. However, previous research often take the proxy model as exogenous and do not ask what specific divergence budget the preference data they are trained on can support\. The dependence onK\\sqrt\{K\}can be traced back to analyses of selection bias for adaptive data analyses\([Russo and Zou, 2016](https://arxiv.org/html/2609.38526#bib.bib7)\)\. Related work on AI alignment further characterizes tradeoffs between reward maximization and policy distribution shift\([Mroueh, 2024](https://arxiv.org/html/2609.38526#bib.bib5);[Laidlaw et al\., 2025](https://arxiv.org/html/2609.38526#bib.bib6)\)under assumptions on the proxy rewards\. Still, these works treat the underlying data as exogenous; in our language, the scaling is with respect toKKbut it does not make explicit the dependence on the amount of preference dataMM\. Instead our work reveals how information obtained from preference data constrains attainable rewards\.

#### Learning from comparisons\.

The proxy in equation[1](https://arxiv.org/html/2609.38526#S3.E1)is fit to pairwise preferences under a Bradley–Terry model\([Bradley and Terry, 1952](https://arxiv.org/html/2609.38526#bib.bib11);[Christiano et al\., 2017](https://arxiv.org/html/2609.38526#bib.bib12)\)\.[Zhu et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib20)give the first sample complexity bounds for this pipeline, showing that the MLE converges for linear rewards under Bradley–Terry–Luce and Plackett–Luce, but that a policy trained on the plain MLE can fail while a pessimistic MLE succeeds under a coverage condition\. This literature bounds how wellMMcomparisons pin down the reward; we instead ask how much reward gainMMcomparisons can support at all, irrespective of the estimator\. Closest to the combinatorial side of our question is work on selection from noisy comparisons:[Falahatgar et al\. \(2017\)](https://arxiv.org/html/2609.38526#bib.bib21)establish the comparison complexity of identifying an approximate maximum, and[Chen and Suh \(2015\)](https://arxiv.org/html/2609.38526#bib.bib22)that of top\-KKrecovery, with rates governed by the approximation tolerance and the gap at the top\-KKboundary, respectively\. However, these works do not explicitly quantify the corresponding expected reward gain nor they account for divergence constraints\.

## 3Model and Results

Consider the following setup for alignment\. For a prompt–response pair\(x,y\)∈𝒳×𝒴\(x,y\)\\in\\mathcal\{X\}\\times\\mathcal\{Y\}, human preferences are encoded by a*gold*rewardr⁡\(x,y\)r\(x,y\)that is never observed\. In its place we have a proxyr^​\(x,y\)\\hat\{r\}\(x,y\), calibrated on a dataset ofMMpairwise comparisons: for triples\(x,y,y′\)\(x,y,y^\{\\prime\}\), an annotator reports which ofy,y′y,y^\{\\prime\}is preferred\. Given a base policyπ0\\pi\_\{0\}that assigns to each prompt a distribution over responses, analignedpolicyπ^K\\hat\{\\pi\}\_\{K\}optimizes the proxy against a divergence budget,

maxπ𝔼x,π\[r^\(x,y\)\]subject to𝔼x\[DKL\(π\(⋅∣x\)∥π0\(⋅∣x\)\)\]≤K\.\\max\_\{\\pi\}\\;\\mathbb\{E\}\_\{x,\\,\\pi\}\\\!\\left\[\\hat\{r\}\(x,y\)\\right\]\\quad\\text\{subject to\}\\quad\\mathbb\{E\}\_\{x\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid x\)\\,\\middle\\\|\\,\\pi\_\{0\}\(\\cdot\\mid x\)\\right\)\\right\]\\;\\leq\\;K\.\(1\)KKconstrains*optimization pressure*\. Becauser^\\hat\{r\}is an imperfect proxy, driving expected proxy reward upward eventually drives expected gold reward downward\. WritingR⁡\(K\):=𝔼x,π^K​\[r⁡\(x,y\)\]R\(K\):=\\mathbb\{E\}\_\{x,\\,\\hat\{\\pi\}\_\{K\}\}\[r\(x,y\)\]for the gold reward earned by the aligned policy,[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)establish an empirical scaling law:R⁡\(K\)R\(K\)initially behaves likeK\\sqrt\{K\}, and beyond someK⋆K^\{\\star\}is decreasing inKK\.

The proxy is itself calibrated fromMMcomparisons, so bothR⁡\(⋅\)R\(\\cdot\)andK⋆K^\{\\star\}must depend onMM; the scaling law of[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)does not address how\. We seek to answer two related practical questions\. First,how does the scaling law depend onMMfor a givenKK, and in particular,how should the divergence budgetKKbe set as a function ofMM?And second,what gold reward gain over the base policy does the resulting policy earn?

### 3\.1An Information\-Theoretic Setup

We propose here a simplified setup that will allow us to make theoretical progress on our questions; candidate scaling laws obtained from the analysis of this setup are then evaluated in the empirical setup of[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)in §[5](https://arxiv.org/html/2609.38526#S5)\. For a universe ofnnpotential responses, indexed byii, we associate with each an independent gold rewardRi∼𝒩⁡\(μ,σ2\)R\_\{i\}\\sim\\mathcal\{N\}\(\\mu,\\sigma^\{2\}\)\. We model a prompt as an*admissible set*A⊆\[n\]A\\subseteq\[n\], withPr⁡\(i∈A\)=p\\Pr\(i\\in A\)=pindependently acrossii\. HereAAis the set of responses that are candidate answers to the realized prompt, andppis the fraction of the universe a prompt renders relevant\.333IfA=∅A=\\varnothingwe allow a dummy fallback action00with deterministic rewardR0=μR\_\{0\}=\\mu\.We remark that this setup is close to one studied recently by[Marklund et al\. \(2026\)](https://arxiv.org/html/2609.38526#bib.bib13)\.

A*preference dataset*contains a labelYi​jY\_\{ij\}for each pair\(i,j\)\(i,j\)in a setEEofMMresponse pairs\. The dataset is*deterministic*whenYi​j=𝟏\{Ri\>Rj\}Y\_\{ij\}=\\mathbf\{1\}\\\{R\_\{i\}\>R\_\{j\}\\\}, and*random*when the labels are sampled conditionally on the rewards\. We writeT:=\{\(i,j,Yi​j\):\(i,j\)∈E\}T:=\\\{\(i,j,Y\_\{ij\}\):\(i,j\)\\in E\\\}for the dataset\.

###### Assumption 1\(Information Structure\)\.

We assume thatE⟂\(R,A\)E\\perp\(R,A\)so that the pairs in the preference dataset are chosen without future knowledge of the admissible set or information aboutRR\. Moreover,A⟂\(R,T\)A\\perp\(R,T\)\.

We letπ0\(⋅∣A,E\)\\pi\_\{0\}\(\\cdot\\mid A,E\)be an arbitrary*reward\-blind base policy*\. Specifically, anAA\-valued random variableJJdistributed according toπ0\(⋅∣A,E\)\\pi\_\{0\}\(\\cdot\\mid A,E\)is dependent onAAand potentially,EEbut independent ofRR\. Consequently,𝔼π0​\[RJ\]=μ\\mathbb\{E\}\_\{\\pi\_\{0\}\}\[R\_\{J\}\]=\\mu\. An*informed policy*π\(⋅∣T,A\)\\pi\(\\cdot\\mid T,A\)selects an indexJ∈AJ\\in Aafter seeing the preference datasetTTand the admissible setAA\. Define the expected policy divergence and the reward gain by

K\(π;π0\)≜𝔼\[DKL\(π\(⋅∣T,A\)∥π0\(⋅∣A,E\)\)\],Δ\(π\)≜𝔼π\[RJ\]−μ\.K\(\\pi;\\pi\_\{0\}\)\\;\\triangleq\\;\\mathbb\{E\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid T,A\)\\,\\middle\\\|\\,\\pi\_\{0\}\(\\cdot\\mid A,E\)\\right\)\\right\],\\qquad\\Delta\(\\pi\)\\;\\triangleq\\;\\mathbb\{E\}\_\{\\pi\}\[R\_\{J\}\]\-\\mu\.\(2\)
We next present our results; taken together they provide a theoretical answer to the practical questions posed above\.

### 3\.2A Preview of Our Results

We present here an informal overview of our key result, which holds for both deterministic and random preference datasets:

###### Theorem\(Informal\)\.

SupposeK⁡\(π,π0\)≤KK\(\\pi;\\pi\_\{0\}\)\\leq Kand\|E\|≤M\|E\|\\leq M\. Then, for any suchπ\\pi,

Δ⁡\(π\)≲σ​2​min⁡\{K,log⁡\(p​M\)\}\.\\Delta\(\\pi\)\\;\\lesssim\\;\\sigma\\sqrt\{2\\min\\\{K,\\log\(pM\)\\\}\}\.Moreover, there exists a preference datasetEEand a resulting proxyr^\\hat\{r\}for which the optimized policyπ\\pisatisfies

Δ⁡\(π\)≳\(1−o⁡\(1\)\)​σ​2​min⁡\{K,log⁡\(p​M\)\}\.\\Delta\(\\pi\)\\;\\gtrsim\\;\(1\-o\(1\)\)\\,\\sigma\\sqrt\{2\\min\\\{K,\\log\(p\\sqrt\{M\}\)\\\}\}\.

The asymptotic regime implicit in the statements above requires thatmin⁡\{K,log⁡\(p​M\)\}\\min\\\{K,\\log\(p\\sqrt\{M\}\)\\\}grow large\.444The dependence onppis not central to our analysis;ppis a stylized measure of how much overlap there is in the admissible responses of distinct prompts—a notion of how well preference data generalizes across prompts\.That is, the regime is one where both the divergence budget, and the effective number of comparisons for a prompt grows\. We briefly discuss the implications of the above result here\. First, observe that the lower and upper bounds onΔ⁡\(π\)\\Delta\(\\pi\)are tight: they are within a constant factor of2\\sqrt\{2\}of each other\. As such, the bounds explain precisely theK\\sqrt\{K\}scaling law established by[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)\. The bounds also prescribe how to pick the divergence budgetKK: specifically, we wantK∼log⁡\(p​M\)K\\sim\\log\(pM\): any slower and we leave improvement opportunities on the table; any faster and we risk over\-optimizing against the proxy\. The next section formalizes the above theorem and presents its proof, following which we turn to empirically verifying that the above result holds in a \(larger scale\) variant of the scaling law study performed by[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)\.

## 4Scaling laws for reward optimization

### 4\.1Reward optimization scaling law under deterministic preferences

We now make the preceding bounds precise\. We allow the informed policy to use the comparison data directly, so our upper bound applies regardless of how a proxy is fit\. For the lower bound, we construct a proxy and a base policy that attain the stated gain\. Writer^​\(i,T\)\\hat\{r\}\(i;T\)for the proxy reward assigned to responseiiusing the comparison dataTT\. As in equation[1](https://arxiv.org/html/2609.38526#S3.E1), the aligned policyπ^K\\hat\{\\pi\}\_\{K\}maximizes expected proxy reward within divergence budgetKK\.

We assume for simplicity thatn≥2​Mn\\geq\\sqrt\{2M\}throughout the analysis\. This ensures enough possible responses for our construction and leaves the scaling law inMMandKKunchanged\. In practice, it is plausible to assume that the number of possible responses is much larger than our preference dataset\.

###### Theorem 2\(Deterministic preferences\)\.

Under the information structure above, every reward\-blind base policyπ0\\pi\_\{0\}, deterministic preference dataset generated on pairsEEwith\|E\|≤M\|E\|\\leq M, and informed policy withK⁡\(π,π0\)≤KK\(\\pi;\\pi\_\{0\}\)\\leq Kmust satisfy

Δ⁡\(π\)≤σ​2​min⁡\{K,log⁡\(1\+2​p​M\)\}\.\\Delta\(\\pi\)\\leq\\sigma\\sqrt\{2\\min\\\{K,\\log\(1\+2pM\)\\\}\}\.\(3\)Reciprocally, there is a universal constantc\>0c\>0such that, wheneverp​2​M≥4p\\sqrt\{2M\}\\geq 4, for everyK≥0K\\geq 0there exist a set of comparison pairsEEwith\|E\|≤M\|E\|\\leq M, a reward\-blind base policy, and a proxy such that, for the resulting deterministic preference dataset, every optimizerπ^K\\hat\{\\pi\}\_\{K\}satisfies

Δ⁡\(π^K\)≥c​σ​min⁡\{K,log⁡\(p​2​M\)\}\.\\Delta\(\\hat\{\\pi\}\_\{K\}\)\\geq c\\,\\sigma\\sqrt\{\\min\\\{K,\\log\(p\\sqrt\{2M\}\)\\\}\}\.\(4\)

The common upper bound is proved below, the lower\-bound construction is given in Appendix[A\.2](https://arxiv.org/html/2609.38526#A1.SS2)\.

### 4\.2The same scaling law holds under noisy preferences

So far, we assumed that preferences were deterministic: a response is recorded as preferred whenever it has the larger gold reward\. In reality, preference data is noisy\. For each\(i,j\)∈E\(i,j\)\\in E, letYi​j=1Y\_\{ij\}=1indicate thatiiis preferred tojj\. As is standard in RLHF, we model preference labels with the Bradley–Terry model\([Bradley and Terry, 1952](https://arxiv.org/html/2609.38526#bib.bib11)\):

Pr⁡\(Yi​j=1∣R,E\)=11\+exp\(−\(Ri−Rj\)/β\),β\>0\.\\Pr\(Y\_\{ij\}=1\\mid R,E\)=\\frac\{1\}\{1\+\\exp\(\-\(R\_\{i\}\-R\_\{j\}\)/\\beta\)\},\\qquad\\beta\>0\.\(5\)Hereβ\>0\\beta\>0controls the preference noise: largerβ\\betamakes labels less sensitive to reward differences\. The finite\-budget lower bound now depends onβ/σ\\beta/\\sigma, the noise scale relative to the variation in gold rewards\.

###### Theorem 3\(Noisy preferences\)\.

Suppose preferences follow the Bradley\-Terry model with conditionally independent labels and noise scaleβ\>0\\beta\>0\. Under the information structure above, every reward\-blind base policyπ0\\pi\_\{0\}, random preference dataset generated on pairsEEwith\|E\|≤M\|E\|\\leq M, and informed policy withK⁡\(π,π0\)≤KK\(\\pi;\\pi\_\{0\}\)\\leq Ksatisfy

Δ⁡\(π\)≤σ​2​min⁡\{K,log⁡\(1\+2​p​M\)\}\.\\Delta\(\\pi\)\\leq\\sigma\\sqrt\{2\\min\\\{K,\\log\(1\+2pM\)\\\}\}\.\(6\)Reciprocally, there is a constantc⁡\(β/σ\)\>0c\(\\beta/\\sigma\)\>0, depending only onβ/σ\\beta/\\sigma, such that, wheneverp​2​M≥4p\\sqrt\{2M\}\\geq 4, for everyK≥0K\\geq 0there exist a set of comparison pairsEEwith\|E\|≤M\|E\|\\leq M, a reward\-blind base policy, and a proxy such that, for the resulting random preference dataset, every optimizerπ^K\\hat\{\\pi\}\_\{K\}satisfies

Δ⁡\(π^K\)≥c⁡\(β/σ\)​σ​min⁡\{K,log⁡\(p​2​M\)\}\.\\Delta\(\\hat\{\\pi\}\_\{K\}\)\\geq c\(\\beta/\\sigma\)\\,\\sigma\\sqrt\{\\min\\\{K,\\log\(p\\sqrt\{2M\}\)\\\}\}\.\(7\)

Both lower bounds fix a set ofm=⌊2​M⌋m=\\lfloor\\sqrt\{2M\}\\rfloorresponses and compare each pair once, using\(m2\)≤M\\binom\{m\}\{2\}\\leq Mcomparisons\. Deterministic preferences reveal the order of these responses by gold reward\. With noisy preferences, we score each response by the fraction of itsm−1m\-1comparisons in which it is preferred\. Within any nonempty subset of these responses, we then select the response with the highest score\. For fixedσ,β\>0\\sigma,\\beta\>0, the expected gap between its gold reward and the largest gold reward in that subset tends to zero asmmgrows\. Appendix[A\.3](https://arxiv.org/html/2609.38526#A1.SS3)gives the full finite\-budget argument\. We prove the common upper bound below\.

###### Proof\.

Consider a preference dataset generated on a possibly random pair setEE, with\|E\|≤M\|E\|\\leq M, and an informed policyπ\\pisatisfyingK⁡\(π,π0\)≤KK\(\\pi;\\pi\_\{0\}\)\\leq K\. Both arguments below apply to deterministic preferences and to the Bradley–Terry model\.

#### The divergence budget\.

LetPπP\_\{\\pi\}andP0P\_\{0\}denote the joint laws of\(R,E,T,A,J\)\(R,E,T,A,J\)underπ\\piandπ0\\pi\_\{0\}\. These laws differ only in the conditional distribution of the chosenJJ\. The chain rule for relative entropy therefore gives

DKL\(Pπ∥P0\)=𝔼\[DKL\(π\(⋅∣T,A\)∥π0\(⋅∣A,E\)\)\]≤K\.D\_\{\\mathrm\{KL\}\}\(P\_\{\\pi\}\\\|P\_\{0\}\)=\\mathbb\{E\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid T,A\)\\,\\middle\\\|\\,\\pi\_\{0\}\(\\cdot\\mid A,E\)\\right\)\\right\]\\leq K\.UnderP0P\_\{0\}, the selected index is independent of the rewards\. Thus, for everyt\>0t\>0,

𝔼P0​\[et⁡\(RJ−μ\)\]=Pr⁡\(A=∅\)\+Pr⁡\(A≠∅\)​et2​σ2/2≤et2​σ2/2\.\\mathbb\{E\}\_\{P\_\{0\}\}\\\!\\left\[e^\{t\(R\_\{J\}\-\\mu\)\}\\right\]=\\Pr\(A=\\varnothing\)\+\\Pr\(A\\neq\\varnothing\)e^\{t^\{2\}\\sigma^\{2\}/2\}\\leq e^\{t^\{2\}\\sigma^\{2\}/2\}\.The entropy inequality𝔼Pf≤DKL\(P∥Q\)\+log𝔼Qef\\mathbb\{E\}\_\{P\}f\\leq D\_\{\\mathrm\{KL\}\}\(P\\\|Q\)\+\\log\\mathbb\{E\}\_\{Q\}e^\{f\}, applied withf=t⁡\(RJ−μ\)f=t\(R\_\{J\}\-\\mu\), gives

Δ⁡\(π\)≤Kt\+t​σ22\.\\Delta\(\\pi\)\\leq\\frac\{K\}\{t\}\+\\frac\{t\\sigma^\{2\}\}\{2\}\.Taking the infimum overt\>0t\>0givesΔ⁡\(π\)≤σ​2​K\\Delta\(\\pi\)\\leq\\sigma\\sqrt\{2K\}, includingK=0K=0by lettingt↓0t\\downarrow 0\.

#### The size of the preference dataset\.

LetV⁡\(E\)V\(E\)be the set of responses appearing inEE, so\|V⁡\(E\)\|≤2​M\|V\(E\)\|\\leq 2M\. Conditional onEE, the labels depend only on rewards inV⁡\(E\)V\(E\)and, in the noisy model, independent preference randomness\. Every response outsideV⁡\(E\)V\(E\)therefore has conditional meanμ\\mugiven\(T,A\)\(T,A\)\. Writingπi=π⁡\(i∣T,A\)\\pi\_\{i\}=\\pi\(i\\mid T,A\)and using the independence of the policy’s randomization fromRRgiven\(T,A\)\(T,A\), we obtain

Δ⁡\(π\)=𝔼⁡\[∑i∈A∩V⁡\(E\)πi​\(Ri−μ\)\]≤𝔼\[Z\],Z:=max\(\{0\}∪\{Ri−μ:i∈A∩V\(E\)\}\)\.\\begin\{split\}\\Delta\(\\pi\)&=\\mathbb\{E\}\\\!\\left\[\\sum\_\{i\\in A\\cap V\(E\)\}\\pi\_\{i\}\(R\_\{i\}\-\\mu\)\\right\]\\\\ &\\leq\\mathbb\{E\}\[Z\],\\qquad Z:=\\max\\bigl\(\\\{0\\\}\\cup\\\{R\_\{i\}\-\\mu:i\\in A\\cap V\(E\)\\\}\\bigr\)\.\\end\{split\}Fort\>0t\>0, independence ofEE,AA, and the rewards gives

𝔼⁡\[et​Z\]≤1\+𝔼⁡\[∑i∈A∩V⁡\(E\)et⁡\(Ri−μ\)\]=1\+p​𝔼​\|V⁡\(E\)\|​et2​σ2/2≤\(1\+2​p​M\)​et2​σ2/2\.\\begin\{split\}\\mathbb\{E\}\\left\[e^\{tZ\}\\right\]&\\leq 1\+\\mathbb\{E\}\\\!\\left\[\\sum\_\{i\\in A\\cap V\(E\)\}e^\{t\(R\_\{i\}\-\\mu\)\}\\right\]\\\\ &=1\+p\\mathbb\{E\}\|V\(E\)\|e^\{t^\{2\}\\sigma^\{2\}/2\}\\leq\(1\+2pM\)e^\{t^\{2\}\\sigma^\{2\}/2\}\.\\end\{split\}Jensen’s inequality and minimization overttnow imply

Δ⁡\(π\)≤inft\>0\{log⁡\(1\+2​p​M\)t\+t​σ22\}=σ​2​log⁡\(1\+2​p​M\)\.\\Delta\(\\pi\)\\leq\\inf\_\{t\>0\}\\left\\\{\\frac\{\\log\(1\+2pM\)\}\{t\}\+\\frac\{t\\sigma^\{2\}\}\{2\}\\right\\\}=\\sigma\\sqrt\{2\\log\(1\+2pM\)\}\.Combining the two bounds proves the upper bounds in Theorems[2](https://arxiv.org/html/2609.38526#Thmtheorem2)and[3](https://arxiv.org/html/2609.38526#Thmtheorem3)\. Notice that neither argument requires the size assumptions used for the lower bounds\. ∎

The constants in the two finite\-budget guarantees differ, but both admit the same sharp asymptotic bound\.

###### Corollary 4\(Asymptotic bound on performance\)\.

Fixσ\>0\\sigma\>0and, for Bradley–Terry preferences,β\>0\\beta\>0\. Under either preference model, along any sequence\(n,M,p,K\)\(n,M,p,K\)withK→∞K\\to\\inftyandp​2​M→∞p\\sqrt\{2M\}\\to\\infty, there exist a pair setEEwith\|E\|≤M\|E\|\\leq M, a reward\-blind base policy, and a proxy such that, for the resulting preference dataset, every optimizerπ^K\\hat\{\\pi\}\_\{K\}satisfies

Δ⁡\(π^K\)≥\(1−o⁡\(1\)\)​σ​2​min⁡\{K,log⁡\(p​2​M\)\}\.\\Delta\(\\hat\{\\pi\}\_\{K\}\)\\geq\(1\-o\(1\)\)\\,\\sigma\\sqrt\{2\\min\\\{K,\\log\(p\\sqrt\{2M\}\)\\\}\}\.\(8\)

## 5Experiments

### 5\.1Summary

In this section, we evaluate our proposed scaling law empirically and test its accuracy against other specifications\. We use a real\-world annotation set\-up and generate preference data from a large gold reward model\. We train proxy reward models with 0\.6B, 1\.7B, and 4B parameters and compare alternative scaling functions\. The square\-root\-logarithmic form fits well across model sizes, preference noise, and optimization methods\. It significantly outperforms linear and square\-root growth in all tested settings

### 5\.2Setup

#### Models and data\.

Following[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1), we study reward optimization using preferences supplied by a fixed gold reward model, Llama\-3\.3\-Nemotron\-70B\-Reward\. Our proxies are Qwen3 base models with 0\.6B, 1\.7B, and 4B parameters, each with a scalar reward head\. Our training corpus contains 1,000 first\-turn prompts from the Arena 140K dataset[LMArena \(2025\)](https://arxiv.org/html/2609.38526#bib.bib2), a highly diverse dataset spanning code, mathematics, and other topics \(the full mixture is described in Appendix[B\.1](https://arxiv.org/html/2609.38526#A2.SS1)\)\. We sample 600 responses per prompt from the fixed reference policy Qwen3\-4B\-Instruct at temperature 1\.

#### Preference data\.

We construct 300 disjoint response pairs per prompt, summing to 300,000 offline comparisons in our largest dataset\. We exclude identical\-response pairs and duplicate text pairs\. In the noiseless setting, the response with highest gold reward is preferred, with exact ties broken uniformly, as in[Gao et al\. \(2023\)](https://arxiv.org/html/2609.38526#bib.bib1)\. For the noisy\-label experiments, we sample a Bradley\-Terry preference with probabilityPr⁡\(ya≻yb∣x\)=σ⁡\(r⁡\(x,ya\)−r⁡\(x,yb\)\)\\Pr\(y\_\{a\}\\succ y\_\{b\}\\mid x\)=\\sigma\(r\(x,y\_\{a\}\)\-r\(x,y\_\{b\}\)\)\.

#### Training and comparison budgets\.

The main scaling experiment increasesMMfrom 600 to 300,000 by expanding the set of prompts in a fixed, reward\-independent order, while holding the number of comparisons per prompt at 300\. We fine\-tune all parameters for one pass using the pairwise logistic loss and AdamW at constant learning rate10−510^\{\-5\}\. Each policy update contains 600 comparisons from two complete prompt groups\.

#### Evaluation\.

We evaluate on 128 prompts disjoint from training, with 8,192 responses each\. Best\-of\-NNselects the highest\-proxy\-reward response amongNNcandidates\. We sweep logarithmically spacedNNup to 8,192, corresponding to the conventional optimization\-pressure coordinateKN=log⁡N−\(N−1\)/NK\_\{N\}=\\log N\-\(N\-1\)/N\([Gao et al\., 2023](https://arxiv.org/html/2609.38526#bib.bib1)\)\. We estimate gold and proxy rewards using the unbiased subset\-average estimator of[Nakano et al\. \(2021, Appendix I\)](https://arxiv.org/html/2609.38526#bib.bib23), which highly reduces the variance of the estimates\. Reward gains are measured relative to the mean reward atN=1N=1over the full response pool for each prompt, and are normalized by a fixed reference gold standard deviation\.

### 5\.3The peak reward follows the predicted square root logarithmic scaling

The comparison\-limited regime of Section[3\.2](https://arxiv.org/html/2609.38526#S3.SS2)predicts square\-root\-logarithmic growth of achievable gold\-reward gains\. We first test this scaling by varying the size of preference datasetMMsizes from600600to300,000300,000\. For each trained proxy reward \(MM\), we determine the best\-of\-N response model, varyingN∈𝒢N\\in\\mathcal\{G\}in our evaluated grid bounded byN≤8,192N\\leq 8\{,\}192\. We tuneNNas the observed peak ofΔ^max​\(M\)=maxN∈𝒢⁡Δ^M​\(N\)\\widehat\{\\Delta\}\_\{\\max\}\(M\)=\\max\_\{N\\in\\mathcal\{G\}\}\\widehat\{\\Delta\}\_\{M\}\(N\), whereΔ^M​\(N\)\\widehat\{\\Delta\}\_\{M\}\(N\)is the mean standardized gold\-reward gain\.

Table 1:Scaling fits across model size, preference noise, and optimization method\.EachR2R^\{2\}fitsa\+b​f​\(M\)a\+bf\(M\)on all evaluation prompts, bold marks the largestR2R^\{2\}in each row\. The final three columns report Holm\-adjusted, two\-sided pairedtt\-testpp\-values againstlog⁡M\\sqrt\{\\log M\}\.R2R^\{2\}pp\-value vs\.log⁡M\\sqrt\{\\log M\}ProxyNoiseMethodlog⁡M\\sqrt\{\\log M\}log⁡M\\log MMMM1/2M^\{1/2\}log⁡M\\log MMMM1/2M^\{1/2\}0\.6BNoneBoN0\.9920\.9900\.5710\.8031\.0006\.1×10−206\.1\\times 10^\{\-20\}3\.4×10−93\.4\\times 10^\{\-9\}0\.6BNoneTilt0\.9910\.9880\.5640\.7951\.0002\.7×10−202\.7\\times 10^\{\-20\}3\.1×10−103\.1\\times 10^\{\-10\}0\.6BBTLBoN0\.9510\.9540\.5370\.7750\.3921\.8×10−201\.8\\times 10^\{\-20\}7\.3×10−117\.3\\times 10^\{\-11\}0\.6BBTLTilt0\.9470\.9490\.5270\.7670\.7081\.3×10−201\.3\\times 10^\{\-20\}4\.7×10−114\.7\\times 10^\{\-11\}1\.7BNoneBoN0\.9890\.9810\.5200\.7570\.0238\.7×10−258\.7\\times 10^\{\-25\}2\.2×10−142\.2\\times 10^\{\-14\}1\.7BNoneTilt0\.9890\.9820\.5190\.7580\.0252\.3×10−242\.3\\times 10^\{\-24\}4\.3×10−154\.3\\times 10^\{\-15\}1\.7BBTLBoN0\.9630\.9570\.5110\.7431\.0004\.4×10−234\.4\\times 10^\{\-23\}2\.0×10−102\.0\\times 10^\{\-10\}1\.7BBTLTilt0\.9530\.9480\.5020\.7341\.0001\.4×10−221\.4\\times 10^\{\-22\}3\.4×10−103\.4\\times 10^\{\-10\}4BNoneBoN0\.9720\.9640\.4760\.7210\.0032\.4×10−282\.4\\times 10^\{\-28\}2\.7×10−172\.7\\times 10^\{\-17\}4BNoneTilt0\.9700\.9630\.4760\.7210\.0024\.2×10−304\.2\\times 10^\{\-30\}1\.5×10−181\.5\\times 10^\{\-18\}Figure[1](https://arxiv.org/html/2609.38526#S5.F1)shows that the peak gold reward closely follows the predicted scaling law\. Figure[2](https://arxiv.org/html/2609.38526#S5.F2)compares the predicted form with affine fitsa\+b​f​\(M\)a\+bf\(M\)using the same models and two free parameters for every function\. The square\-root\-logarithmic form gives the best fit, withR2=0\.992R^\{2\}=0\.992, compared with 0\.990, 0\.803, and 0\.571 forlog⁡M\\log M,M\\sqrt\{M\}, andMM, respectively\.

![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/gold_reward_scaling_0.6B.png)Figure 1:Gold reward follows square\-root\-logarithmic comparison\-budget scaling\.Ten 0\.6B models are selected by uniform spacing inlog⁡M\\sqrt\{\\log M\}\. Each point is the maximum mean gold\-reward gain over the evaluated best\-of\-NNgrid\. Both panels show the same fitted functiona\+b​log⁡Ma\+b\\sqrt\{\\log M\}: left on a square\-root\-logarithmic axis, right on a linearMMaxis\. Point labels denote comparison counts\.![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/gold_reward_alternative_fits_0.6B.png)Figure 2:Comparison of alternative scaling functions\.All curves are fitted to the same ten models as Figure[1](https://arxiv.org/html/2609.38526#S5.F1)\. The fitsa\+b​f​\(M\)a\+bf\(M\)are shown on a common linear comparison\-count axis\.Table[1](https://arxiv.org/html/2609.38526#S5.T1)summarizes the fits across proxy sizes, preference noise, and optimization methods, which we examine below\.

### 5\.4Robustness to the optimization algorithm

We test robustness to the optimization algorithm by replacing best\-of\-NNselection with exponential tilting, an idealized form of KL\-regularized reinforcement learning that approximates its optimal policy using samples from the reference policy\. For inverse temperatureβ\>0\\beta\>0, the objectivemaxπ\{𝔼π\[r^M\]−β−1DKL\(π∥π0\)\}\\max\_\{\\pi\}\\\{\\mathbb\{E\}\_\{\\pi\}\[\\hat\{r\}\_\{M\}\]\-\\beta^\{\-1\}D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{0\}\)\\\}has the exponentially tilted optimizerπβ,M​\(y∣x\)∝π0​\(y∣x\)​exp⁡\(β​r^M​\(x,y\)\)\\pi\_\{\\beta,M\}\(y\\mid x\)\\propto\\pi\_\{0\}\(y\\mid x\)\\exp\(\\beta\\hat\{r\}\_\{M\}\(x,y\)\)\([Rafailov et al\., 2023](https://arxiv.org/html/2609.38526#bib.bib3)\)\. We approximate this policy using responses from the reference policy\. The same exponential reweighting is used in soft best\-of\-NNsampling to approximate KL\-regularized alignment\([Verdun et al\., 2025](https://arxiv.org/html/2609.38526#bib.bib24)\), including empirical studies with imperfect reward models\([Aminian et al\., 2025](https://arxiv.org/html/2609.38526#bib.bib25)\)\.

For each evaluation promptxx, we place uniform reference mass on itsS=8,192S=8\{,\}192sampled responses and assign the tilted policy weights

wβ,M,i​\(x\)=exp⁡\(β​r^M​\(x,yi\)\)∑j=1Sexp⁡\(β​r^M​\(x,yj\)\)\.w\_\{\\beta,M,i\}\(x\)=\\frac\{\\exp\\\!\\left\(\\beta\\hat\{r\}\_\{M\}\(x,y\_\{i\}\)\\right\)\}\{\\sum\_\{j=1\}^\{S\}\\exp\\\!\\left\(\\beta\\hat\{r\}\_\{M\}\(x,y\_\{j\}\)\\right\)\}\.\(9\)We compute the weighted gold reward and conditional KL exactly on each pool, averaging over the same test prompts and retaining the gold\-reward normalization from Section[5\.2](https://arxiv.org/html/2609.38526#S5.SS2)\.

The right column of Figure[5](https://arxiv.org/html/2609.38526#A3.F5)in Appendix[C](https://arxiv.org/html/2609.38526#A3)shows the same square\-root\-logarithmic scaling under exponential tilting\.

### 5\.5Robustness to preference noise

We next test robustness to preference noise using the Bradley–Terry–Luce \(BTL\) model\([Bradley and Terry, 1952](https://arxiv.org/html/2609.38526#bib.bib11)\)\. Labels are drawn once and held fixed throughout training, 33\.6% of the 300,000 labels differ from their noiseless counterparts\. For each model size, we keep the same model initialization, comparison stream, and evaluation pools, so the comparison isolates the effect of preference noise\. Panels \(c\), \(d\), \(g\), and \(h\) of Figure[5](https://arxiv.org/html/2609.38526#A3.F5)show that the square\-root\-logarithmic trend persists under noisy preferences\.

The fit is slightly less tight than in the noiseless setting, its agreement with the scaling predicted by our theory is on par with the theory’s expected behavior for noisy labels\. In particular, the noisy\-label theorem predicts the same square\-root\-logarithmic scaling, but only in a higher\-comparison regime\. We hypothesize that in practice, additional samples are needed for the noise to average out and for the asymptotic scaling to emerge more cleanly\.

### 5\.6Robustness to proxy model size

Finally, we increase the proxy size from 0\.6B to 1\.7B and 4B parameters within the Qwen3 base\-model family\. Both proxies use the same noiseless preferences, comparison order, and evaluation pools\.

Table[1](https://arxiv.org/html/2609.38526#S5.T1)shows that the square\-root\-logarithmic form remains the best fit among the four specifications for both larger proxies and both optimization methods\. For 1\.7B,R2=0\.989R^\{2\}=0\.989under both methods; for 4B,R2=0\.972R^\{2\}=0\.972under best\-of\-NNand0\.9700\.970under tilting\. The paired prediction tests also favor this form overlog⁡M\\log M,M\\sqrt\{M\}, andMMfor both larger proxies under both methods, with Holm\-adjustedp<0\.05p<0\.05in each comparison\.

## 6Conclusion

Our work provides a theoretical and empirical characterization of how post\-training performance scales jointly with preference data and optimization effort\. A simple information\-theoretic model yields the scalingOPENmin⁡\(log⁡\(M\),KCLOSE\)\\sqrt\{\\min\(\\log\(M\),K\}\), with constructive procedures that attain this order\. Our experiments show that the predicted dependence on preference data provides a strong fit across model sizes, noisy preferences, and optimization procedures\. Extending the theory to more general reward distributions and adaptive preference collection, and testing the scaling with human preferences and policy training, are natural directions for future work\.

## References

- Aminianet al\.\(2025\)G\. Aminian, I\. Shenfeld, A\. R\. Asadi, A\. Beirami, and Y\. MrouehBest\-of\-N through the Smoothing Lens: KL Divergence and Regret Analysis\.arXiv\.Note:arXiv:2507\.05913 \[stat\.ML\]External Links:[Link](http://arxiv.org/abs/2507.05913),[Document](https://dx.doi.org/10.48550/arXiv.2507.05913)Cited by:[§5\.4](https://arxiv.org/html/2609.38526#S5.SS4.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2609.38526#S4.SS2.p1.1),[§5\.5](https://arxiv.org/html/2609.38526#S5.SS5.p1.1)\.
- Chen and Suh \(2015\)Y\. Chen and C\. SuhSpectral MLE: top\-KKrank aggregation from pairwise comparisons\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px3.p1.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px3.p1.1)\.
- Costeet al\.\(2024\)T\. Coste, U\. Anwar, R\. Kirk, and D\. KruegerReward model ensembles help mitigate overoptimization\.InInternational Conference on Learning Representations,Cited by:[§B\.3](https://arxiv.org/html/2609.38526#A2.SS3.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Eisensteinet al\.\(2023\)J\. Eisenstein, C\. Nagpal, A\. Agarwal, A\. Beirami, A\. D’Amour, D\. Dvijotham, A\. Fisch, K\. Heller, S\. Pfohl, D\. Ramachandran, P\. Shaw, and J\. BerantHelping or herding? reward model ensembles mitigate but do not eliminate reward hacking\.arXiv preprint arXiv:2312\.09244\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Falahatgaret al\.\(2017\)M\. Falahatgar, A\. Orlitsky, V\. Pichapati, and A\. T\. SureshMaximum selection and ranking under noisy comparisons\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 10835–10866\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.38526#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2609.38526#S3.SS2.p2.1),[§3](https://arxiv.org/html/2609.38526#S3.p1.2),[§3](https://arxiv.org/html/2609.38526#S3.p2.1),[§5\.2](https://arxiv.org/html/2609.38526#S5.SS2.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2609.38526#S5.SS2.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2609.38526#S5.SS2.SSS0.Px4.p1.1)\.
- Huanget al\.\(2025\)A\. Huang, A\. Block, Q\. Liu, N\. Jiang, A\. Krishnamurthy, and D\. J\. FosterIs best\-of\-n the best of them? coverage, scaling, and optimality in inference\-time alignment\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Karwowskiet al\.\(2024\)J\. Karwowski, O\. Hayman, X\. Bai, K\. Kiendlhofer, C\. Griffin, and J\. SkalseGoodhart’s law in reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1)\.
- Laidlawet al\.\(2025\)C\. Laidlaw, S\. Singhal, and A\. DraganCorrelated proxies: a new definition and improved mitigation for reward hacking\.InInternational Conference on Learning Representations,Vol\.2025\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Leikeet al\.\(2017\)J\. Leike, M\. Martic, V\. Krakovna, P\. A\. Ortega, T\. Everitt, A\. Lefrancq, L\. Orseau, and S\. LeggAI safety gridworlds\.arXiv preprint arXiv:1711\.09883\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1)\.
- LMArena \(2025\)LMArenaArena human preference 140k\.Note:[https://huggingface\.co/datasets/lmarena\-ai/arena\-human\-preference\-140k](https://huggingface.co/datasets/lmarena-ai/arena-human-preference-140k)Hugging Face datasetCited by:[§5\.2](https://arxiv.org/html/2609.38526#S5.SS2.SSS0.Px1.p1.1)\.
- Manheim and Garrabrant \(2018\)D\. Manheim and S\. GarrabrantCategorizing variants of goodhart’s law\.arXiv preprint arXiv:1803\.04585\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1)\.
- Marklundet al\.\(2026\)H\. Marklund, A\. Infanger, and B\. Van RoyConsequentialist objectives and catastrophe\.arXiv preprint arXiv:2603\.15017\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.38526#S3.SS1.p1.1)\.
- Mroueh \(2024\)Y\. MrouehInformation theoretic guarantees for policy alignment in large language models\.arXiv preprint arXiv:2406\.05883\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Nakanoet al\.\(2021\)R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders, X\. Jiang, K\. Cobbe, T\. Eloundou, G\. Krueger, K\. Button, M\. Knight, B\. Chess, and J\. SchulmanWebGPT: Browser\-assisted question\-answering with human feedback\.\(en\)\.External Links:[Link](https://arxiv.org/abs/2112.09332v3)Cited by:[§5\.2](https://arxiv.org/html/2609.38526#S5.SS2.SSS0.Px4.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§B\.3](https://arxiv.org/html/2609.38526#A2.SS3.p1.1),[§1](https://arxiv.org/html/2609.38526#S1.p1.1)\.
- Rafailovet al\.\(2024\)R\. Rafailov, Y\. Chittepu, R\. Park, H\. Sikchi, J\. Hejna, B\. Knox, C\. Finn, and S\. NiekumScaling laws for reward model overoptimization in direct alignment algorithms\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§5\.4](https://arxiv.org/html/2609.38526#S5.SS4.p1.1)\.
- Russo and Zou \(2016\)D\. Russo and J\. ZouControlling bias in adaptive data analysis using information theory\.InArtificial Intelligence and Statistics,pp\. 1232–1240\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px2.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward hacking\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2609.38526#S1.p1.1),[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1)\.
- Verdunet al\.\(2025\)C\. M\. Verdun, A\. Oesterling, H\. Lakkaraju, and F\. P\. CalmonSoft Best\-of\-n Sampling for Model Alignment\.arXiv\.Note:arXiv:2505\.03156 \[cs\.IT\]External Links:[Link](http://arxiv.org/abs/2505.03156),[Document](https://dx.doi.org/10.48550/arXiv.2505.03156)Cited by:[§5\.4](https://arxiv.org/html/2609.38526#S5.SS4.p1.1)\.
- Wiener \(1960\)N\. WienerSome moral and technical consequences of automation: as machines learn they may develop unforeseen strategies at rates that baffle their programmers\.\.Science131\(3410\),pp\. 1355–1358\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2023\)B\. Zhu, M\. Jordan, and J\. JiaoPrincipled reinforcement learning with human feedback from pairwise orKK\-wise comparisons\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202\.Cited by:[§2](https://arxiv.org/html/2609.38526#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AProofs

We first establish the upper bound shared by the two preference models\. We then prove the lower bound for exact preferences, extend the construction to noisy preferences, and derive the common asymptotic bound\. Throughout the lower\-bound proofs,σ\>0\\sigma\>0\. Ifσ=0\\sigma=0, every policy has zero gain\.

### A\.1The reward optimization upper bound

Consider a preference dataset generated on a possibly random pair setEE, with\|E\|≤M\|E\|\\leq M, and an informed policyπ\\pisatisfyingK⁡\(π,π0\)≤KK\(\\pi;\\pi\_\{0\}\)\\leq K\. Both arguments below apply to exact preferences and to the Bradley–Terry model\.

#### The divergence budget\.

LetPπP\_\{\\pi\}andP0P\_\{0\}denote the joint laws of\(R,E,T,A,J\)\(R,E,T,A,J\)underπ\\piandπ0\\pi\_\{0\}\. These laws differ only in the conditional distribution of the chosenJJ\. The chain rule for relative entropy therefore gives

DKL\(Pπ∥P0\)=𝔼\[DKL\(π\(⋅∣T,A\)∥π0\(⋅∣A,E\)\)\]≤K\.D\_\{\\mathrm\{KL\}\}\(P\_\{\\pi\}\\\|P\_\{0\}\)=\\mathbb\{E\}\\\!\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid T,A\)\\,\\middle\\\|\\,\\pi\_\{0\}\(\\cdot\\mid A,E\)\\right\)\\right\]\\leq K\.UnderP0P\_\{0\}, the selected index is independent of the rewards\. Thus, for everyt\>0t\>0,

𝔼P0​\[et⁡\(RJ−μ\)\]=Pr⁡\(A=∅\)\+Pr⁡\(A≠∅\)​et2​σ2/2≤et2​σ2/2\.\\mathbb\{E\}\_\{P\_\{0\}\}\\\!\\left\[e^\{t\(R\_\{J\}\-\\mu\)\}\\right\]=\\Pr\(A=\\varnothing\)\+\\Pr\(A\\neq\\varnothing\)e^\{t^\{2\}\\sigma^\{2\}/2\}\\leq e^\{t^\{2\}\\sigma^\{2\}/2\}\.The entropy inequality𝔼Pf≤DKL\(P∥Q\)\+log𝔼Qef\\mathbb\{E\}\_\{P\}f\\leq D\_\{\\mathrm\{KL\}\}\(P\\\|Q\)\+\\log\\mathbb\{E\}\_\{Q\}e^\{f\}, applied withf=t⁡\(RJ−μ\)f=t\(R\_\{J\}\-\\mu\), gives

Δ⁡\(π\)≤Kt\+t​σ22\.\\Delta\(\\pi\)\\leq\\frac\{K\}\{t\}\+\\frac\{t\\sigma^\{2\}\}\{2\}\.Taking the infimum overt\>0t\>0givesΔ⁡\(π\)≤σ​2​K\\Delta\(\\pi\)\\leq\\sigma\\sqrt\{2K\}, includingK=0K=0by lettingt↓0t\\downarrow 0\.

#### The size of the preference dataset\.

LetV⁡\(E\)V\(E\)be the set of responses appearing inEE, so\|V⁡\(E\)\|≤2​M\|V\(E\)\|\\leq 2M\. Conditional onEE, the labels depend only on rewards inV⁡\(E\)V\(E\)and, in the noisy model, independent preference randomness\. Every response outsideV⁡\(E\)V\(E\)therefore has conditional meanμ\\mugiven\(T,A\)\(T,A\)\. Writingπi=π⁡\(i∣T,A\)\\pi\_\{i\}=\\pi\(i\\mid T,A\)and using the independence of the policy’s randomization fromRRgiven\(T,A\)\(T,A\), we obtain

Δ⁡\(π\)=𝔼⁡\[∑i∈A∩V⁡\(E\)πi​\(Ri−μ\)\]≤𝔼Z,Z:=max\(\{0\}∪\{Ri−μ:i∈A∩V\(E\)\}\)\.\\begin\{split\}\\Delta\(\\pi\)&=\\mathbb\{E\}\\\!\\left\[\\sum\_\{i\\in A\\cap V\(E\)\}\\pi\_\{i\}\(R\_\{i\}\-\\mu\)\\right\]\\\\ &\\leq\\mathbb\{E\}Z,\\qquad Z:=\\max\\bigl\(\\\{0\\\}\\cup\\\{R\_\{i\}\-\\mu:i\\in A\\cap V\(E\)\\\}\\bigr\)\.\\end\{split\}Fort\>0t\>0, independence ofEE,AA, and the rewards gives

𝔼​et​Z≤1\+𝔼⁡\[∑i∈A∩V⁡\(E\)et⁡\(Ri−μ\)\]=1\+p​𝔼​\|V⁡\(E\)\|​et2​σ2/2≤\(1\+2​p​M\)​et2​σ2/2\.\\begin\{split\}\\mathbb\{E\}e^\{tZ\}&\\leq 1\+\\mathbb\{E\}\\\!\\left\[\\sum\_\{i\\in A\\cap V\(E\)\}e^\{t\(R\_\{i\}\-\\mu\)\}\\right\]\\\\ &=1\+p\\mathbb\{E\}\|V\(E\)\|e^\{t^\{2\}\\sigma^\{2\}/2\}\\leq\(1\+2pM\)e^\{t^\{2\}\\sigma^\{2\}/2\}\.\\end\{split\}Jensen’s inequality and minimization overttnow imply

Δ⁡\(π\)≤inft\>0\{log⁡\(1\+2​p​M\)t\+t​σ22\}=σ​2​log⁡\(1\+2​p​M\)\.\\Delta\(\\pi\)\\leq\\inf\_\{t\>0\}\\left\\\{\\frac\{\\log\(1\+2pM\)\}\{t\}\+\\frac\{t\\sigma^\{2\}\}\{2\}\\right\\\}=\\sigma\\sqrt\{2\\log\(1\+2pM\)\}\.Combining the two bounds proves the upper bounds in Theorems[2](https://arxiv.org/html/2609.38526#Thmtheorem2)and[3](https://arxiv.org/html/2609.38526#Thmtheorem3)\. Notice that neither argument requires the size assumptions used for the lower bounds\.

### A\.2Proof of Theorem[2](https://arxiv.org/html/2609.38526#Thmtheorem2)

It remains to prove the lower bound\. We first relate proxy optimization to the construction, then establish its feasibility and reward gain\.

#### The proxy\.

For either preference model, taker^​\(i,T\)=𝔼⁡\[Ri∣T\]\\hat\{r\}\(i;T\)=\\mathbb\{E\}\[R\_\{i\}\\mid T\], withr^​\(0,T\)=μ\\hat\{r\}\(0;T\)=\\mu\. SinceA⟂\(R,T\)A\\perp\(R,T\), every informed policy satisfies

𝔼πRJ=𝔼\[∑iπ\(i∣T,A\)𝔼\[Ri∣T,A\]\]=𝔼πr^\(J;T\)\.\\mathbb\{E\}\_\{\\pi\}R\_\{J\}=\\mathbb\{E\}\\\!\\left\[\\sum\_\{i\}\\pi\(i\\mid T,A\)\\mathbb\{E\}\[R\_\{i\}\\mid T,A\]\\right\]=\\mathbb\{E\}\_\{\\pi\}\\hat\{r\}\(J;T\)\.\(10\)It therefore suffices to construct a feasible informed policy with the claimed gain\. An optimizer exists for the following reason\. There are only finitely many possible pairs\(T,A\)\(T,A\): the dataset records pair identities and binary labels, andA⊆\[n\]A\\subseteq\[n\]\. Ignore pairs of probability zero\. A policy is then specified by finitely many choice probabilities\. A finite divergence budget requiresπ⁡\(i∣T,A\)=0\\pi\(i\\mid T,A\)=0wheneverπ0​\(i∣A,E\)=0\\pi\_\{0\}\(i\\mid A,E\)=0\. The remaining probabilities are nonnegative and sum to one for each\(T,A\)\(T,A\), so their domain is compact\. On this domain, the divergence is continuous: each termx​log⁡\(x/b\)x\\log\(x/b\)withb\>0b\>0extends continuously to zero by assigning it the value zero\. Hence the budget constraint defines a closed subset of this compact domain\. The feasible set is nonempty because it containsπ0\\pi\_\{0\}\. Finally, expected proxy reward is a continuous linear function of the choice probabilities, so it must attain its maximum on the feasible set\.

#### The preference dataset and base policy\.

For the remainder of the proof, write

m=⌊2​M⌋,q=p​2​M,h=min⁡\{K,log⁡q\}\.m=\\lfloor\\sqrt\{2M\}\\rfloor,\\qquad q=p\\sqrt\{2M\},\\qquad h=\\min\\\{K,\\log q\\\}\.We assumed thatm≤nm\\leq nandq≥4q\\geq 4\. We then choose a fixed setSSofmmresponses and compare every pair inSS\. This requires\(m2\)≤M\\binom\{m\}\{2\}\\leq Mcomparisons\. Thus the exact preference dataset reveals the full ranking ofSS\.

LetB=A∩SB=A\\cap SandN=\|B\|∼Bin⁡\(m,p\)N=\|B\|\\sim\\mathrm\{Bin\}\(m,p\)\. WhenN\>0N\>0, take the base policy to be uniform onBB\. WhenN=0N=0, take it to be uniform onAA, or to select action00ifA=∅A=\\varnothing\. We will use

p​m≥q−1≥3,p​m≤q,log⁡\(p​m\)≥12​log⁡q\.pm\\geq q\-1\\geq 3,\\qquad pm\\leq q,\\qquad\\log\(pm\)\\geq\\tfrac\{1\}\{2\}\\log q\.\(11\)where the last inequality follows fromq−1≥qq\-1\\geq\\sqrt\{q\}forq≥4q\\geq 4\.

#### Selection within the divergence budget\.

ForK≥log⁡2K\\geq\\log 2, sets=min⁡\{N,⌊eK⌋\}s=\\min\\\{N,\\lfloor e^\{K\}\\rfloor\\\}\. WhenN\>0N\>0, we draw anss\-element subsetUUuniformly fromBB, independently of the rewards and labels conditional onAA, and select its highest\-ranked element\. To bound divergence, consider a reference rule that draws the same subset and then selects uniformly from it\. Its marginal distribution on responses is uniform onBB, exactly the base policy\. Conditional on\(T,A,U\)\(T,A,U\), selecting one member instead of a uniform member has divergencelog⁡s\\log s\. Marginalizing overUUcannot increase relative entropy, so

DKL\(π\(⋅∣T,A\)∥π0\(⋅∣A,E\)\)≤logs≤K\(N\>0\)\.D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid T,A\)\\,\\middle\\\|\\,\\pi\_\{0\}\(\\cdot\\mid A,E\)\\right\)\\leq\\log s\\leq K\\qquad\(N\>0\)\.\(12\)This feasibility holds for any ranking obtained from the preference dataset\.

For0≤K<log⁡20\\leq K<\\log 2, a subset of size⌊eK⌋\\lfloor e^\{K\}\\rfloorwould give no gain\. Instead, whenN≥2N\\geq 2, sample a pair uniformly fromBBand choose its higher\-ranked element with probability\(1\+θ\)/2\(1\+\\theta\)/2, whereθ=K\\theta=\\sqrt\{K\}, choosing the other member otherwise\. Follow the base policy whenN<2N<2\. The reference rule again chooses uniformly from the same pair\. Its marginal is thus the base policy, and the conditional divergence is

d⁡\(θ\):=1\+θ2​log⁡\(1\+θ\)\+1−θ2​log⁡\(1−θ\)≤θ2=K\.d\(\\theta\):=\\frac\{1\+\\theta\}\{2\}\\log\(1\+\\theta\)\+\\frac\{1\-\\theta\}\{2\}\\log\(1\-\\theta\)\\leq\\theta^\{2\}=K\.\(13\)The inequality follows fromlog⁡x≤x−1\\log x\\leq x\-1\. Marginalizing over the sampled pair proves feasibility, again for any ranking\.

#### Two estimates for the reward gain\.

For independent standard GaussiansZ1,Z2,…Z\_\{1\},Z\_\{2\},\\ldots, defineaj=𝔼​max1≤i≤j​Zia\_\{j\}=\\mathbb\{E\}\\max\_\{1\\leq i\\leq j\}Z\_\{i\}forj≥1j\\geq 1, and seta0=0a\_\{0\}=0\. We use the Gaussian maximal inequalities:

aj≥c0​log⁡j\(j≥2\),aj∼2​log⁡j\(j→∞\),a\_\{j\}\\geq c\_\{0\}\\sqrt\{\\log j\}\\quad\(j\\geq 2\),\\qquad a\_\{j\}\\sim\\sqrt\{2\\log j\}\\quad\(j\\to\\infty\),\(14\)for a universalc0\>0c\_\{0\}\>0\.

We also need a uniform lower\-tail estimate forNN\. Letν=p​m≥3\\nu=pm\\geq 3andF=\{N≥ν/2\}F=\\\{N\\geq\\nu/2\\\}\. Since𝔼​N2≤ν2\+ν\\mathbb\{E\}N^\{2\}\\leq\\nu^\{2\}\+\\nu, the inequality

ν=𝔼​N≤ν/2\+𝔼​N2​Pr⁡\(F\)\\nu=\\mathbb\{E\}N\\leq\\nu/2\+\\sqrt\{\\mathbb\{E\}N^\{2\}\\Pr\(F\)\}gives

Pr\(F\)≥ν4​\(ν\+1\)≥16,N≥2,logN≥12logνonF\.\\Pr\(F\)\\geq\\frac\{\\nu\}\{4\(\\nu\+1\)\}\\geq\\frac\{1\}\{6\},\\qquad N\\geq 2,\\quad\\log N\\geq\\tfrac\{1\}\{2\}\\log\\nu\\quad\\text\{on \}F\.\(15\)To see the last two claims, if3≤ν≤43\\leq\\nu\\leq 4, integrality givesN≥2≥νN\\geq 2\\geq\\sqrt\{\\nu\}onFF: ifν≥4\\nu\\geq 4, thenN≥ν/2≥νN\\geq\\nu/2\\geq\\sqrt\{\\nu\}\.

#### The achieved gain\.

Under exact preferences, the subset rule selects its largest gold reward\. The subset was drawn independently of rewards, so

Δ⁡\(π\)=σ​𝔼​as\.\\Delta\(\\pi\)=\\sigma\\mathbb\{E\}a\_\{s\}\.\(16\)ForK≥log⁡2K\\geq\\log 2, the inequality⌊eK⌋≥eK/2\\lfloor e^\{K\}\\rfloor\\geq e^\{K/2\}, together with equation[11](https://arxiv.org/html/2609.38526#A1.E11)and equation[15](https://arxiv.org/html/2609.38526#A1.E15), implies, onFF,

log⁡s≥12​min​\{K,log⁡\(p​m\)\}≥14​h\.\\log s\\geq\\tfrac\{1\}\{2\}\\min\\\{K,\\log\(pm\)\\\}\\geq\\tfrac\{1\}\{4\}h\.Sinceas≥0a\_\{s\}\\geq 0also outsideFF, we obtain

Δ⁡\(π\)≥σ​Pr⁡\(F\)​c0​h/4≥c012​σ​h\.\\Delta\(\\pi\)\\geq\\sigma\\Pr\(F\)c\_\{0\}\\sqrt\{h/4\}\\geq\\frac\{c\_\{0\}\}\{12\}\\sigma\\sqrt\{h\}\.ForK<log⁡2K<\\log 2, the sampled pair has independent Gaussian rewards and𝔼​\|Ri−Rj\|=2​σ/π\\mathbb\{E\}\|R\_\{i\}\-R\_\{j\}\|=2\\sigma/\\sqrt\{\\pi\}\. Biasing the uniform choice toward the better response byθ=K\\theta=\\sqrt\{K\}therefore gives

Δ⁡\(π\)=σ​Kπ​Pr⁡\(N≥2\)≥σ​K6​π\.\\Delta\(\\pi\)=\\frac\{\\sigma\\sqrt\{K\}\}\{\\sqrt\{\\pi\}\}\\Pr\(N\\geq 2\)\\geq\\frac\{\\sigma\\sqrt\{K\}\}\{6\\sqrt\{\\pi\}\}\.\(17\)Hereh=Kh=K, sinceq≥4q\\geq 4\. Both ranges are thus covered by the universal constant

cex:=min⁡\{c012,16​π\}\>0\.c\_\{\\mathrm\{ex\}\}:=\\min\\left\\\{\\frac\{c\_\{0\}\}\{12\},\\frac\{1\}\{6\\sqrt\{\\pi\}\}\\right\\\}\>0\.\(18\)AtK=0K=0, the pair rule is the base policy and the claimed lower bound is zero\. Applying equation[10](https://arxiv.org/html/2609.38526#A1.E10)completes the proof of Theorem[2](https://arxiv.org/html/2609.38526#Thmtheorem2)\.

### A\.3Proof of Theorem[3](https://arxiv.org/html/2609.38526#Thmtheorem3)

The upper bound follows from Appendix[A\.1](https://arxiv.org/html/2609.38526#A1.SS1)\. For the lower bound, remember the setSS, base policy, and notation from Appendix[A\.2](https://arxiv.org/html/2609.38526#A1.SS2), withq≥4q\\geq 4\. The subset and pair rules remain feasible for any ordering ofSS\. We proceed by bounding the reward lost when this ordering is obtained from a random preference dataset\.

#### Estimating the ranking\.

Fori∈Si\\in S, letwiw\_\{i\}be its fraction of wins and let

ui=𝔼⁡\[wi∣R\]=1m−1​∑j∈S∖\{i\}ℓ⁡\(Ri−Rjβ\),ℓ⁡\(x\)=11\+e−x\.u\_\{i\}=\\mathbb\{E\}\[w\_\{i\}\\mid R\]=\\frac\{1\}\{m\-1\}\\sum\_\{j\\in S\\setminus\\\{i\\\}\}\\ell\\\!\\left\(\\frac\{R\_\{i\}\-R\_\{j\}\}\{\\beta\}\\right\),\\qquad\\ell\(x\)=\\frac\{1\}\{1\+e^\{\-x\}\}\.Order responses bywiw\_\{i\}, breaking ties by index\. Set

bm=2​σ​log⁡m,tm=2​log⁡mm−1,b\_\{m\}=2\\sigma\\sqrt\{\\log m\},\\qquad t\_\{m\}=\\sqrt\{\\frac\{2\\log m\}\{m\-1\}\},and define

GR=\{maxi∈S\|Ri−μ\|≤bm\},GC=\{maxi∈S\|wi−ui\|≤tm\}\.G\_\{R\}=\\left\\\{\\max\_\{i\\in S\}\|R\_\{i\}\-\\mu\|\\leq b\_\{m\}\\right\\\},\\qquad G\_\{C\}=\\left\\\{\\max\_\{i\\in S\}\|w\_\{i\}\-u\_\{i\}\|\\leq t\_\{m\}\\right\\\}\.The Gaussian tail bound and a union bound givePr⁡\(GRc\)≤2/m\\Pr\(G\_\{R\}^\{c\}\)\\leq 2/m\. Conditional onRR, eachwiw\_\{i\}averagesm−1m\-1independent Bernoulli variables\. Hoeffding’s inequality and another union bound therefore give

Pr⁡\(GCc∣R\)≤2​m​exp⁡\{−2​\(m−1\)​tm2\}=2m3\.\\Pr\(G\_\{C\}^\{c\}\\mid R\)\\leq 2m\\exp\\\{\-2\(m\-1\)t\_\{m\}^\{2\}\\\}=\\frac\{2\}\{m^\{3\}\}\.The scores of different responses need not be independent for this bound\.

OnGRG\_\{R\}, every argument ofℓ\\ellin these scores lies in\[−2bm/β,2bm/β\]\[\-2b\_\{m\}/\\beta,2b\_\{m\}/\\beta\]\. Sinceℓ′​\(x\)≥e−\|x\|/4\\ell^\{\\prime\}\(x\)\\geq e^\{\-\|x\|\}/4, ifRi≥RjR\_\{i\}\\geq R\_\{j\}then

\(m−1\)​\(ui−uj\)\\displaystyle\(m\-1\)\(u\_\{i\}\-u\_\{j\}\)=ℓ⁡\(Ri−Rjβ\)−ℓ⁡\(Rj−Riβ\)\\displaystyle=\\ell\\\!\\left\(\\frac\{R\_\{i\}\-R\_\{j\}\}\{\\beta\}\\right\)\-\\ell\\\!\\left\(\\frac\{R\_\{j\}\-R\_\{i\}\}\{\\beta\}\\right\)\+∑k∈S∖\{i,j\}\[ℓ\(Ri−Rkβ\)−ℓ\(Rj−Rkβ\)\]\\displaystyle\\quad\+\\sum\_\{k\\in S\\setminus\\\{i,j\\\}\}\\left\[\\ell\\\!\\left\(\\frac\{R\_\{i\}\-R\_\{k\}\}\{\\beta\}\\right\)\-\\ell\\\!\\left\(\\frac\{R\_\{j\}\-R\_\{k\}\}\{\\beta\}\\right\)\\right\]≥me−2bm/β4​β​\(Ri−Rj\)\.\\displaystyle\\geq\\frac\{me^\{\-2b\_\{m\}/\\beta\}\}\{4\\beta\}\(R\_\{i\}\-R\_\{j\}\)\.\(19\)The mutual comparison contributes two terms to this lower bound, each of the otherm−2m\-2opponents contributes one\. Ifwj≥wiw\_\{j\}\\geq w\_\{i\}despiteRi≥RjR\_\{i\}\\geq R\_\{j\}, thenui−uj≤2​tmu\_\{i\}\-u\_\{j\}\\leq 2t\_\{m\}onGCG\_\{C\}\. Consequently, onGR∩GCG\_\{R\}\\cap G\_\{C\}, every such inverted pair satisfies

Ri−Rj≤δm,δm=8​β​e2​bm/β​tm\.R\_\{i\}\-R\_\{j\}\\leq\\delta\_\{m\},\\qquad\\delta\_\{m\}=8\\beta e^\{2b\_\{m\}/\\beta\}t\_\{m\}\.\(20\)This holds simultaneously for every pair\. In particular, in any nonempty subset ofSS, the response with the largest score has reward withinδm\\delta\_\{m\}of the largest true reward\.

To control the loss off this event, letDm=maxi∈S⁡Ri−mini∈S⁡RiD\_\{m\}=\\max\_\{i\\in S\}R\_\{i\}\-\\min\_\{i\\in S\}R\_\{i\}andUm=maxi∈S⁡\|Ri−μ\|/σU\_\{m\}=\\max\_\{i\\in S\}\|R\_\{i\}\-\\mu\|/\\sigma\. A Gaussian union bound gives

𝔼Um2≤∫0∞min\{1,2me−v/2\}dv=2log\(2m\)\+2,\\mathbb\{E\}U\_\{m\}^\{2\}\\leq\\int\_\{0\}^\{\\infty\}\\min\\\{1,2me^\{\-v/2\}\\\}\\,dv=2\\log\(2m\)\+2,so𝔼​Dm2≤8​σ2​\(log⁡\(2​m\)\+1\)\\mathbb\{E\}D\_\{m\}^\{2\}\\leq 8\\sigma^\{2\}\(\\log\(2m\)\+1\)\. SincePr⁡\(\(GR∩GC\)c\)≤4/m\\Pr\(\(G\_\{R\}\\cap G\_\{C\}\)^\{c\}\)\\leq 4/m, Cauchy–Schwarz shows that the expected loss from maximizing the score rather than the reward in any subset is at most

ηm=8​β​e\(4​σ/β\)​log⁡m​2​log⁡mm−1\+4​2​σ​log⁡\(2​m\)\+1m\.\\eta\_\{m\}=8\\beta e^\{\(4\\sigma/\\beta\)\\sqrt\{\\log m\}\}\\sqrt\{\\frac\{2\\log m\}\{m\-1\}\}\+4\\sqrt\{2\}\\sigma\\sqrt\{\\frac\{\\log\(2m\)\+1\}\{m\}\}\.\(21\)The bound is uniform over the subset, and hence applies to the random subsets used by our policy\. It also applies to a random pair\. Empty admissible sets withinSSuse the common fallback and incur no loss\.

#### The resulting reward gain\.

ForK≥log⁡2K\\geq\\log 2, draw a uniform subset ofBBof sizes=min⁡\{N,⌊eK⌋\}s=\\min\\\{N,\\lfloor e^\{K\}\\rfloor\\\}and select its highest\-scoring member\. The subset is sampled independently of rewards and comparison data conditional onAA\. Equation[16](https://arxiv.org/html/2609.38526#A1.E16)and the uniform loss bound imply

Δ⁡\(π\)≥σ​𝔼​as−ηm,\\Delta\(\\pi\)\\geq\\sigma\\mathbb\{E\}a\_\{s\}\-\\eta\_\{m\},\(22\)wheres=0s=0onN=0N=0anda0=0a\_\{0\}=0\. For0≤K<log⁡20\\leq K<\\log 2, favor the highest\-scoring member of a pair sampled uniformly fromBBwith biasK\\sqrt\{K\}\. This rule is the mixture of selecting that member with probabilityK\\sqrt\{K\}and selecting uniformly from the pair otherwise\. Its loss relative to the exact\-ranking rule is therefore at mostK​ηm\\sqrt\{K\}\\,\\eta\_\{m\}\. By equation[17](https://arxiv.org/html/2609.38526#A1.E17),

Δ⁡\(π\)≥K​\(σπ​Pr⁡\(N≥2\)−ηm\)\.\\Delta\(\\pi\)\\geq\\sqrt\{K\}\\left\(\\frac\{\\sigma\}\{\\sqrt\{\\pi\}\}\\Pr\(N\\geq 2\)\-\\eta\_\{m\}\\right\)\.\(23\)The base policy is used whenN≤1N\\leq 1\. The factorK\\sqrt\{K\}on the error makes this estimate uniform asKKdecreases to zero\.

#### A bound for every dataset size\.

Setr=β/σr=\\beta/\\sigma\. Dividing equation[21](https://arxiv.org/html/2609.38526#A1.E21)byσ\\sigmagives

ηmσ=8​r​e\(4/r\)​log⁡m​2​log⁡mm−1\+4​2​log⁡\(2​m\)\+1m⟶0\\frac\{\\eta\_\{m\}\}\{\\sigma\}=8re^\{\(4/r\)\\sqrt\{\\log m\}\}\\sqrt\{\\frac\{2\\log m\}\{m\-1\}\}\+4\\sqrt\{2\}\\sqrt\{\\frac\{\\log\(2m\)\+1\}\{m\}\}\\longrightarrow 0\(24\)for every fixedr\>0r\>0: the exponential factor ismo⁡\(1\)m^\{o\(1\)\}\. Recall thatcex≤1/\(6​π\)c\_\{\\mathrm\{ex\}\}\\leq 1/\(6\\sqrt\{\\pi\}\)by equation[18](https://arxiv.org/html/2609.38526#A1.E18)\. Choose an integerm0​\(r\)≥4m\_\{0\}\(r\)\\geq 4such that, for allm≥m0​\(r\)m\\geq m\_\{0\}\(r\),

ηmσ≤min⁡\{cex​log⁡22,112​π\}\.\\frac\{\\eta\_\{m\}\}\{\\sigma\}\\leq\\min\\left\\\{\\frac\{c\_\{\\mathrm\{ex\}\}\\sqrt\{\\log 2\}\}\{2\},\\frac\{1\}\{12\\sqrt\{\\pi\}\}\\right\\\}\.This threshold depends only onrr\. WhenK≥log⁡2K\\geq\\log 2, we haveh≥log⁡2h\\geq\\log 2, and the deterministic subset bound together with equation[22](https://arxiv.org/html/2609.38526#A1.E22)givesΔ⁡\(π\)≥\(cex/2\)​σ​h\\Delta\(\\pi\)\\geq\(c\_\{\\mathrm\{ex\}\}/2\)\\sigma\\sqrt\{h\}\. WhenK<log⁡2K<\\log 2, we haveh=Kh=K; usingPr⁡\(N≥2\)≥1/6\\Pr\(N\\geq 2\)\\geq 1/6in equation[23](https://arxiv.org/html/2609.38526#A1.E23)gives the same conclusion\.

To cover the remaining finite rangem<m0​\(r\)m<m\_\{0\}\(r\), we sample a pair uniformly fromBBand use its observed preference, following the base policy ifN<2N<2\. IfK≥log⁡2K\\geq\\log 2, select the preferred member, otherwise we favor it with biasK\\sqrt\{K\}\. The pair calculation in the deterministic proof gives divergence at mostKKfor both rules, regardless of the label\. IfD∼𝒩⁡\(0,2\)D\\sim\\mathcal\{N\}\(0,2\), define

γ⁡\(r\)=12​𝔼​\[D​tanh⁡\(D2​r\)\]\>0\.\\gamma\(r\)=\\frac\{1\}\{2\}\\mathbb\{E\}\\left\[D\\tanh\\\!\\left\(\\frac\{D\}\{2r\}\\right\)\\right\]\>0\.Conditioning on the two rewards shows that following their observed label has expected gainσ​γ​\(r\)\\sigma\\gamma\(r\)\. Favoring it with biasK\\sqrt\{K\}multiplies this gain byK\\sqrt\{K\}\. The expectation definingγ⁡\(r\)\\gamma\(r\)is finite, and its integrand is positive almost surely\. Therefore

Δ⁡\(π\)≥σ​γ​\(r\)6​min⁡\{K,1\}\.\\Delta\(\\pi\)\\geq\\frac\{\\sigma\\gamma\(r\)\}\{6\}\\min\\\{\\sqrt\{K\},1\\\}\.In this rangeq≤2​M<m\+1≤m0​\(r\)\+1q\\leq\\sqrt\{2M\}<m\+1\\leq m\_\{0\}\(r\)\+1, so

min⁡\{K,1\}≥hmax⁡\{1,log⁡\(m0​\(r\)\+1\)\}\.\\min\\\{\\sqrt\{K\},1\\\}\\geq\\frac\{\\sqrt\{h\}\}\{\\sqrt\{\\max\\\{1,\\log\(m\_\{0\}\(r\)\+1\)\\\}\}\}\.Thus a positive constant valid at every finite budget is

cnoisy​\(r\)=min⁡\{cex2,γ⁡\(r\)6​max⁡\{1,log⁡\(m0​\(r\)\+1\)\}\}\.c\_\{\\mathrm\{noisy\}\}\(r\)=\\min\\left\\\{\\frac\{c\_\{\\mathrm\{ex\}\}\}\{2\},\\frac\{\\gamma\(r\)\}\{6\\sqrt\{\\max\\\{1,\\log\(m\_\{0\}\(r\)\+1\)\\\}\}\}\\right\\\}\.All the constructed policies use the same reward\-blind base and meet the divergence constraint\. Equation[10](https://arxiv.org/html/2609.38526#A1.E10)transfers their gain to every constrained proxy optimizer, proving the theorem\.

### A\.4Proof of Corollary[4](https://arxiv.org/html/2609.38526#Thmtheorem4)

We compare every pair inSSand use the same subset rule as above\. Along the sequences in the corollary,q=p​2​M→∞q=p\\sqrt\{2M\}\\to\\inftyandK→∞K\\to\\infty\. Sinceq−1≤p​m≤qq\-1\\leq pm\\leq q, we havep​m/q→1pm/q\\to 1andm→∞m\\to\\infty\. Write

hm:=min⁡\{K,log⁡\(p​m\)\}=h\+o⁡\(1\),h→∞\.h\_\{m\}:=\\min\\\{K,\\log\(pm\)\\\}=h\+o\(1\),\\qquad h\\to\\infty\.A binomial Chernoff bound givesPr\(N≥pm/2\)≥1−e−pm/8\\Pr\(N\\geq pm/2\)\\geq 1\-e^\{\-pm/8\}\. On this event, for all sufficiently large budgets,

s=min⁡\{N,⌊eK⌋\}≥12​min⁡\{p​m,eK\}=12​ehm\.s=\\min\\\{N,\\lfloor e^\{K\}\\rfloor\\\}\\geq\\tfrac\{1\}\{2\}\\min\\\{pm,e^\{K\}\\\}=\\tfrac\{1\}\{2\}e^\{h\_\{m\}\}\.Monotonicity and nonnegativity ofaja\_\{j\}, followed by equation[14](https://arxiv.org/html/2609.38526#A1.E14), give

𝔼as≥\(1−e−pm/8\)a⌊ehm/2⌋≥\(1−o\(1\)\)2​h\.\\mathbb\{E\}a\_\{s\}\\geq\(1\-e^\{\-pm/8\}\)a\_\{\\lfloor e^\{h\_\{m\}\}/2\\rfloor\}\\geq\(1\-o\(1\)\)\\sqrt\{2h\}\.\(25\)The exact\-preference construction has gainσ​𝔼​as\\sigma\\mathbb\{E\}a\_\{s\}\. For noisy preferences, the construction loses at mostηm\\eta\_\{m\}, as established in equation[22](https://arxiv.org/html/2609.38526#A1.E22)\. For fixedσ,β\>0\\sigma,\\beta\>0,ηm=o⁡\(1\)\\eta\_\{m\}=o\(1\), so this loss is negligible relative toσ​h\\sigma\\sqrt\{h\}\. Thus either preference model admits a feasible policy with gain

\(1−o⁡\(1\)\)​σ​2​min⁡\{K,log⁡\(p​2​M\)\}\.\(1\-o\(1\)\)\\sigma\\sqrt\{2\\min\\\{K,\\log\(p\\sqrt\{2M\}\)\\\}\}\.Thus the posterior\-mean proxy transfers this bound to every optimizer by equation[10](https://arxiv.org/html/2609.38526#A1.E10)\.

## Appendix BAdditional Experimental Details

### B\.1Data construction and topic composition

We use English first\-turn prompts from LMArena’sarena\-human\-preference\-140k\. The final training set contains 1,000 prompts, each contributing 300 disjoint response pairs and hence 600 retained responses\. Responses are sampled from Qwen3\-4B\-Instruct\-2507 with temperature 1, top\-p=1p=1, no top\-kkrestriction, and at most 1,024 new tokens\. Pair construction excludes identical responses and repeated text pairs, additional responses are sampled when necessary to obtain 300 eligible pairs\.

Table[2](https://arxiv.org/html/2609.38526#A2.T2)gives the prompt\-topic distribution\.

Table 2:Training and evaluation prompt topics\.Counts and percentages within each set, using original Arena annotations\. Tags may overlap\. The dataset has no finer topic annotations for prompts marked*Topic unspecified*\.
### B\.2Conditional gold\-reward distributions

The theoretical model in Section[3\.1](https://arxiv.org/html/2609.38526#S3.SS1)assumes Gaussian rewards\. We examine this idealization using the empirical distribution of gold scores within individual prompts\. Each example uses all 8,192 responses in its test\-prompt pool, scored by the fixed gold reward model before best\-of\-NNselection\.

We choose four prompts independently of their reward values\. The examples concern AutoHotkey scrolling, a modal\-logic agent, motivational story topics, and railway recruitment exam questions\.

For a prompt with scoresr1,…,rnr\_\{1\},\\ldots,r\_\{n\}, letr¯=n−1​∑iri\\bar\{r\}=n^\{\-1\}\\sum\_\{i\}r\_\{i\}ands2=n−1​∑i\(ri−r¯\)2s^\{2\}=n^\{\-1\}\\sum\_\{i\}\(r\_\{i\}\-\\bar\{r\}\)^\{2\}\. The normal Q–Q plot compares the ordered standardized scoresz\(i\)=\(r\(i\)−r¯\)/sz\_\{\(i\)\}=\(r\_\{\(i\)\}\-\\bar\{r\}\)/swith

qi=Φ−1\(i−1/2n\),i=1,…,n,q\_\{i\}=\\Phi^\{\-1\}\\\!\\left\(\\frac\{i\-1/2\}\{n\}\\right\),\\qquad i=1,\\ldots,n,\(26\)whereΦ\\Phiis the standard normal cumulative distribution function\. Gaussian scores would approximately follow the linez=qz=q\.

Figures[3](https://arxiv.org/html/2609.38526#A2.F3)and[4](https://arxiv.org/html/2609.38526#A2.F4)show that the conditional gold\-reward distributions need not be Gaussian\. All four examples have negative empirical skewness, ranging from−0\.75\-0\.75to−0\.22\-0\.22\. The creative\-writing example has the strongest asymmetry, with skewness−0\.75\-0\.75and excess kurtosis1\.191\.19\. Departures from the reference line in the Q–Q plots reveal differences in distributional shape even after matching each prompt’s mean and variance\. Thus, the empirical scaling results arise in a setting whose reward distributions depart from the Gaussian idealization\.

![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/gold_reward_distribution_examples.png)Figure 3:Gold\-reward distributions for four test prompts\.Each histogram contains all 8,192 raw gold scores for one prompt\. The overlaid Gaussian has the same empirical mean and variance\. Panels use the same prompt order as Figure[4](https://arxiv.org/html/2609.38526#A2.F4); prompts are selected by topic and question ID without inspecting rewards\.![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/gold_reward_qq_examples.png)Figure 4:Normal Q–Q plots for the same four prompts\.Ordered gold scores, standardized within each prompt, are plotted against standard normal quantiles at probabilities\(i−1/2\)/8,192\(i\-1/2\)/8\{,\}192\. The diagonal is the Gaussian reference\. Systematic departures, particularly in the tails, illustrate that empirical gold rewards need not follow a Gaussian distribution\.
### B\.3Hyperparameters

We found reward\-model training to work over a broad range of learning rates in preliminary sweeps, from2×10−62\\times 10^\{\-6\}to4×10−54\\times 10^\{\-5\}, consistent with the findings of[Ouyang et al\. \(2022, Appendix C\.2\)](https://arxiv.org/html/2609.38526#bib.bib4)\. We then compared2×10−62\\times 10^\{\-6\}and10−510^\{\-5\}in 75,000\-comparison pilots, selecting by final logistic loss on 512 validation comparisons from 64 held\-out prompts distinct from training and test sets\. All validations selected10−510^\{\-5\}, which we use for the scaling experiments;[Coste et al\. \(2024\)](https://arxiv.org/html/2609.38526#bib.bib15)use the same reward\-model learning rate\. Table[3](https://arxiv.org/html/2609.38526#A2.T3)gives the complete shared settings\.

Table 3:Training hyperparameters\.

## Appendix CAdditional experiments

Figure[5](https://arxiv.org/html/2609.38526#A3.F5)collects the square\-root\-logarithmic fits for all ten settings in Table[1](https://arxiv.org/html/2609.38526#S5.T1)\.

Figure[6](https://arxiv.org/html/2609.38526#A3.F6)shows the best\-of\-NNreward curves for four comparison budgets, using 0\.6B proxies trained on noiseless and noisy preferences\.

![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/all_scaling_fits.png)Figure 5:Square\-root\-logarithmic scaling across all settings in Table[1](https://arxiv.org/html/2609.38526#S5.T1)\.Columns show best\-of\-NN\(BoN\) and exponential tilting \(Tilt\); rows vary proxy size and preference noise\. Each point is the peak mean gold\-reward gain over the evaluated optimization grid\. Lines fita\+b​log⁡Ma\+b\\sqrt\{\\log M\}to ten models betweenM=600M=600and300,000300\{,\}000\.![Refer to caption](https://arxiv.org/html/2609.38526v1/figures/bon_examples_06b_log.png)Figure 6:Best\-of\-NNoptimization with noiseless and noisy preferences\.Each panel shows four 0\.6B checkpoints trained onM=600M=600,4,8004\{,\}800,38,40038\{,\}400, and300,000300\{,\}000comparisons\. TheNN\-axis is logarithmic\. Solid and dashed lines show mean gold\- and proxy\-reward gains over the same evaluation prompts, using the unbiased subset\-average estimator on 8,192 responses per prompt\. Gains are centered atN=1N=1\. Gold rewards use the fixed reference standard deviation, and each proxy uses its own standard deviation over the full evaluation pool\.