Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classification

# BoN-Style Distillation via Rank-Based Classification
Source: [https://arxiv.org/html/2608.19748](https://arxiv.org/html/2608.19748)
## Truncate Bad, Upweight Good: BoN\-Style Distillation via Rank\-Based Classification

Yaniv Romano\[0\.5em\] Department of ElectricalComputer Engineering\[0\.2em\] Department of Computer Science\[0\.3em\] Technion–Israel Institute of Technology\[0\.5em\]yarinbar@campus\.technion\.ac\.ilyromano@technion\.ac\.il

###### Abstract

Inference\-time selection methods, such as Best\-of\-N, improve generation by sampling a pool of candidates and selecting the top\-ranked completion according to a reward model\. Distillation seeks to amortize this procedure into a single policy by replacing raw rewards with in\-pool ranks and learning a policy that upweights higher\-ranked completions\. However, existing rank\-based policies typically use smooth full\-support reweighting, so low\-ranked completions receive less mass but remain in the target support\. Although a sharper reweighting reduces lower\-tail mass, it also increases reliance on brittle ranking at the top made by a single reward model\. We proposeTUP: aTruncate\-bad,Upweight\-goodPolicy that removes low\-ranked completions from the support and reweights only the retained upper tail with a tunable sharpness\. TUP admits a closed\-form, prompt\-independent normalization and can be trained fully offline via binary cross\-entropy, using shifted\-truncated win\-rates as soft labels and distilled\-to\-reference log\-likelihood ratios as logits\. Theoretically, under certain assumptions, we show that for any unknown oracle reward, the best monotone rank\-reweighting can be matched by a lower\-tail truncation rule, providing formal support for removing the lower tail rather than merely downweighting it\. Empirically, we show that TUP is competitive with strong offline alignment baselines\.

## 1Introduction

Inference\-time selection methods such as best\-of\-NN\(BoN\) have emerged as an effective tool for improving language models’ generation quality\[[15](https://arxiv.org/html/2608.19748#bib.bib30),[4](https://arxiv.org/html/2608.19748#bib.bib29)\]\. BoN samples several completions from a base policy, scores them using a reward model, and returns the highest\-scoring one\[[17](https://arxiv.org/html/2608.19748#bib.bib28),[1](https://arxiv.org/html/2608.19748#bib.bib26),[36](https://arxiv.org/html/2608.19748#bib.bib27)\]\. The price is computational, since each prompt requires multiple generated completions and reward evaluations\[[5](https://arxiv.org/html/2608.19748#bib.bib13)\]\. This has motivated a growing line of work on BoN\-style distillation\[[34](https://arxiv.org/html/2608.19748#bib.bib31),[26](https://arxiv.org/html/2608.19748#bib.bib40),[3](https://arxiv.org/html/2608.19748#bib.bib44),[14](https://arxiv.org/html/2608.19748#bib.bib46),[8](https://arxiv.org/html/2608.19748#bib.bib22),[4](https://arxiv.org/html/2608.19748#bib.bib29)\], where rather than carrying out selection during deployment, one trains a single policy to internalize the behavior of the inference\-time selector\.

A key insight from recent BoN\-style distillation work is that the relevant training signal for a completion is not its raw reward in isolation, but its relative rank within the sampled pool\. This relative rank is often referred to as the completion*win\-rate*\. In turn, existing distillation methods aim to maximize the win\-rate while penalizing the KL distance from the base policy\[[26](https://arxiv.org/html/2608.19748#bib.bib40),[3](https://arxiv.org/html/2608.19748#bib.bib44),[14](https://arxiv.org/html/2608.19748#bib.bib46)\]\. This results in policies that apply smooth full\-support reweighting, favoring higher\-ranked completions while keeping every completion in the target support\. Sharper reweighting reduces the probability mass of clearly low\-ranked completions, but it also concentrates more mass on the very top of the reward\-model ranking\. As a result, over\-sharpening can move the policy further from the reference model \(smaller KL\-penalty\), increasing the risk of reward hacking\[[28](https://arxiv.org/html/2608.19748#bib.bib4),[38](https://arxiv.org/html/2608.19748#bib.bib3),[19](https://arxiv.org/html/2608.19748#bib.bib2),[31](https://arxiv.org/html/2608.19748#bib.bib7)\]\. Since reward models are imperfect proxies for unknown oracle or human preferences\[[10](https://arxiv.org/html/2608.19748#bib.bib33),[21](https://arxiv.org/html/2608.19748#bib.bib34),[15](https://arxiv.org/html/2608.19748#bib.bib30),[20](https://arxiv.org/html/2608.19748#bib.bib24),[29](https://arxiv.org/html/2608.19748#bib.bib17),[45](https://arxiv.org/html/2608.19748#bib.bib12),[11](https://arxiv.org/html/2608.19748#bib.bib8),[31](https://arxiv.org/html/2608.19748#bib.bib7)\], recent work shows that coarse preferences tend to persist across different evaluators, while fine\-grained distinctions among top\-ranked completions are less consistently agreed upon\[[23](https://arxiv.org/html/2608.19748#bib.bib38),[43](https://arxiv.org/html/2608.19748#bib.bib41),[24](https://arxiv.org/html/2608.19748#bib.bib23)\]\.

Consequently, we argue that the design of a BoN\-style distillation policy should separate two choices: \(i\) how much of the lower\-ranked tail should be removed from the support, and \(ii\) how sharply the retained higher\-ranked completions should be upweighted\. To this end, we proposeTUP, aTruncate\-bad,Upweight\-goodPolicy for BoN\-style distillation\. TUP decouples these choices with a shifted\-truncated win\-rate transform\. Completions whose win\-rate falls below the threshold are assigned zero mass, while those above it are softly reweighted by their relative ranking\. The threshold parameter controls lower\-tail truncation, whereas the upper\-tail sharpness parameter controls the upweighting strength within the retained set\. By introducing separate parameters for lower\-tail truncation and upper\-tail sharpness, this design preserves the BoN principle of favoring higher\-ranked completions while \(i\) preventing probability mass from being assigned to clearly undesirable completions, and \(ii\) avoiding over\-sharpening due to possible ranking mistakes at the top\.

The resulting target policy remains simple to train from offline data\. Each prompt is paired with a pool of completions sampled from the reference policy and scored by a reward model\. From these scores, we compute empirical in\-pool ranks and convert them into shifted\-truncated win\-rate labels; by subtracting the truncation parameter, we assign the lower\-tail completions an exact zero label\. Pairing with the logit transformation, we then train the distilled language model with binary cross\-entropy \(BCE\) loss\.

##### Contributions\.

In this work, we make three key contributions\.

1. 1\.A truncate\-bad, upweight\-good policy for BoN\-style distillation\.We propose a rank\-based target policy that removes low\-ranked completions using a win\-rate threshold and softly reweights the retained upper tail\.
2. 2\.Theoretical support for lower\-tail truncation and within\-tail upweighting\.Under certain assumptions, we give two theoretical justifications for TUP, using the oracle win\-rate as the performance criterion\. First, we show that the best hard lower\-tail truncation rule can match the best monotone policy that reweights completions by proxy\-reward win\-rate\. Second, after fixing a truncation threshold, we show that smooth upweighting of retained completions can further improve the oracle win\-rate when, informally, higher proxy\-reward win\-rates tend to correspond to higher oracle win\-rates within the retained tail\.
3. 3\.Benchmark results and ablations\.We evaluate TUP on the QRPO benchmark training Llama\-8B Tülu 3 SFT on UltraFeedbak and Magpie Air and Mistral\-7B\-Instruct\-v0\.2 on Magpie Air, comparing to four leading offline\-alignment baseline methods\. We evaluate performance using three different reward models, both on held\-out data and AlpacaEval\[[9](https://arxiv.org/html/2608.19748#bib.bib11)\]\. Full implementation of TUP and the experiments are available at[github\.com/yarinbar/truncate\-bad\-upweight\-good](https://github.com/yarinbar/truncate-bad-upweight-good)\.

## 2Preliminaries and related works

![Refer to caption](https://arxiv.org/html/2608.19748v1/figures/method_ill.png)Figure 1:Illustration of TUP\.We denote a language model as a policyπ⁡\(y∣x\)\\pi\(y\\mid x\)mapping a promptx∈𝒳x\\in\\mathcal\{X\}to a distribution over completionsy∈𝒴y\\in\\mathcal\{Y\}\. Prompts are drawn from a distribution𝒟\\mathcal\{D\}over𝒳\\mathcal\{X\}\. Letπref​\(y∣x\)\\pi\_\{\\text\{ref\}\}\(y\\mid x\)denote the reference policy, e\.g\., obtained via supervised fine\-tuning, and letr⁡\(x,y\)∈ℝr\(x,y\)\\in\\mathbb\{R\}be a reward function\.

The standard post\-training alignment objective seeks a policy that maximizes expected reward while remaining close to the reference policy:

πr,β⋆\(⋅\|x\)=argmaxπ𝔼y∼π\(⋅\|x\)\[r\(x,y\)\]−βDKL\(π\(⋅\|x\)∥πref\(⋅\|x\)\),\\pi\_\{r,\\beta\}^\{\\star\}\(\\cdot\|x\)=\\arg\\max\_\{\\pi\}\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\|x\)\}\[r\(x,y\)\]\-\\beta D\_\{\\text\{KL\}\}\(\\pi\(\\cdot\|x\)\\,\\\|\\,\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\),\(1\)whereβ\>0\\beta\>0controls the reward–KL tradeoff\. This KL\-regularized reward\-maximization objective is a standard formulation in RLHF for language\-model alignment\[[6](https://arxiv.org/html/2608.19748#bib.bib16),[37](https://arxiv.org/html/2608.19748#bib.bib18),[46](https://arxiv.org/html/2608.19748#bib.bib14),[30](https://arxiv.org/html/2608.19748#bib.bib37),[2](https://arxiv.org/html/2608.19748#bib.bib15)\], and is commonly optimized with online reinforcement\-learning methods such as PPO\[[33](https://arxiv.org/html/2608.19748#bib.bib43)\]and GRPO\[[35](https://arxiv.org/html/2608.19748#bib.bib42)\]\.

An alternative paradigm, which we follow in this paper, builds directly on the closed\-form solution to \([1](https://arxiv.org/html/2608.19748#S2.E1)\), known as the Gibbs policy:

πr,β⋆\(y\|x\)=1Z⁡\(x\)πref\(y\|x\)exp\(r⁡\(x,y\)β\),Z\(x\)=𝔼Y∼πref\(⋅\|x\)\[exp\(r⁡\(x,Y\)β\)\],\\pi\_\{r,\\beta\}^\{\\star\}\(y\|x\)=\\frac\{1\}\{Z\(x\)\}\\pi\_\{\\text\{ref\}\}\(y\|x\)\\exp\\left\(\\frac\{r\(x,y\)\}\{\\beta\}\\right\),\\ \\ \\ Z\(x\)=\\mathbb\{E\}\_\{Y\\sim\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\}\\left\[\\exp\\left\(\\frac\{r\(x,Y\)\}\{\\beta\}\\right\)\\right\],\(2\)whereZ⁡\(x\)Z\(x\)is the partition function\. Rearranging \([2](https://arxiv.org/html/2608.19748#S2.E2)\) yields

r⁡\(x,y\)=β​log⁡Z⁡\(x\)\+β​log⁡πr,β⋆​\(y\|x\)πref​\(y\|x\)\.r\(x,y\)=\\beta\\log Z\(x\)\+\\beta\\log\\frac\{\\pi\_\{r,\\beta\}^\{\\star\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\}\.\(3\)This relation suggests a direct fitting strategy\. Replacing the unknown optimal policyπr,β⋆\\pi\_\{r,\\beta\}^\{\\star\}with a parameterized modelπθ\\pi\_\{\\theta\}would allow one to train the policy by regressing the right\-hand side of \([3](https://arxiv.org/html/2608.19748#S2.E3)\) to predict the reward\. The difficulty is that the partition functionZ⁡\(x\)Z\(x\)is prompt\-dependent and impractical to compute\. Preference\-optimization methods avoid the need to computeZ⁡\(x\)Z\(x\)in different but related ways\. DPO\[[32](https://arxiv.org/html/2608.19748#bib.bib45)\], IPO\[[13](https://arxiv.org/html/2608.19748#bib.bib32)\], and SimPO\[[27](https://arxiv.org/html/2608.19748#bib.bib39)\]learn from pairwise or binary preference signals, while REBEL\[[12](https://arxiv.org/html/2608.19748#bib.bib25)\]regresses relative reward differences between completions\.

Although standard post\-training alignment typically treats the raw rewardr⁡\(x,y\)r\(x,y\)as the quantity to optimize, this is not the most natural choice for inference\-time selection algorithms such as BoN\. Recent inference\-aware and rank\-based methods, including InfAlign\[[3](https://arxiv.org/html/2608.19748#bib.bib44)\]and QRPO\[[26](https://arxiv.org/html/2608.19748#bib.bib40)\], motivate replacing raw reward values with relative\-rank quantities\. A natural choice is the population*win\-rate*

wr\(x,y\):=ℙY′∼πref\(⋅\|x\)\(r\(x,Y′\)≤r\(x,y\)\),w\_\{r\}\(x,y\):=\\mathbb\{P\}\_\{Y^\{\\prime\}\\sim\\pi\_\{\\text\{ref\}\}\(\\cdot\|x\)\}\\bigl\(r\(x,Y^\{\\prime\}\)\\leq r\(x,y\)\\bigr\),\(4\)which measures how oftenyybeats an independent reference sample\. In practice, one can estimate the win\-rate from a finite pool of completions\.

This win\-rate parametrization is useful not only because it matches the relative nature of BoN\-style selection, but also because it removes the prompt\-dependent partition function in \([3](https://arxiv.org/html/2608.19748#S2.E3)\)\. WhenY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)and the conditional reward distribution is continuous, the probability integral transform implies thatwr​\(x,Y\)∼U⁡\[0,1\]w\_\{r\}\(x,Y\)\\sim U\[0,1\]\. Therefore, for any nonnegative reweighting functiong:\[0,1\]→\[0,∞\)g:\[0,1\]\\to\[0,\\infty\)with finite integral, the rank\-based policy

πg​\(y∣x\)=1Zg​g​\(wr​\(x,y\)\)​πref​\(y∣x\),Zg=∫01g⁡\(u\)​𝑑u,\\pi\_\{g\}\(y\\mid x\)=\\frac\{1\}\{Z\_\{g\}\}g\(w\_\{r\}\(x,y\)\)\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\),\\qquad Z\_\{g\}=\\int\_\{0\}^\{1\}g\(u\)\\,du,\(5\)has a normalizerZgZ\_\{g\}independent of the promptxx\. Many rank\-based policies follow this construction\. For example, classic Best\-of\-NNinducesgBoN,N​\(w\)∝wN−1g\_\{\\mathrm\{BoN\},N\}\(w\)\\propto w^\{N\-1\}\[[14](https://arxiv.org/html/2608.19748#bib.bib46),[4](https://arxiv.org/html/2608.19748#bib.bib29)\], and the standard QRPO formulation inducesgQRPO,β​\(w\)∝ew/βg\_\{\\mathrm\{QRPO\},\\beta\}\(w\)\\propto e^\{w/\\beta\}\[[26](https://arxiv.org/html/2608.19748#bib.bib40)\]\. Both assign positive weight over the entire support, leaving open how the win\-rate should be transformed into a target\-policy reweighting\.

This full\-support property clarifies how our target differs from related alignment methods\. Iterative BoN\-distillation methods such as BOND\[[34](https://arxiv.org/html/2608.19748#bib.bib31)\]and Faster\-WIND\[[41](https://arxiv.org/html/2608.19748#bib.bib19)\]also aim to approximate or distill Best\-of\-NN\-type behavior, but they inherit the smooth reweighting view rather than defining a hard lower\-tail removal rule\. Other objectives, including RAFT\[[8](https://arxiv.org/html/2608.19748#bib.bib22)\], SLiC\-HF\[[44](https://arxiv.org/html/2608.19748#bib.bib20)\], and RRHF\[[42](https://arxiv.org/html/2608.19748#bib.bib21)\], use reward filtering or relative preference rankings to construct training signals, but they do not specify a prompt\-independent rank\-transform density that first truncates the lower tail and then softly reweights the retained upper tail\. Our work makes this truncation\-and\-reweighting operation explicit, yielding a finite prompt\-independent normalization and a target that can be fit as a soft\-classification problem using the transformed win\-rate as labels\.

## 3Method

Figure 2:Reward model agreement\(left\)and the cost–benefit tradeoff ofλ\\lambda\-truncation\(right\)on UltraFeedback reference completions\. We compare ArmoRM, used as the proxy reward model, against the independent Skywork\-v2\-Llama reward model\.Left:Fraction of completions placed by ArmoRM in the top or bottomppregion that Skywork also places in the same region, as a function ofpp\.Right:Probability thatλ\\lambda\-truncation discards a Skywork\-top\-25%25\\%completion or retains a Skywork\-bottom\-25%25\\%completion\. The crossover marks where these error probabilities are equal\.### 3\.1The proposed truncate\-bad, upweight\-good target policy

Gibbs\-like policies \([5](https://arxiv.org/html/2608.19748#S2.E5)\) that reweightπref\\pi\_\{\\mathrm\{ref\}\}by a smooth monotone function of the proxy win\-rate, such as QRPO\[[26](https://arxiv.org/html/2608.19748#bib.bib40)\], face the sharpness tradeoff we discussed above\.

Figure[2](https://arxiv.org/html/2608.19748#S3.F2)\(left\) provides an empirical illustration motivating our choice to decouple lower\-tail truncation from upper\-tail sharpness\. When the same reference completions are ranked by ArmoRM\[[39](https://arxiv.org/html/2608.19748#bib.bib35)\]and by an independent Skywork\-v2\-Llama reward\[[25](https://arxiv.org/html/2608.19748#bib.bib36)\], the models tend to agree more on the bottom of the pool than on the top\. Thus, removing the lower tail targets a comparatively stable region across reward models, whereas sharpening toward the top places disproportionate trust in fine\-grained top\-rank distinctions with weaker cross\-model agreement\. Additional such comparisons are available in Appendix[B](https://arxiv.org/html/2608.19748#A2)\.

We instantiate this principle by introducing a truncation levelλ∈\(0,1\)\\lambda\\in\(0,1\), which determines whether a completion remains in the target support, and a sharpness parameterβ\>0\\beta\>0, which controls how strongly the retained completions are reweighted\. Givenλ\\lambda, we define the*shifted\-truncated win\-rate*

wλ,r​\(x,y\):=max⁡\(wr​\(x,y\)−λ,0\),w\_\{\\lambda,r\}\(x,y\)\\;:=\\;\\max\\\!\\bigl\(w\_\{r\}\(x,y\)\-\\lambda,\\,0\\bigr\),\(6\)and use it to define the transformed reward

Rλ​\(x,y\):=logit⁡\(wλ,r​\(x,y\)\)=log⁡wλ,r​\(x,y\)1−wλ,r​\(x,y\)\.R\_\{\\lambda\}\(x,y\)\\;:=\\;\\mathrm\{logit\}\\\!\\bigl\(w\_\{\\lambda,r\}\(x,y\)\\bigr\)\\;=\\;\\log\\\!\\frac\{w\_\{\\lambda,r\}\(x,y\)\}\{1\-w\_\{\\lambda,r\}\(x,y\)\}\.\(7\)The truncation levelλ\\lambdaacts as a quality threshold\. Completions withwr​\(x,y\)≤λw\_\{r\}\(x,y\)\\leq\\lambdaare mapped toRλ​\(x,y\)=−∞R\_\{\\lambda\}\(x,y\)=\-\\infty, while completions above the threshold are retained and scored by the log\-odds of their shifted win\-rate\. Thus,λ\\lambdaimposes a hard quality floor, whereas the transformed reward preserves a smooth ordering among the retained completions\. As shown in Figure[2](https://arxiv.org/html/2608.19748#S3.F2)\(right\), increasingλ\\lambdalowers the probability of retaining completions that the independent reward model rank poorly, but also raises the probability of discarding completions this model rank highly\.

We now derive the KL\-regularized target policy induced by the transformed rewardRλ​\(x,y\)R\_\{\\lambda\}\(x,y\)\. The key point is that truncation preserves the tractability of win\-rate\-based objectives\. When the reward distribution is continuous underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\), the population win\-ratewr​\(x,Y\)w\_\{r\}\(x,Y\)is uniform on\[0,1\]\[0,1\]\. After shifting and truncating this variable, the Gibbs normalizer remains independent of the prompt and can be computed in closed form\.

###### Proposition 3\.1\(Closed\-form target under the truncated reward\)\.

Assume that, for the fixed promptxx, the distribution ofr⁡\(x,Y\)r\(x,Y\)underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)is continuous\. Then, for every truncation levelλ∈\(0,1\)\\lambda\\in\(0,1\)and regularization parameterβ\>0\\beta\>0, the KL\-regularized objective in \([1](https://arxiv.org/html/2608.19748#S2.E1)\) with rewardRλR\_\{\\lambda\}admits the unique Gibbs solution

πλ,β⋆​\(y∣x\)=1Zλ,β​πref​\(y∣x\)​\(wλ,r​\(x,y\)1−wλ,r​\(x,y\)\)1/β,\\pi\_\{\\lambda,\\beta\}^\{\\star\}\(y\\mid x\)\\;=\\;\\frac\{1\}\{Z\_\{\\lambda,\\beta\}\}\\,\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\left\(\\frac\{w\_\{\\lambda,r\}\(x,y\)\}\{1\-w\_\{\\lambda,r\}\(x,y\)\}\\right\)^\{1/\\beta\},\(8\)where the normalization constant is

Zλ,β=∫01−λt1/β\(1−t\)−1/βdt=Beta1−λ\(1\+1β,1−1β\)\.Z\_\{\\lambda,\\beta\}\\;=\\;\\int\_\{0\}^\{1\-\\lambda\}t^\{1/\\beta\}\(1\-t\)^\{\-1/\\beta\}\\,dt\\;=\\;\\mathrm\{Beta\}\_\{1\-\\lambda\}\\\!\\bigl\(1\+\\tfrac\{1\}\{\\beta\},\\,1\-\\tfrac\{1\}\{\\beta\}\\bigr\)\.\(9\)Above,Betau​\(a,b\)\\mathrm\{Beta\}\_\{u\}\(a,b\)denotes the incomplete Beta function\.

The proposition follows by substitutingRλR\_\{\\lambda\}into the Gibbs solution in \([2](https://arxiv.org/html/2608.19748#S2.E2)\); Appendix[A\.1](https://arxiv.org/html/2608.19748#A1.SS1)gives the population normalizer calculation\. The result separates support selection from within\-support reweighting, whereλ\\lambdacontrols which completions remain in the support and the sharpening parameterβ\\betacontrols how strongly the target policy departs fromπref\\pi\_\{\\mathrm\{ref\}\}within the retained set\.

Figure 3:Fixed\-threshold illustration for the quadratic un\-normalized oracle\-proxy profilems​\(w\)=1−\(w−s\)2m\_\{s\}\(w\)=1\-\(w\-s\)^\{2\}atλ=1/2\\lambda=1/2\. The x axis is the proxy\-rank locationsswhere oracle utility is maximized, and the y axis shows the unnormalized oracle utility𝒰s\\mathcal\{U\}\_\{s\}\. The curves compare the best QRPO policy, pure truncation, and the best TUP reweighting at the fixed thresholdsupβ∈\(0,∞\]𝒰s​\(1/2,β\)\\sup\_\{\\beta\\in\(0,\\infty\]\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\. The vertical dashed line markssβs\_\{\\beta\}, above which finiteβ\\betaimproves over pure truncation, and the dotted line markssTs\_\{T\}, where pure truncation no longer outperforms the best QRPO curve\.
### 3\.2Theoretical merits of truncate\-bad, upweight\-good policies

Algorithm 1TUP offline BoN distillation with BCE1:Offline prompts

\{xi∼𝒟\}i=1n\\\{x\_\{i\}\\sim\\mathcal\{D\}\\\}\_\{i=1\}^\{n\}, reference policy

πref\\pi\_\{\\mathrm\{ref\}\}, reward model

rr, truncation

λ\\lambda, sharpness

β\\beta
2:For each

xix\_\{i\}, a pool

\{yi,j∼πref\(⋅∣xi\)\}j=1K\\\{y\_\{i,j\}\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\_\{i\}\)\\\}\_\{j=1\}^\{K\}with empirical in\-pool win\-rates

w^r​\(xi,yi,j\)=1\+∑ℓ≠j𝟙\[r\(xi,yi,j\)≥r\(xi,yi,ℓ\)\]K∈\{1K,…,1\}\\hat\{w\}\_\{r\}\(x\_\{i\},y\_\{i,j\}\)=\\frac\{1\+\\sum\_\{\\ell\\neq j\}\\mathds\{1\}\[r\(x\_\{i\},y\_\{i,j\}\)\\geq r\(x\_\{i\},y\_\{i,\\ell\}\)\]\}\{K\}\\in\\left\\\{\\frac\{1\}\{K\},\\dots,1\\right\\\}
3:Precompute truncated win\-rates and bias term:

w^λ,r​\(xi,yi,j\)=max⁡\(w^r​\(xi,yi,j\)−λ,0\),bλ,β=β​log⁡Zλ,β\\hat\{w\}\_\{\\lambda,r\}\(x\_\{i\},y\_\{i,j\}\)=\\max\\\!\\bigl\(\\hat\{w\}\_\{r\}\(x\_\{i\},y\_\{i,j\}\)\-\\lambda,\\,0\\bigr\),\\qquad b\_\{\\lambda,\\beta\}=\\beta\\log Z\_\{\\lambda,\\beta\}
4:foreach training stepdo

5:Sample a minibatch of indexed pool elements

\(xi,yi,j\)\(x\_\{i\},y\_\{i,j\}\)from the reference\-sampled pool

6:Compute the logit and predicted probability:

sθ​\(xi,yi,j\)=β​log⁡πθ​\(yi,j∣xi\)πref​\(yi,j∣xi\)\+bλ,β,pθ​\(xi,yi,j\)=σ⁡\(sθ​\(xi,yi,j\)\)s\_\{\\theta\}\(x\_\{i\},y\_\{i,j\}\)=\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{i,j\}\\mid x\_\{i\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,j\}\\mid x\_\{i\}\)\}\+b\_\{\\lambda,\\beta\},\\qquad p\_\{\\theta\}\(x\_\{i\},y\_\{i,j\}\)=\\sigma\\\!\\bigl\(s\_\{\\theta\}\(x\_\{i\},y\_\{i,j\}\)\\bigr\)
7:Take a gradient step to minimize the BCE loss:

ℒBCE​\(θ\)=−w^λ,r​\(xi,yi,j\)​log⁡pθ​\(xi,yi,j\)−\(1−w^λ,r​\(xi,yi,j\)\)​log⁡\(1−pθ​\(xi,yi,j\)\)\\mathcal\{L\}\_\{\\mathrm\{BCE\}\}\(\\theta\)=\-\\hat\{w\}\_\{\\lambda,r\}\(x\_\{i\},y\_\{i,j\}\)\\log p\_\{\\theta\}\(x\_\{i\},y\_\{i,j\}\)\-\\bigl\(1\-\\hat\{w\}\_\{\\lambda,r\}\(x\_\{i\},y\_\{i,j\}\)\\bigr\)\\log\\\!\\bigl\(1\-p\_\{\\theta\}\(x\_\{i\},y\_\{i,j\}\)\\bigr\)
8:endfor

We now provide a theoretical analysis that sheds light on the roles of truncation and reweighting in our target policy\. To set the stage, fix a promptxx, and letY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\. Recall thatwr​\(x,Y\)w\_\{r\}\(x,Y\)is the win\-rate computed using the proxy rewardr⁡\(x,⋅\)r\(x,\\cdot\)\. Define also the*oracle win\-rate*

wu\(x,Y\):=ℙY′∼πref\(⋅∣x\)\(u\(x,Y′\)≤u\(x,Y\)\),w\_\{u\}\(x,Y\):=\\mathbb\{P\}\_\{Y^\{\\prime\}\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\\\!\\bigl\(u\(x,Y^\{\\prime\}\)\\leq u\(x,Y\)\\bigr\),whereu⁡\(x,⋅\)u\(x,\\cdot\)is an unknown oracle reward function\.

The following analysis builds on the general formulation of rank\-based policiesπg\\pi\_\{g\}from \([5](https://arxiv.org/html/2608.19748#S2.E5)\)\. Notably, our proposed policy also falls under this construction, with the functiongλ,β\(w\)∝\(w−λ1−w\+λ\)1/β𝟙\{w\>λ\}g\_\{\\lambda,\\beta\}\(w\)\\propto\\bigl\(\\frac\{w\-\\lambda\}\{1\-w\+\\lambda\}\\bigr\)^\{1/\\beta\}\\mathds\{1\}\\\{w\>\\lambda\\\}\. With this in place, we can evaluate a candidate policyπg\\pi\_\{g\}using the unknown oracle reward, through the expected oracle win\-rate of completionsYYsampled fromπg\\pi\_\{g\}:

𝒰x\(g\):=𝔼Y∼πg\(⋅∣x\)\[wu\(x,Y\)\]\.\\mathcal\{U\}\_\{x\}\(g\):=\\mathbb\{E\}\_\{Y\\sim\\pi\_\{g\}\(\\cdot\\mid x\)\}\[w\_\{u\}\(x,Y\)\]\.
Our first result shows that, among monotone reweightings of proxy rank, it is enough to consider hard truncation rules\. Letℱ↑\\mathcal\{F\}\_\{\\uparrow\}denote the set of nondecreasing densities on\[0,1\]\[0,1\], and for eachλ∈\[0,1\)\\lambda\\in\[0,1\), letνλ\(w\):=11−λ𝟙\{w∈\[λ,1\]\}\\nu\_\{\\lambda\}\(w\):=\\frac\{1\}\{1\-\\lambda\}\\mathds\{1\}\\\{w\\in\[\\lambda,1\]\\\}denote the uniform density on the upper tail, which is the limitgλ,∞\(w\)∝𝟙\{w\>λ\}g\_\{\\lambda,\\infty\}\(w\)\\propto\\mathds\{1\}\\\{w\>\\lambda\\\}\. With this notation, we state the following result\.

###### Theorem 3\.2\(Informal\)\.

Fix a promptxx, and assume that the proxy rewardr⁡\(x,Y\)r\(x,Y\)is continuous underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\), so that the proxy win\-ratewr​\(x,Y\)w\_\{r\}\(x,Y\)is uniform on\[0,1\]\[0,1\]\. Letℱ↑\\mathcal\{F\}\_\{\\uparrow\}be the class of nonnegative nondecreasing densities on\[0,1\]\[0,1\]\. Then

supf∈ℱ↑𝒰x​\(f\)=supλ∈\[0,1\)𝒰x​\(νλ\)\.\\sup\_\{f\\in\\mathcal\{F\}\_\{\\uparrow\}\}\\mathcal\{U\}\_\{x\}\(f\)=\\sup\_\{\\lambda\\in\[0,1\)\}\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.

The formal statement and proof are given in Appendix[A\.2\.1](https://arxiv.org/html/2608.19748#A1.SS2.SSS1)\. This result implies that the best hard\-threshold rule, obtained by optimizingλ\\lambdafor each promptxx, can match the best rank\-based rule in those families, such as QRPO and BoNBoN\. In practice, tuningλ\\lambdafor each promptxxis infeasible, and thus we treat it as a global quality floor\. To empirically quantify the potential gain of prompt\-adaptive truncation over a fixed global threshold, we consider a simplified, training\-free experiment that treats a strong reward model as an “oracle” evaluator\. We show that the difference is relatively insignificant\. For full details, refer to Section[B\.3](https://arxiv.org/html/2608.19748#A2.SS3)of the Appendix\.

Once the threshold is fixed, we show that tail sharpening can further improve the policy if the proxy win rates within the retained tail is informative about oracle preferences\. To formalize this relationship, define the oracle–proxy profile asmx\(w\):=𝔼Y∼πref\(⋅∣x\)\[wu\(x,Y\)∣wr\(x,Y\)=w\]m\_\{x\}\(w\):=\\mathbb\{E\}\_\{Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\[w\_\{u\}\(x,Y\)\\mid w\_\{r\}\(x,Y\)=w\], the average oracle win rate among completions with proxy win rateww\. In oracle rank space, the following result states that a finiteβ\\betacan improve over pure truncation when this profile is positively aligned with the within\-tail log\-odds tilt\. The formal statement and proof are given in Appendix[A\.2\.2](https://arxiv.org/html/2608.19748#A1.SS2.SSS2)\.

###### Proposition 3\.3\(Informal\)\.

Fix a thresholdλ0∈\(0,1\)\\lambda\_\{0\}\\in\(0,1\)\. If, within the retained tail\[λ0,1\]\[\\lambda\_\{0\},1\], the oracle profilemxm\_\{x\}has positive covariance with the within\-tail log\-odds tiltlog⁡\(\(w−λ0\)/\(1−w\+λ0\)\)\\log\(\(w\-\\lambda\_\{0\}\)/\(1\-w\+\\lambda\_\{0\}\)\), then there exists a finiteβ0∈\(0,∞\)\\beta\_\{0\}\\in\(0,\\infty\)such thatπλ0,β0⋆\\pi^\{\\star\}\_\{\\lambda\_\{0\},\\beta\_\{0\}\}achieves strictly larger expected oracle win\-rate than pure truncation policyπλ0,∞⋆\\pi^\{\\star\}\_\{\\lambda\_\{0\},\\infty\}\.

As with the truncation thresholdλ\\lambda, the theoretically optimal tail\-sharpening parameterβ\\betais prompt\-specific\. In practice, however, we use a single global value ofβ\\betaacross all prompts\. In this sense, Theorem[3\.2](https://arxiv.org/html/2608.19748#S3.Thmtheorem2)and Proposition[3\.3](https://arxiv.org/html/2608.19748#S3.Thmtheorem3)provide idealized theoretical support for separating truncation from sharpness, but they do not directly explain how a training model can benefit from it\.

To demonstrate the effects of thresholdλ\\lambdaand the reweighting effect of a finiteβ\\beta, Appendix[A\.2\.3](https://arxiv.org/html/2608.19748#A1.SS2.SSS3)and Figure[3](https://arxiv.org/html/2608.19748#S3.F3)use a stylized, unnormalized quadratic oracle–proxy example; the appendix shows that normalization does not affect the relevant comparisons\. The improvement of the pure\-truncation policyπλ,∞⋆\\pi^\{\\star\}\_\{\\lambda,\\infty\}over QRPO reflects the gain from using a threshold to remove low\-ranked completions\. The improvement of the best finite\-β\\betaTUP policy overπλ,∞⋆\\pi^\{\\star\}\_\{\\lambda,\\infty\}reflects the additional gain from reweighting within the retained tail once the threshold is fixed\.

### 3\.3Training objective: policy learning as a classification problem

Having established the theoretical properties of our proposed policyπλ,β⋆\\pi\_\{\\lambda,\\beta\}^\{\\star\}, we now turn to formulate the training objective induced by this target\. For completions in the support ofπλ,β⋆\\pi\_\{\\lambda,\\beta\}^\{\\star\}, rearranging \([8](https://arxiv.org/html/2608.19748#S3.E8)\) gives

wλ,r​\(x,y\)=σ⁡\(β​log⁡πλ,β⋆​\(y∣x\)πref​\(y∣x\)\+β​log⁡Zλ,β\),w\_\{\\lambda,r\}\(x,y\)\\;=\\;\\sigma\\left\(\\beta\\log\\frac\{\\pi\_\{\\lambda,\\beta\}^\{\\star\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\+\\beta\\log Z\_\{\\lambda,\\beta\}\\right\),\(10\)whereσ\\sigmais the sigmoid function andlogitis its inverse on\(0,1\)\(0,1\), soσ​\(logit​\(x\)\)=x\\sigma\(\\mathrm\{\\textnormal\{logit\}\}\(x\)\)=xforx∈\(0,1\)x\\in\(0,1\), with endpoint limits00and11asx→0\+x\\to 0^\{\+\}andx→1−x\\to 1^\{\-\}\. The above relation suggests that we can fit a parameterized policyπθ\\pi\_\{\\theta\}that estimatesπλ,β⋆\\pi\_\{\\lambda,\\beta\}^\{\\star\}using logistic regression of the shifted\-truncated win\-rate\. Indeed, we can define our classifier as

pθ​\(x,y\):=σ⁡\(β​log⁡πθ​\(y∣x\)πref​\(y∣x\)\+β​log⁡Zλ,β\),p\_\{\\theta\}\(x,y\)\\;:=\\;\\sigma\\left\(\\beta\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\+\\beta\\log Z\_\{\\lambda,\\beta\}\\right\),\(11\)whereβ​log⁡Zλ,β\\beta\\log Z\_\{\\lambda,\\beta\}acts as a constant and known intercept\. For empirical finite\-pool ranks, the normalizer has an exact finite\-KKanalogue of the same form, with the integral replaced by a discrete average over rank locations\. As in QRPO, our reported experiments use the population intercept, while Appendix[A\.1](https://arxiv.org/html/2608.19748#A1.SS1)gives the finite\-pool alternative\. Empirically, the population and finite\-pool normalizers performs similarly to the analytical normalizer, except at extremely smallβ\\beta, where the finite\-pool form is needed to keep the normalizer finite\.

The construction above naturally results in a fully offline binary cross\-entropy \(BCE\) objective with the shifted\-truncated win\-rate as a soft label:

TheBCE TUP distillationobjective is:ℒ\(θ\)=𝔼x∼𝒟,y∼πref\(⋅∣x\)\[−wλ,r\(x,y\)log\(pθ\(x,y\)\)−\(1−wλ,r\(x,y\)\)log\(1−pθ\(x,y\)\)\]\.\\mathcal\{L\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\\Bigl\[\-w\_\{\\lambda,r\}\(x,y\)\\log\\\!\\bigl\(p\_\{\\theta\}\(x,y\)\\bigr\)\-\(1\-w\_\{\\lambda,r\}\(x,y\)\)\\log\\bigl\(1\-p\_\{\\theta\}\(x,y\)\\bigr\)\\Bigr\]\.\(12\)

This loss requires only precomputed scalar rank labelswλ,r​\(x,y\)w\_\{\\lambda,r\}\(x,y\)and the global interceptβ​log⁡Zλ,β\\beta\\log Z\_\{\\lambda,\\beta\}\. It requires no pairwise preferences, no online sampling, and no runtime estimation of a prompt\-dependent partition function\.

Algorithm[1](https://arxiv.org/html/2608.19748#alg1)summarizes the resulting fully offline training procedure\. In practice, for each prompt we first sample a reference pool, compute each completion’s empirical in\-pool win\-ratew^r​\(x,y\)\\hat\{w\}\_\{r\}\(x,y\)against the otherK−1K\-1completions in that same pool, apply the global truncationλ\\lambda, and then fitπθ\\pi\_\{\\theta\}by binary cross\-entropy on the distilled\-to\-reference log\-likelihood ratio\.

## 4Experiments

Table 1:In\-dataset evaluation for Llama 8B Tülu 3 SFT trained separately on UltraFeedback and Magpie Air\. We report the LC reward metric for each reward model\.Table 2:AlpacaEval performance for Llama 8B Tülu 3 SFT trained on UltraFeedback \(top\) and Magpie Air \(bottom\)\. We report the LC reward metric for each reward model, and win\-rate usinggpt\-4ojudge results; length is measured in characters\. Methods marked “random” are trained on a random pair of completions from each pool, whereas unmarked pairwise methods use the best and worst completions\. TUP is trained on a single random completion per pool\.##### Experimental setup\.

We build on the QRPO experimental framework of[26](https://arxiv.org/html/2608.19748#bib.bib40), which evaluates offline alignment methods across multiple models and datasets\. We compare TUP with six baselines across two model families, two datasets, three reward\-model evaluators, and a GPT judge\. We use UltraFeedback\[[7](https://arxiv.org/html/2608.19748#bib.bib10)\]and Magpie Air\[[40](https://arxiv.org/html/2608.19748#bib.bib9)\]as training datasets\. In our experiments, we apply each alignment method to Llama 8B Tülu 3 SFT\[[22](https://arxiv.org/html/2608.19748#bib.bib6)\]separately on both datasets and to Mistral\-7B\[[18](https://arxiv.org/html/2608.19748#bib.bib1)\]on Magpie Air; for Mistral, we first perform dedicated supervised fine\-tuning \(SFT\) to obtain the initial reference policy\.

Following QRPO, we use their published datasets, which use ArmoRM\[[39](https://arxiv.org/html/2608.19748#bib.bib35)\]as the offline reward model for scoring the reference\-model training completions\. To test generalization beyond the training reward model, we evaluate using two Skywork\-v2 reward models, Skywork\-Reward\-V2\-Llama\-3\.1\-8B and Skywork\-Reward\-V2\-Qwen3\-8B\[[25](https://arxiv.org/html/2608.19748#bib.bib36)\], referred to as Skywork\-Llama and Skywork\-Qwen respectively\. These models are ranked \#1 and \#2, respectively, among available classifier reward models on the RewardBench leaderboard at the time of writing\[[23](https://arxiv.org/html/2608.19748#bib.bib38)\]\.111[https://huggingface\.co/spaces/allenai/reward\-bench](https://huggingface.co/spaces/allenai/reward-bench)We additionally evaluate on AlpacaEval usinggpt\-4oas a judge\[[45](https://arxiv.org/html/2608.19748#bib.bib12),[16](https://arxiv.org/html/2608.19748#bib.bib5)\]\. Table[8](https://arxiv.org/html/2608.19748#A3.T8)of the Appendix reports the RewardBench standings of all reward and judge models in our experiments\. We report length\-controlled \(LC\) rewards in the main tables; raw reward results are given in Appendix[B\.1](https://arxiv.org/html/2608.19748#A2.SS1)\.

The most natural baselines for our setting are rank\-based distillation methods\. We include BoNBoN\[[14](https://arxiv.org/html/2608.19748#bib.bib46)\]and the QRPO variant that computes the loss with random completions\[[26](https://arxiv.org/html/2608.19748#bib.bib40)\], denoted QRPO \(random\)\. While less related, we also compare to strong offline\-alignment baselines using pairwise or relative\-reward signals, namely DPO\[[32](https://arxiv.org/html/2608.19748#bib.bib45)\]and REBEL\[[12](https://arxiv.org/html/2608.19748#bib.bib25)\]\. Unless marked random, DPO and REBEL use the best–worst pair from each pool; REBEL \(random\) uses random pairs\. We also include QRPO trained on best–worst pairs\. For all DPO, REBEL, and QRPO variants, we adopt the best\-performing configurations from the*off\-policy*setting reported in[26](https://arxiv.org/html/2608.19748#bib.bib40)\. To avoid clutter, tables omit the best–worst label when it is the default\. We do not include SimPO\[[27](https://arxiv.org/html/2608.19748#bib.bib39)\], whose length\-normalized objective makes it structurally different from baselines that do not explicitly account for length\. Because BoNBoN is not covered by the QRPO benchmark, we run a dedicated hyperparameter search and select the best checkpoint by validation reward\. Full implementation details are provided in Appendix[C](https://arxiv.org/html/2608.19748#A3)\.

In all tables, unless stated otherwise, we show TUP with 3 different global truncation values;λ=0\.2\\lambda=0\.2\(mild\) which truncates only the worst rollout out of the 6 completions,λ=0\.5\\lambda=0\.5\(mid\.\) andλ=0\.8\\lambda=0\.8\(aggressive\) that keeps only the top two completions\.

Table 3:LC rewards for Mistral 7B SFT, trained on Magpie Air\. Tested on in\-dataset test split \(MA\) and AlpacaEval \(AE\) evaluated by ArmoRM, Skywork\-Llama and Skywork\-Qwen reward models, with AlpacaEval also judged bygpt\-4o\.
##### Results\.

Table[1](https://arxiv.org/html/2608.19748#S4.T1)reports two in\-dataset experiments on Llama 8B, where each method is evaluated on the held\-out split of its respective training dataset, UltraFeedback\[[7](https://arxiv.org/html/2608.19748#bib.bib10)\]or Magpie Air\[[40](https://arxiv.org/html/2608.19748#bib.bib9)\]\. As shown, TUP is competitive with strong offline alignment baselines on both datasets\. Under ArmoRM, all methods achieve broadly similar results\. Larger differences appear under the two Skywork reward models\. On both UltraFeedback and Magpie Air, TUP obtains the best in\-dataset results under both Skywork\-Llama and Skywork\-Qwen\. If these reward models are viewed as different proxies for the oracle reward, the results suggest that TUP generalizes well across proxy rewards compared to the baselines considered\.

Table[2](https://arxiv.org/html/2608.19748#S4.T2)reports AlpacaEval results for the above mentioned trained Llama models\. Under both datasets, TUP obtains the best results under both Skywork\-Llama and Skywork\-Qwen, including on AlpacaEval LC reward, and remains highly competitive under the GPT judge\. While the Magpie\-Air\-trained models on AlpacaEval results are more mixed, TUP remains competitive\. The supplementary raw reward tables in Appendix[B\.1](https://arxiv.org/html/2608.19748#A2.SS1)provide the corresponding non\-LC scores and in\-dataset response lengths, showing that the main trends are not limited to the LC metric\. Table[3](https://arxiv.org/html/2608.19748#S4.T3), which reports both in\-dataset and AlpacaEval LC rewards, shows that TUP remains competitive when applied to the Mistral model family\.

In addition to the LC\-reward results reported above, we conduct a pairwise length\-matched reward comparison to further isolate TUP’s reward gains from the effect of its longer average responses\. As reported in Table[7](https://arxiv.org/html/2608.19748#A2.T7)of the Appendix, TUP responses are largely preferred against similar\-length responses from the baseline methods on the same prompt\. This provides additional evidence that our advantage does not stem from response length alone\. Full details are in Section[B\.2](https://arxiv.org/html/2608.19748#A2.SS2)of the Appendix\.

Figure[4](https://arxiv.org/html/2608.19748#S4.F4)shows the effects of TUP’s truncation thresholdλ\\lambdaand preference sharpnessβ\\beta, using the same validation\-based checkpoint\-selection protocol as the main experiments \(Tables[1](https://arxiv.org/html/2608.19748#S4.T1)–[5](https://arxiv.org/html/2608.19748#A2.T5)\)\. To approximate the no\-truncation setting, we useλ=0\.2\\lambda=0\.2, which removes only the lowest\-ranked of the six responses, a response that would otherwise receive an extremely low weight\. To approximate “pure” truncation, we useβ=100\\beta=100, which produces an approximately uniform weighting over the retained tail while still remaining finite and allowing for some deviation fromπref\\pi\_\{\\text\{ref\}\}\. Generally, intermediate values of bothλ\\lambdaandβ\\betatend to yield the highest LC rewards\. These results suggest that the best performance comes from combining moderate truncation with non\-uniform within\-tail reweighting, supporting our choice to treatλ\\lambdaandβ\\betaas separate parameters\.

Figure 4:Ablation study of TUP’s truncation thresholdλ\\lambdaand sharpnessβ\\betafor Llama 8B Tülu 3 SFT trained on Magpie Air and evaluated on AlpacaEval with Skywork\-Llama reward model\. TUP usesK=6K=6reference completions per prompt\. Cells report the mean and sample standard deviation across three generation seeds\.Left:LC rewards; higher is better\.Right:Mean response length in tokens\.![Refer to caption](https://arxiv.org/html/2608.19748v1/llama_magpieair_beta_lambda_grid_transposed_skywork_v2_llama_alpaca_eval.png)

## 5Discussion

One limitation of our method is that bothλ\\lambdaandβ\\betamust be tuned based on validation performance, whereas other baselines tune onlyβ\\beta\. This may increase the computational cost when a suitable initial range forλ\\lambdais unknown\. A practical way to narrow this range is to construct a plot such as the right panel of Figure[2](https://arxiv.org/html/2608.19748#S3.F2)\. Given a dataset, a proxy reward model, and an auxiliary reward model treated as an “oracle,” this plot provides an efficient way to identify a promising range of truncation levels\. Once a truncation level that retains responses ranked highly by the “oracle” while removing poorly ranked ones is identified, the remaining hyperparameter search is once again only onβ\\betavalues\.

TUP also leaves several directions for future work\. First, although TUP improves rank\-based policy distillation, it is still exposed to reward hacking; incorporating reward\-hacking mitigation methods could further improve its performance\. Second, future work could replace the fixed globalλ\\lambdaandβ\\betawith potentially learned prompt\-specific parameters\.

More broadly, TUP may support safety alignment when unsafe or otherwise undesirable completions are consistently assigned low proxy ranks, allowing truncation to remove them from the target support rather than merely down\-weight them\.

## References

- \[1\]Z\. Ankner, M\. Paul, B\. Cui, J\. D\. Chang, and P\. Ammanabrolu\(2024\)Critique\-out\-Loud Reward Models\.InPluralistic Alignment Workshop at NeurIPS,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1)\.
- \[2\]Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[3\]A\. Balashankar, Z\. Sun, J\. Berant, J\. Eisenstein, M\. Collins, A\. Hutter, J\. Lee, C\. Nagpal, F\. Prost, and A\. Sinha\(2025\)InfAlign: Inference\-aware language model alignment\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§1](https://arxiv.org/html/2608.19748#S1.p2.1),[§2](https://arxiv.org/html/2608.19748#S2.p4.1)\.
- \[4\]A\. Beirami, A\. Agarwal, J\. Berant, A\. D’Amour, J\. Eisenstein, C\. Nagpal, and A\. T\. Suresh\(2025\)Theoretical guarantees on the best\-of\-N alignment policy\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§2](https://arxiv.org/html/2608.19748#S2.p5.2)\.
- \[5\]B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. Mirhoseini\(2024\)Large language monkeys: scaling inference compute with repeated sampling\.arXiv preprint arXiv:2407\.21787\.Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1)\.
- \[6\]P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei\(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[7\]G\. Cui, L\. Yuan, N\. Ding, G\. Yao, W\. Zhu, Y\. Ni, G\. Xie, Z\. Liu, and M\. Sun\(2023\)Ultrafeedback: boosting language models with high\-quality feedback\.Cited by:[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px2.p1.1)\.
- \[8\]H\. Dong, W\. Xiong, D\. Goyal, Y\. Zhang, W\. Chow, R\. Pan, S\. Diao, J\. Zhang, K\. Shum, and T\. Zhang\(2023\)RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§2](https://arxiv.org/html/2608.19748#S2.p6.1)\.
- \[9\]Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto\(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[item 3](https://arxiv.org/html/2608.19748#S1.I1.i3.p1.1)\.
- \[10\]E\. Frick, T\. Li, C\. Chen, W\. Chiang, A\. N\. Angelopoulos, J\. Jiao, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica\(2025\)How to Evaluate Reward Models for RLHF\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[11\]L\. Gao, J\. Schulman, and J\. Hilton\(2023\)Scaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[12\]Z\. Gao, J\. D\. Chang, W\. Zhan, O\. Oertell, G\. Swamy, K\. Brantley, T\. Joachims, J\. A\. Bagnell, J\. Lee, and W\. Sun\(2024\)REBEL: Reinforcement Learning via Regressing Relative Rewards\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p3.3),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p3.1)\.
- \[13\]S\. Garg, A\. Singh, S\. Singh, and P\. Chopra\(2025\)IPO: Your Language Model is Secretly a Preference Classifier\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p3.3)\.
- \[14\]L\. Gui, C\. Gârbacea, and V\. Veitch\(2024\)BoNBoN alignment for large language models and the sweetness of best\-of\-n sampling\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§1](https://arxiv.org/html/2608.19748#S1.p2.1),[§2](https://arxiv.org/html/2608.19748#S2.p5.2),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p3.1)\.
- \[15\]A\. Huang, A\. Block, Q\. Liu, N\. Jiang, A\. Krishnamurthy, and D\. J\. Foster\(2025\)Is Best\-of\-N the Best of Them? Coverage, Scaling, and Optimality in Inference\-Time Alignment\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[16\]A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p2.1)\.
- \[17\]H\. Ivison, Y\. Wang, J\. Liu, Z\. Wu, V\. Pyatkin, N\. Lambert, N\. A\. Smith, Y\. Choi, and H\. Hajishirzi\(2024\)Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1)\.
- \[18\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p1.1)\.
- \[19\]Y\. Jinnai, T\. Morimura, K\. Ariu, and K\. Abe\(2024\)Regularized best\-of\-n sampling to mitigate reward hacking for language model alignment\.InICML 2024 Workshop on Models of Human Feedback for AI Alignment,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[20\]H\. Khalaf, C\. M\. Verdun, A\. Oesterling, H\. Lakkaraju, and F\. d\. P\. Calmon\(2025\)Inference\-Time Reward Hacking in Large Language Models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[21\]S\. Kim, D\. Kang, T\. Kwon, H\. Chae, D\. Lee, and J\. Yeo\(2025\)Rethinking reward model evaluation through the lens of reward overoptimization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[22\]N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, X\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. Hajishirzi\(2025\)Tulu 3: pushing frontiers in open language model post\-training\.InSecond Conference on Language Modeling,Cited by:[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p1.1)\.
- \[23\]N\. Lambert, V\. Pyatkin, J\. Morrison, L\. J\. V\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, and Y\. Choi\(2025\)RewardBench: Evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p2.1)\.
- \[24\]E\. Landesberg\(2026\)When LLM judge scores look good but best\-of\-n decisions fail\.arXiv preprint arXiv:2603\.12520\.Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[25\]C\. Y\. Liu, L\. Zeng, Y\. Xiao, J\. He, J\. Liu, C\. Wang, R\. Yan, W\. Shen, F\. Zhang, and J\. Xu\(2026\)Skywork\-reward\-v2: Scaling preference data curation via human\-ai synergy\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§3\.1](https://arxiv.org/html/2608.19748#S3.SS1.p2.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p2.1)\.
- \[26\]S\. Matrenok, S\. Moalla, and C\. Gulcehre\(2025\)Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions\.InAdvances in Neural Information Processing Systems,Cited by:[§C\.1](https://arxiv.org/html/2608.19748#A3.SS1.p6.1),[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§1](https://arxiv.org/html/2608.19748#S1.p2.1),[§2](https://arxiv.org/html/2608.19748#S2.p4.1),[§2](https://arxiv.org/html/2608.19748#S2.p5.2),[§3\.1](https://arxiv.org/html/2608.19748#S3.SS1.p1.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p3.1)\.
- \[27\]Y\. Meng, M\. Xia, and D\. Chen\(2024\)SimPO: simple preference optimization with a reference\-free reward\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p3.3),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p3.1)\.
- \[28\]E\. J\. Michaud, A\. Gleave, and S\. Russell\(2020\)Understanding learned reward functions\.arXiv preprint arXiv:2012\.05862\.Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[29\]R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders,et al\.\(2021\)Webgpt: browser\-assisted question\-answering with human feedback\.arXiv preprint arXiv:2112\.09332\.Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[30\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, and A\. Ray\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[31\]R\. Rafailov, Y\. Chittepu, R\. Park, H\. Sikchi, J\. Hejna, W\. B\. Knox, C\. Finn, and S\. Niekum\(2024\)Rafailov laws for reward model overoptimization in direct alignment algorithms\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[32\]R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p3.3),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p3.1)\.
- \[33\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[34\]P\. G\. Sessa, R\. Dadashi\-Tazehozi, L\. Hussenot, J\. Ferret, N\. Vieillard, A\. Rame, B\. Shahriari, S\. Perrin, A\. L\. Friesen, and G\. Cideron\(2025\)BOND: Aligning LLMs with Best\-of\-N Distillation\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1),[§2](https://arxiv.org/html/2608.19748#S2.p6.1)\.
- \[35\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[36\]Y\. Song, G\. Wang, S\. Li, and B\. Y\. Lin\(2025\)The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non\-Determinism\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p1.1)\.
- \[37\]N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano\(2020\)Learning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.
- \[38\]J\. Tien, J\. Z\. He, Z\. Erickson, A\. Dragan, and D\. S\. Brown\(2023\)Causal confusion and reward misidentification in preference\-based reward learning\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[39\]H\. Wang, W\. Xiong, T\. Xie, H\. Zhao, and T\. Zhang\(2024\)Interpretable preferences via multi\-objective reward modeling and mixture\-of\-experts\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Cited by:[§3\.1](https://arxiv.org/html/2608.19748#S3.SS1.p2.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p2.1)\.
- \[40\]Z\. Xu, F\. Jiang, L\. Niu, Y\. Deng, R\. Poovendran, Y\. Choi, and B\. Y\. Lin\(2025\)Magpie: alignment data synthesis from scratch by prompting aligned LLMs with nothing\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px2.p1.1)\.
- \[41\]T\. Yang, J\. Mei, H\. Dai, Z\. Wen, S\. Cen, D\. Schuurmans, Y\. Chi, and B\. Dai\(2025\)Faster wind: accelerating iterative best\-of\-n distillation for LLM alignment\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p6.1)\.
- \[42\]H\. Yuan, Z\. Yuan, C\. Tan, W\. Wang, S\. Huang, and F\. Huang\(2023\)RRHF: Rank Responses to Align Language Models with Human Feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p6.1)\.
- \[43\]J\. Zhang, Z\. Wang, L\. Gui, S\. M\. Sathyendra, J\. Jeong, V\. Veitch, W\. Wang, Y\. He, B\. Liu, and L\. Jin\(2026\)Chasing the Tail: Effective Rubric\-based Reward Modeling for Large Language Model Post\-Training\.InThe Fifteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1)\.
- \[44\]Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu\(2023\)SLIC\-HF: sequence likelihood calibration with human feedback\.arXiv preprint arXiv:2305\.10425\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p6.1)\.
- \[45\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica\(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2608.19748#S1.p2.1),[§4](https://arxiv.org/html/2608.19748#S4.SS0.SSS0.Px1.p2.1)\.
- \[46\]D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving\(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§2](https://arxiv.org/html/2608.19748#S2.p2.2)\.

## Appendix ATheory and proofs

### A\.1Normalization constant for Proposition[3\.1](https://arxiv.org/html/2608.19748#S3.Thmtheorem1)

###### Proof\.

By the Gibbs solution in \([2](https://arxiv.org/html/2608.19748#S2.E2)\), substitutingr=Rλr=R\_\{\\lambda\}yields

π⋆​\(y∣x\)=1Z⁡\(x\)​πref​\(y∣x\)​exp⁡\(Rλ​\(x,y\)β\)=1Z⁡\(x\)​πref​\(y∣x\)​\(wλ,r​\(x,y\)1−wλ,r​\(x,y\)\)1/β\.\\pi^\{\\star\}\(y\\mid x\)\\;=\\;\\frac\{1\}\{Z\(x\)\}\\,\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{R\_\{\\lambda\}\(x,y\)\}\{\\beta\}\\right\)\\;=\\;\\frac\{1\}\{Z\(x\)\}\\,\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\left\(\\frac\{w\_\{\\lambda,r\}\(x,y\)\}\{1\-w\_\{\\lambda,r\}\(x,y\)\}\\right\)^\{1/\\beta\}\.It therefore remains to identify the normalizerZ⁡\(x\)Z\(x\)and show that it is independent ofxx\. Under the induced uniform win\-rate variablew∼Uniform⁡\[0,1\]w\\sim\\mathrm\{Uniform\}\[0,1\], the normalization term becomes

Z⁡\(x\)=∫01\(max⁡\(w−λ,0\)1−max⁡\(w−λ,0\)\)1/β​𝑑w\.Z\(x\)\\;=\\;\\int\_\{0\}^\{1\}\\left\(\\frac\{\\max\(w\-\\lambda,\\,0\)\}\{1\-\\max\(w\-\\lambda,\\,0\)\}\\right\)^\{\\\!1/\\beta\}dw\.\(13\)Forw≤λw\\leq\\lambda, the shifted win\-rate is zero, so the integrand vanishes\. Therefore,

Z⁡\(x\)=∫λ1\(w−λ1−\(w−λ\)\)1/β​𝑑w\.Z\(x\)\\;=\\;\\int\_\{\\lambda\}^\{1\}\\left\(\\frac\{w\-\\lambda\}\{1\-\(w\-\\lambda\)\}\\right\)^\{\\\!1/\\beta\}dw\.\(14\)Applying the change of variablesu:=w−λu:=w\-\\lambdagives

Z\(x\)=∫01−λ\(u1−u\)1/βdu=∫01−λu1/β\(1−u\)−1/βdu\.Z\(x\)\\;=\\;\\int\_\{0\}^\{1\-\\lambda\}\\left\(\\frac\{u\}\{1\-u\}\\right\)^\{\\\!1/\\beta\}du\\;=\\;\\int\_\{0\}^\{1\-\\lambda\}u^\{\\,1/\\beta\}\(1\-u\)^\{\-1/\\beta\}\\,du\.\(15\)
Recalling that the incomplete Beta function is defined in Proposition[3\.1](https://arxiv.org/html/2608.19748#S3.Thmtheorem1)asBz​\(a,b\)=∫0zta−1​\(1−t\)b−1​𝑑tB\_\{z\}\(a,b\)=\\int\_\{0\}^\{z\}t^\{a\-1\}\(1\-t\)^\{b\-1\}\\,dt, we identify

a=1\+1β,b=1−1β\.a=1\+\\tfrac\{1\}\{\\beta\},\\qquad b=1\-\\tfrac\{1\}\{\\beta\}\.Hence

Z⁡\(x\)=∫01−λu\(1\+1/β\)−1​\(1−u\)\(1−1/β\)−1​𝑑u=Beta1−λ​\(1\+1β,1−1β\)=Zλ,β,Z\(x\)\\;=\\;\\int\_\{0\}^\{1\-\\lambda\}u^\{\\,\(1\+1/\\beta\)\-1\}\(1\-u\)^\{\\,\(1\-1/\\beta\)\-1\}\\,du\\;=\\;\\mathrm\{Beta\}\_\{1\-\\lambda\}\\\!\\Bigl\(1\+\\tfrac\{1\}\{\\beta\},\\,1\-\\tfrac\{1\}\{\\beta\}\\Bigr\)\\;=\\;Z\_\{\\lambda,\\beta\},\(16\)which is independent ofxx\. ∎

##### Finite\-pool normalizer\.

The closed\-form normalizer in Proposition[3\.1](https://arxiv.org/html/2608.19748#S3.Thmtheorem1)is the population normalizer induced by the continuous win\-ratewr​\(x,Y\)w\_\{r\}\(x,Y\)\. In the experiments, labels are computed from finite prompt\-specific pools\. Under the empirical\-rank convention in Algorithm[1](https://arxiv.org/html/2608.19748#alg1), the corresponding discrete prompt\-independent normalizer is

ZK,λ,β=1K​∑j=1K\(\(jK−λ\)\+1−\(jK−λ\)\+\)1/β\.Z\_\{K,\\lambda,\\beta\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\left\(\\frac\{\(\\frac\{j\}\{K\}\-\\lambda\)\_\{\+\}\}\{1\-\(\\frac\{j\}\{K\}\-\\lambda\)\_\{\+\}\}\\right\)^\{1/\\beta\}\.This finite\-pool form can replaceZλ,βZ\_\{\\lambda,\\beta\}in the BCE intercept without changing the labels, the truncation support, or the loss\. In our experiments, we use the population normalizerZλ,βZ\_\{\\lambda,\\beta\}, following the continuous win\-rate normalization used in QRPO; in our experiments, replacing it with the finite\-pool expression led to negligible performance differences\. In extremely small\-β\\betaregimes, where the continuous calculation can become numerically unstable, the discrete expression above provides the exact finite\-KKalternative for this empirical\-rank convention\.

### A\.2Formal oracle rank\-space results for Section[3\.2](https://arxiv.org/html/2608.19748#S3.SS2)

This appendix formalizes the oracle rank\-space discussion from Section[3\.2](https://arxiv.org/html/2608.19748#S3.SS2)\. The aim is to separate two decisions that smooth full\-support reweighting conflates: which proxy ranks should remain in the support at all, and how the retained ranks should be reweighted\. The first result shows that, among monotone proxy\-rank reweightings, the oracle\-best rule is always a hard lower\-tail truncation\. The second result is a fixed\-threshold refinement statement: once a truncation level has been chosen, finite within\-tail reweighting can improve over pure truncation when oracle value remains positively aligned with proxy rank inside the retained tail\. We close with the quadratic worked example behind Figure[3](https://arxiv.org/html/2608.19748#S3.F3)\.

Fix a promptxx, and letY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\. Define the proxy and oracle population win\-rates by

wr\(x,y\):=ℙY′∼πref\(⋅∣x\)\(r\(x,Y′\)≤r\(x,y\)\),wu\(x,y\):=ℙY′∼πref\(⋅∣x\)\(u\(x,Y′\)≤u\(x,y\)\)\.w\_\{r\}\(x,y\):=\\mathbb\{P\}\_\{Y^\{\\prime\}\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\\\!\\bigl\(r\(x,Y^\{\\prime\}\)\\leq r\(x,y\)\\bigr\),\\qquad w\_\{u\}\(x,y\):=\\mathbb\{P\}\_\{Y^\{\\prime\}\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\\\!\\bigl\(u\(x,Y^\{\\prime\}\)\\leq u\(x,y\)\\bigr\)\.and write

Wr:=wr​\(x,Y\),Wu:=wu​\(x,Y\)\.W\_\{r\}:=w\_\{r\}\(x,Y\),\\qquad W\_\{u\}:=w\_\{u\}\(x,Y\)\.Since BoN\-style methods act on relative rank rather than raw reward, the natural policy class in this analysis reweights a completion according to its proxy win\-rate\. For any measurableg:\[0,1\]→\[0,∞\)g:\[0,1\]\\to\[0,\\infty\)satisfying

0<𝔼πref​\[g⁡\(Wr\)\]<∞,0<\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[g\(W\_\{r\}\)\]<\\infty,define

πg\(y∣x\):=g⁡\(wr​\(x,y\)\)​πref​\(y∣x\)𝔼Y′∼πref\(⋅∣x\)\[g\(wr\(x,Y′\)\)\],𝒰x\(g\):=𝔼Y∼πg\(⋅∣x\)\[Wu\]\.\\pi\_\{g\}\(y\\mid x\):=\\frac\{g\(w\_\{r\}\(x,y\)\)\\,\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\{\\mathbb\{E\}\_\{Y^\{\\prime\}\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\}\[g\(w\_\{r\}\(x,Y^\{\\prime\}\)\)\]\},\\qquad\\mathcal\{U\}\_\{x\}\(g\):=\\mathbb\{E\}\_\{Y\\sim\\pi\_\{g\}\(\\cdot\\mid x\)\}\[W\_\{u\}\]\.Finally, let

mx​\(w\):=𝔼⁡\[Wu∣Wr=w\]m\_\{x\}\(w\):=\\mathbb\{E\}\[W\_\{u\}\\mid W\_\{r\}=w\]denote a measurable version of the conditional oracle profile\. Once the problem is reduced to rank space,mxm\_\{x\}is the only prompt\-specific object that matters\.

#### A\.2\.1Adaptive truncation among monotone reweightings

We first isolate the support\-selection question of which proxy ranks should receive positive mass at all\.

###### Lemma A\.1\(Exact reduction to proxy\-rank space\)\.

For every measurableggas above,

𝒰x​\(g\)=𝔼πref​\[mx​\(Wr\)​g​\(Wr\)\]𝔼πref​\[g⁡\(Wr\)\]\.\\mathcal\{U\}\_\{x\}\(g\)=\\frac\{\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[m\_\{x\}\(W\_\{r\}\)g\(W\_\{r\}\)\]\}\{\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[g\(W\_\{r\}\)\]\}\.If moreover the law ofr⁡\(x,Y\)r\(x,Y\)underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)is continuous, thenWr∼Unif⁡\[0,1\]W\_\{r\}\\sim\\mathrm\{Unif\}\[0,1\], and therefore

𝒰x​\(g\)=∫01mx​\(w\)​g​\(w\)​𝑑w∫01g⁡\(w\)​𝑑w\.\\mathcal\{U\}\_\{x\}\(g\)=\\frac\{\\int\_\{0\}^\{1\}m\_\{x\}\(w\)g\(w\)\\,dw\}\{\\int\_\{0\}^\{1\}g\(w\)\\,dw\}\.

###### Proof\.

By definition ofπg\\pi\_\{g\},

𝒰x​\(g\)=𝔼πref​\[Wu​g​\(Wr\)\]𝔼πref​\[g⁡\(Wr\)\]\.\\mathcal\{U\}\_\{x\}\(g\)=\\frac\{\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[W\_\{u\}g\(W\_\{r\}\)\]\}\{\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[g\(W\_\{r\}\)\]\}\.Applying conditional expectation with respect toWrW\_\{r\}gives

𝔼πref​\[Wu​g​\(Wr\)\]=𝔼πref​\[𝔼⁡\[Wu​g​\(Wr\)∣Wr\]\]=𝔼πref​\[g⁡\(Wr\)​𝔼​\[Wu∣Wr\]\]=𝔼πref​\[mx​\(Wr\)​g​\(Wr\)\]\.\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[W\_\{u\}g\(W\_\{r\}\)\]=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\\\!\\left\[\\mathbb\{E\}\[W\_\{u\}g\(W\_\{r\}\)\\mid W\_\{r\}\]\\right\]=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\\\!\\left\[g\(W\_\{r\}\)\\mathbb\{E\}\[W\_\{u\}\\mid W\_\{r\}\]\\right\]=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}\[m\_\{x\}\(W\_\{r\}\)g\(W\_\{r\}\)\]\.This proves the first identity\. If the law ofr⁡\(x,Y\)r\(x,Y\)is continuous, the probability integral transform yieldsWr∼Unif⁡\[0,1\]W\_\{r\}\\sim\\mathrm\{Unif\}\[0,1\], and the second identity follows by writing expectations as Lebesgue integrals on\[0,1\]\[0,1\]\. ∎

Under the continuity assumption, the problem becomes a pure rank\-space optimization over densities on\[0,1\]\[0,1\]\. Let

ℱ↑:=\{f:\[0,1\]→\[0,∞\):∫01f\(w\)dw=1,fis nondecreasing\},\\mathcal\{F\}\_\{\\uparrow\}:=\\left\\\{f:\[0,1\]\\to\[0,\\infty\):\\int\_\{0\}^\{1\}f\(w\)\\,dw=1,\\ \\ f\\text\{ is nondecreasing\}\\right\\\},and for each thresholdλ∈\[0,1\)\\lambda\\in\[0,1\), define the lower\-tail truncation density

νλ\(w\):=11−λ𝟙\{w∈\[λ,1\]\}\.\\nu\_\{\\lambda\}\(w\):=\\frac\{1\}\{1\-\\lambda\}\\mathds\{1\}\\\{w\\in\[\\lambda,1\]\\\}\.
The next lemma shows that every monotone density is a mixture of these top\-interval uniforms\. This is the structural reason hard truncation is enough\.

###### Lemma A\.2\(Mixture representation of nondecreasing densities\)\.

Letf∈ℱ↑f\\in\\mathcal\{F\}\_\{\\uparrow\}\. Then there exists a probability measureξ\\xion\[0,1\)\[0,1\)such that

f⁡\(w\)=∫\[0,1\)νλ​\(w\)​ξ​\(𝑑λ\)for a\.e\.​w∈\[0,1\]\.f\(w\)=\\int\_\{\[0,1\)\}\\nu\_\{\\lambda\}\(w\)\\,\\xi\(d\\lambda\)\\qquad\\text\{for a\.e\. \}w\\in\[0,1\]\.

###### Proof\.

Define a finite Stieltjes measure on\[0,1\)\[0,1\)by

ξ:=f⁡\(0\)​δ0\+\(1−t\)​d​f​\(t\)\.\\xi:=f\(0\)\\delta\_\{0\}\+\(1\-t\)\\,df\(t\)\.Sinceffis nondecreasing and∫01f⁡\(w\)​𝑑w=1\\int\_\{0\}^\{1\}f\(w\)\\,dw=1, integration by parts gives

ξ⁡\(\[0,1\)\)=f⁡\(0\)\+∫\(0,1\)\(1−t\)​𝑑f​\(t\)=1\.\\xi\(\[0,1\)\)=f\(0\)\+\\int\_\{\(0,1\)\}\(1\-t\)\\,df\(t\)=1\.Thusξ\\xiis a probability measure\. For a\.e\.w∈\[0,1\]w\\in\[0,1\],

∫\[0,1\)νλ​\(w\)​ξ​\(𝑑λ\)=∫\[0,w\]11−λ​ξ​\(𝑑λ\)=f⁡\(0\)\+∫\(0,w\]d​f​\(λ\)=f⁡\(w\)\.\\int\_\{\[0,1\)\}\\nu\_\{\\lambda\}\(w\)\\,\\xi\(d\\lambda\)=\\int\_\{\[0,w\]\}\\frac\{1\}\{1\-\\lambda\}\\,\\xi\(d\\lambda\)=f\(0\)\+\\int\_\{\(0,w\]\}df\(\\lambda\)=f\(w\)\.∎

###### Theorem A\.3\(Hard truncation is optimal among monotone reweightings\)\.

Assume the law ofr⁡\(x,Y\)r\(x,Y\)underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)is continuous\. Then

supf∈ℱ↑𝒰x​\(f\)=supλ∈\[0,1\)𝒰x​\(νλ\)\.\\sup\_\{f\\in\\mathcal\{F\}\_\{\\uparrow\}\}\\mathcal\{U\}\_\{x\}\(f\)=\\sup\_\{\\lambda\\in\[0,1\)\}\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.Equivalently, in the oracle rank\-space view, optimizing over monotone proxy\-rank reweightings reduces to optimizing over hard lower\-tail truncation rules\.

###### Proof\.

By Lemma[A\.1](https://arxiv.org/html/2608.19748#A1.Thmtheorem1), the continuity assumption implies

𝒰x​\(f\)=∫01mx​\(w\)​f​\(w\)​𝑑wfor every​f∈ℱ↑\.\\mathcal\{U\}\_\{x\}\(f\)=\\int\_\{0\}^\{1\}m\_\{x\}\(w\)f\(w\)\\,dw\\qquad\\text\{for every \}f\\in\\mathcal\{F\}\_\{\\uparrow\}\.Fixf∈ℱ↑f\\in\\mathcal\{F\}\_\{\\uparrow\}\. By Lemma[A\.2](https://arxiv.org/html/2608.19748#A1.Thmtheorem2),

f⁡\(w\)=∫\[0,1\)νλ​\(w\)​ξ​\(𝑑λ\)for a\.e\.​w,f\(w\)=\\int\_\{\[0,1\)\}\\nu\_\{\\lambda\}\(w\)\\,\\xi\(d\\lambda\)\\qquad\\text\{for a\.e\. \}w,for some probability measureξ\\xion\[0,1\)\[0,1\)\. Therefore,

𝒰x​\(f\)=∫01mx​\(w\)​\(∫\[0,1\)νλ​\(w\)​ξ​\(𝑑λ\)\)​𝑑w\.\\mathcal\{U\}\_\{x\}\(f\)=\\int\_\{0\}^\{1\}m\_\{x\}\(w\)\\left\(\\int\_\{\[0,1\)\}\\nu\_\{\\lambda\}\(w\)\\,\\xi\(d\\lambda\)\\right\)dw\.By Fubini’s theorem,

𝒰x​\(f\)=∫\[0,1\)\(∫01mx​\(w\)​νλ​\(w\)​𝑑w\)​ξ​\(𝑑λ\)≤supλ∈\[0,1\)𝒰x​\(νλ\)\.\\mathcal\{U\}\_\{x\}\(f\)=\\int\_\{\[0,1\)\}\\left\(\\int\_\{0\}^\{1\}m\_\{x\}\(w\)\\nu\_\{\\lambda\}\(w\)\\,dw\\right\)\\xi\(d\\lambda\)\\leq\\sup\_\{\\lambda\\in\[0,1\)\}\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.Taking the supremum overf∈ℱ↑f\\in\\mathcal\{F\}\_\{\\uparrow\}gives

supf∈ℱ↑𝒰x​\(f\)≤supλ∈\[0,1\)𝒰x​\(νλ\)\.\\sup\_\{f\\in\\mathcal\{F\}\_\{\\uparrow\}\}\\mathcal\{U\}\_\{x\}\(f\)\\leq\\sup\_\{\\lambda\\in\[0,1\)\}\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.The reverse inequality is immediate because eachνλ\\nu\_\{\\lambda\}belongs toℱ↑\\mathcal\{F\}\_\{\\uparrow\}\. ∎

Theorem[A\.3](https://arxiv.org/html/2608.19748#A1.Thmtheorem3)is an oracle statement, so it does not identify a single global threshold for practice\. Its role is structural: in rank space, support restriction is the right family for the first decision, namely which proxy ranks to keep\.

###### Corollary A\.4\(QRPO comparison in the same rank\-space model\)\.

Under the assumptions of Theorem[A\.3](https://arxiv.org/html/2608.19748#A1.Thmtheorem3), let

qτ​\(w\):=τ​eτ​weτ−1,τ\>0,q\_\{\\tau\}\(w\):=\\frac\{\\tau e^\{\\tau w\}\}\{e^\{\\tau\}\-1\},\\qquad\\tau\>0,withq0​\(w\)≡1q\_\{0\}\(w\)\\equiv 1\. Then

supτ≥0𝒰x​\(qτ\)≤supλ∈\[0,1\)𝒰x​\(νλ\)\.\\sup\_\{\\tau\\geq 0\}\\mathcal\{U\}\_\{x\}\(q\_\{\\tau\}\)\\leq\\sup\_\{\\lambda\\in\[0,1\)\}\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.

###### Proof\.

Eachqτq\_\{\\tau\}is nondecreasing on\[0,1\]\[0,1\], so\{qτ:τ≥0\}⊆ℱ↑\\\{q\_\{\\tau\}:\\tau\\geq 0\\\}\\subseteq\\mathcal\{F\}\_\{\\uparrow\}\. The claim follows immediately from Theorem[A\.3](https://arxiv.org/html/2608.19748#A1.Thmtheorem3)\. ∎

Thus, within the same oracle rank\-space model, the best hard truncation rule weakly dominates the best QRPO\-style full\-support monotone tilt\.

#### A\.2\.2Within\-tail reweighting after truncation

We now hold the truncation level fixed and show that, when oracle value remains positively aligned with proxy rank inside the retained tail, uniform weighting over the retained interval is locally suboptimal, so a finite within\-tail tilt can improve over pure truncation at the same threshold\.

For a fixed thresholdλ∈\(0,1\)\\lambda\\in\(0,1\)and parameterβ∈\(0,∞\)\\beta\\in\(0,\\infty\), define the truncated\-soft family

fλ,β\(w\):=1Zλ​\(β\)\(w−λ1−w\+λ\)1/β𝟙\{w∈\(λ,1\)\},f\_\{\\lambda,\\beta\}\(w\):=\\frac\{1\}\{Z\_\{\\lambda\}\(\\beta\)\}\\left\(\\frac\{w\-\\lambda\}\{1\-w\+\\lambda\}\\right\)^\{1/\\beta\}\\mathds\{1\}\\\{w\\in\(\\lambda,1\)\\\},where

Zλ​\(β\):=∫λ1\(w−λ1−w\+λ\)1/β​𝑑w\.Z\_\{\\lambda\}\(\\beta\):=\\int\_\{\\lambda\}^\{1\}\\left\(\\frac\{w\-\\lambda\}\{1\-w\+\\lambda\}\\right\)^\{1/\\beta\}dw\.This is the rank\-space form induced by the shifted\-truncated odds\-ratio tilt at thresholdλ\\lambda\. We use the conventionfλ,∞=νλf\_\{\\lambda,\\infty\}=\\nu\_\{\\lambda\}, soβ=∞\\beta=\\inftyrecovers hard truncation, while finiteβ\\betatilts mass toward larger retained proxy ranks without changing the support\. Define

𝒰x​\(λ,β\):=∫λ1mx​\(w\)​fλ,β​\(w\)​𝑑w,𝒰x​\(λ,∞\):=𝒰x​\(νλ\)\.\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\):=\\int\_\{\\lambda\}^\{1\}m\_\{x\}\(w\)f\_\{\\lambda,\\beta\}\(w\)\\,dw,\\qquad\\mathcal\{U\}\_\{x\}\(\\lambda,\\infty\):=\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.
###### Proposition A\.5\(Local criterion for improving over pure truncation\)\.

Assume the law ofr⁡\(x,Y\)r\(x,Y\)underY∼πref\(⋅∣x\)Y\\sim\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)is continuous, and fixλ∈\(0,1\)\\lambda\\in\(0,1\)\. Let

hλ​\(w\):=log⁡\(w−λ1−w\+λ\),w∈\(λ,1\)\.h\_\{\\lambda\}\(w\):=\\log\\\!\\left\(\\frac\{w\-\\lambda\}\{1\-w\+\\lambda\}\\right\),\\qquad w\\in\(\\lambda,1\)\.Then

∂∂\(β−1\)​𝒰x​\(λ,β\)\|β=∞=CovWr∼Unif⁡\[λ,1\]⁡\(mx​\(Wr\),hλ​\(Wr\)\)\.\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\)\\right\|\_\{\\beta=\\infty\}=\\operatorname\{Cov\}\_\{W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\}\\\!\\bigl\(m\_\{x\}\(W\_\{r\}\),\\,h\_\{\\lambda\}\(W\_\{r\}\)\\bigr\)\.In particular, if this covariance is positive, then there exists a finiteβ\\betasuch that

𝒰x​\(λ,β\)\>𝒰x​\(λ,∞\)=𝒰x​\(νλ\)\.\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\)\>\\mathcal\{U\}\_\{x\}\(\\lambda,\\infty\)=\\mathcal\{U\}\_\{x\}\(\\nu\_\{\\lambda\}\)\.

###### Proof\.

Letfλ,∞=νλf\_\{\\lambda,\\infty\}=\\nu\_\{\\lambda\}denote the uniform density on\[λ,1\]\[\\lambda,1\]\. Then

fλ,β​\(w\)=ehλ​\(w\)/β​fλ,∞​\(w\)𝔼Wr∼Unif⁡\[λ,1\]​\[ehλ​\(Wr\)/β\]\.f\_\{\\lambda,\\beta\}\(w\)=\\frac\{e^\{h\_\{\\lambda\}\(w\)/\\beta\}f\_\{\\lambda,\\infty\}\(w\)\}\{\\mathbb\{E\}\_\{W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\}\[e^\{h\_\{\\lambda\}\(W\_\{r\}\)/\\beta\}\]\}\.Therefore,

𝒰x​\(λ,β\)=𝔼Wr∼Unif⁡\[λ,1\]​\[mx​\(Wr\)​ehλ​\(Wr\)/β\]𝔼Wr∼Unif⁡\[λ,1\]​\[ehλ​\(Wr\)/β\]\.\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\)=\\frac\{\\mathbb\{E\}\_\{W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\}\\\!\\left\[m\_\{x\}\(W\_\{r\}\)e^\{h\_\{\\lambda\}\(W\_\{r\}\)/\\beta\}\\right\]\}\{\\mathbb\{E\}\_\{W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\}\\\!\\left\[e^\{h\_\{\\lambda\}\(W\_\{r\}\)/\\beta\}\\right\]\}\.Differentiating with respect toβ−1\\beta^\{\-1\}atβ=∞\\beta=\\inftygives

∂∂\(β−1\)​𝒰x​\(λ,β\)\|β=∞=𝔼⁡\[mx​\(Wr\)​hλ​\(Wr\)\]−𝔼⁡\[mx​\(Wr\)\]​𝔼​\[hλ​\(Wr\)\],\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\)\\right\|\_\{\\beta=\\infty\}=\\mathbb\{E\}\[m\_\{x\}\(W\_\{r\}\)h\_\{\\lambda\}\(W\_\{r\}\)\]\-\\mathbb\{E\}\[m\_\{x\}\(W\_\{r\}\)\]\\,\\mathbb\{E\}\[h\_\{\\lambda\}\(W\_\{r\}\)\],where all expectations are underWr∼Unif⁡\[λ,1\]W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\. The differentiation may be passed under the expectation becausehλh\_\{\\lambda\}is integrable on\(λ,1\)\(\\lambda,1\)under the uniform law\. This is exactly

CovWr∼Unif⁡\[λ,1\]⁡\(mx​\(Wr\),hλ​\(Wr\)\)\.\\operatorname\{Cov\}\_\{W\_\{r\}\\sim\\mathrm\{Unif\}\[\\lambda,1\]\}\\\!\\bigl\(m\_\{x\}\(W\_\{r\}\),\\,h\_\{\\lambda\}\(W\_\{r\}\)\\bigr\)\.If this covariance is positive, then the derivative atβ=∞\\beta=\\inftyin theβ−1\\beta^\{\-1\}direction is positive, so by continuity there exists a finiteβ\\betasuch that

𝒰x​\(λ,β\)\>𝒰x​\(λ,∞\)\.\\mathcal\{U\}\_\{x\}\(\\lambda,\\beta\)\>\\mathcal\{U\}\_\{x\}\(\\lambda,\\infty\)\.∎

This is a fixed\-λ\\lambdarefinement statement, not a global dominance claim\. It does not say that finite\-β\\betareweighting beats the best threshold whenλ\\lambdais optimized freely\. Instead, once a global quality floor has been chosen, the covariance criterion identifies when a softer within\-tail tilt extracts additional oracle value from the retained tail\.

#### A\.2\.3Quadratic illustration underlying Figure[3](https://arxiv.org/html/2608.19748#S3.F3)

This subsection gives a minimal analytic example that separates the two effects shown in Figure[3](https://arxiv.org/html/2608.19748#S3.F3): changing the retained support and reweighting points within a fixed retained tail\. Consider the stylized oracle\-utility shape

ms​\(w\):=1−\(w−s\)2,s∈\(0,1\),m\_\{s\}\(w\):=1\-\(w\-s\)^\{2\},\\qquad s\\in\(0,1\),wheressis the proxy\-rank location at which oracle value is maximized\. We usemsm\_\{s\}only as an unnormalized analytic shape\. It is not itself a realizable conditional oracle win\-rate profile, since

∫01ms​\(w\)​𝑑w=23\+s−s2,\\int\_\{0\}^\{1\}m\_\{s\}\(w\)\\,dw=\\frac\{2\}\{3\}\+s\-s^\{2\},which is generally not1/21/2\. A normalized profile with the same shape is

m¯s​\(w\):=12\+12​\(ms​\(w\)−\(23\+s−s2\)\)\.\\bar\{m\}\_\{s\}\(w\):=\\frac\{1\}\{2\}\+\\frac\{1\}\{2\}\\left\(m\_\{s\}\(w\)\-\\left\(\\frac\{2\}\{3\}\+s\-s^\{2\}\\right\)\\right\)\.Then∫01m¯s​\(w\)​𝑑w=1/2\\int\_\{0\}^\{1\}\\bar\{m\}\_\{s\}\(w\)\\,dw=1/2, andm¯s​\(w\)∈\[1/6,2/3\]\\bar\{m\}\_\{s\}\(w\)\\in\[1/6,2/3\]for alls,w∈\[0,1\]s,w\\in\[0,1\]\. For any densityffon\[0,1\]\[0,1\],

𝒰¯s​\(f\):=∫01m¯s​\(w\)​f​\(w\)​𝑑w=12\+12​\(𝒰s​\(f\)−\(23\+s−s2\)\),\\bar\{\\mathcal\{U\}\}\_\{s\}\(f\):=\\int\_\{0\}^\{1\}\\bar\{m\}\_\{s\}\(w\)f\(w\)\\,dw=\\frac\{1\}\{2\}\+\\frac\{1\}\{2\}\\left\(\\mathcal\{U\}\_\{s\}\(f\)\-\\left\(\\frac\{2\}\{3\}\+s\-s^\{2\}\\right\)\\right\),where

𝒰s​\(f\):=∫01ms​\(w\)​f​\(w\)​𝑑w\.\\mathcal\{U\}\_\{s\}\(f\):=\\int\_\{0\}^\{1\}m\_\{s\}\(w\)f\(w\)\\,dw\.Thus, for each fixedss,𝒰¯s\\bar\{\\mathcal\{U\}\}\_\{s\}is a positive affine transformation of𝒰s\\mathcal\{U\}\_\{s\}\. All maximizers, strict comparisons, crossing locations, and the thresholdsβs\_\{\\beta\}below are unchanged by this normalization\. We therefore use the shorter unnormalized formmsm\_\{s\}in the algebra\.

For the QRPO\-style retained\-tail family, write

𝒰s​\(λ,β\):=∫λ1ms​\(w\)​fλ,β​\(w\)​𝑑w,𝒰s​\(λ,∞\):=𝒰s​\(νλ\)\.\\mathcal\{U\}\_\{s\}\(\\lambda,\\beta\):=\\int\_\{\\lambda\}^\{1\}m\_\{s\}\(w\)f\_\{\\lambda,\\beta\}\(w\)\\,dw,\\qquad\\mathcal\{U\}\_\{s\}\(\\lambda,\\infty\):=\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda\}\)\.The first proposition isolates the value of adaptive support restriction by comparing the best hard truncation rule with the best full\-support QRPO tilt\. The second proposition fixesλ=1/2\\lambda=1/2, matching Figure[3](https://arxiv.org/html/2608.19748#S3.F3), and asks when finite within\-tail reweighting improves over pure truncation at that same threshold\.

###### Proposition A\.6\(Adaptive truncation beats best QRPO under the quadratic profile\)\.

For everys\>1/2s\>1/2, the hard truncation threshold

λ⋆​\(s\):=3​s−12\\lambda^\{\\star\}\(s\):=\\frac\{3s\-1\}\{2\}satisfies

𝒰s​\(νλ⋆​\(s\)\)=1−\(1−s\)24\>supτ≥0𝒰s​\(qτ\)\.\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda^\{\\star\}\(s\)\}\)=1\-\\frac\{\(1\-s\)^\{2\}\}\{4\}\>\\sup\_\{\\tau\\geq 0\}\\mathcal\{U\}\_\{s\}\(q\_\{\\tau\}\)\.

###### Proof\.

Sincems​\(w\)=1−\(w−s\)2m\_\{s\}\(w\)=1\-\(w\-s\)^\{2\}, maximizing𝒰s​\(f\)\\mathcal\{U\}\_\{s\}\(f\)is equivalent to minimizing

R⁡\(f,s\):=𝔼Wr∼f​\[\(Wr−s\)2\]\.R\(f;s\):=\\mathbb\{E\}\_\{W\_\{r\}\\sim f\}\\\!\\left\[\(W\_\{r\}\-s\)^\{2\}\\right\]\.For the top\-interval uniform lawνλ\\nu\_\{\\lambda\}, we have

𝔼νλ​\[Wr\]=1\+λ2,Varνλ⁡\(Wr\)=\(1−λ\)212\.\\mathbb\{E\}\_\{\\nu\_\{\\lambda\}\}\[W\_\{r\}\]=\\frac\{1\+\\lambda\}\{2\},\\qquad\\operatorname\{Var\}\_\{\\nu\_\{\\lambda\}\}\(W\_\{r\}\)=\\frac\{\(1\-\\lambda\)^\{2\}\}\{12\}\.Therefore

R⁡\(νλ,s\)=\(1\+λ2−s\)2\+\(1−λ\)212\.R\(\\nu\_\{\\lambda\};s\)=\\left\(\\frac\{1\+\\lambda\}\{2\}\-s\\right\)^\{2\}\+\\frac\{\(1\-\\lambda\)^\{2\}\}\{12\}\.Differentiating with respect toλ\\lambdagives

dd​λ​R​\(νλ,s\)=2​λ3\+13−s\.\\frac\{d\}\{d\\lambda\}R\(\\nu\_\{\\lambda\};s\)=\\frac\{2\\lambda\}\{3\}\+\\frac\{1\}\{3\}\-s\.Thus the unique minimizer overλ∈\[0,1\]\\lambda\\in\[0,1\]is

λ⋆​\(s\)=3​s−12,\\lambda^\{\\star\}\(s\)=\\frac\{3s\-1\}\{2\},which lies in\(0,1\)\(0,1\)fors∈\(1/2,1\)s\\in\(1/2,1\)\. Substituting this value gives

R⁡\(νλ⋆​\(s\),s\)=\(1−s\)24,𝒰s​\(νλ⋆​\(s\)\)=1−\(1−s\)24\.R\(\\nu\_\{\\lambda^\{\\star\}\(s\)\};s\)=\\frac\{\(1\-s\)^\{2\}\}\{4\},\\qquad\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda^\{\\star\}\(s\)\}\)=1\-\\frac\{\(1\-s\)^\{2\}\}\{4\}\.
By Lemma[A\.2](https://arxiv.org/html/2608.19748#A1.Thmtheorem2), everyf∈ℱ↑f\\in\\mathcal\{F\}\_\{\\uparrow\}is a mixture of the top\-interval uniform lawsνλ\\nu\_\{\\lambda\}\. Sinceλ↦𝒰s​\(νλ\)\\lambda\\mapsto\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda\}\)has the unique maximizerλ⋆​\(s\)\\lambda^\{\\star\}\(s\), the unique maximizer overℱ↑\\mathcal\{F\}\_\{\\uparrow\}isνλ⋆​\(s\)\\nu\_\{\\lambda^\{\\star\}\(s\)\}\. Hence

supτ≥0𝒰s​\(qτ\)≤𝒰s​\(νλ⋆​\(s\)\)\.\\sup\_\{\\tau\\geq 0\}\\mathcal\{U\}\_\{s\}\(q\_\{\\tau\}\)\\leq\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda^\{\\star\}\(s\)\}\)\.
It remains only to rule out equality\. No finite\-τ\\tauQRPO density equalsνλ⋆​\(s\)\\nu\_\{\\lambda^\{\\star\}\(s\)\}:q0q\_\{0\}is uniform on\[0,1\]\[0,1\], whileqτq\_\{\\tau\}is strictly positive on all of\[0,1\]\[0,1\]for everyτ\>0\\tau\>0, whereasνλ⋆​\(s\)\\nu\_\{\\lambda^\{\\star\}\(s\)\}has zero density belowλ⋆​\(s\)\>0\\lambda^\{\\star\}\(s\)\>0\. Moreover, sincemsm\_\{s\}is bounded and continuous andqτq\_\{\\tau\}converges weakly to a point mass atw=1w=1asτ→∞\\tau\\to\\infty,

limτ→∞𝒰s​\(qτ\)=ms​\(1\)=1−\(1−s\)2\.\\lim\_\{\\tau\\to\\infty\}\\mathcal\{U\}\_\{s\}\(q\_\{\\tau\}\)=m\_\{s\}\(1\)=1\-\(1\-s\)^\{2\}\.This limiting value is strictly smaller than

1−\(1−s\)24=𝒰s​\(νλ⋆​\(s\)\)\.1\-\\frac\{\(1\-s\)^\{2\}\}\{4\}=\\mathcal\{U\}\_\{s\}\(\\nu\_\{\\lambda^\{\\star\}\(s\)\}\)\.Therefore neither a finite QRPO tilt nor its limiting point mass atw=1w=1attains the hard\-truncation optimum, and the desired strict inequality follows\. ∎

This comparison isolates the gain from support restriction itself\. Even in the quadratic example, a full\-support QRPO tilt cannot recover the oracle value achieved by the best hard truncation rule\.

###### Proposition A\.7\(Atλ=1/2\\lambda=1/2, finiteβ\\betahelps above an explicit threshold\)\.

Define

sβ:=1112−log⁡21−log⁡2≈0\.7284\.s\_\{\\beta\}:=\\frac\{\\frac\{11\}\{12\}\-\\log 2\}\{1\-\\log 2\}\\approx 0\.7284\.Then

∂∂\(β−1\)​𝒰s​\(1/2,β\)\|β=∞=\(1−log⁡2\)​s−\(1112−log⁡2\)\.\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\\right\|\_\{\\beta=\\infty\}=\(1\-\\log 2\)s\-\\left\(\\frac\{11\}\{12\}\-\\log 2\\right\)\.In particular, ifs\>sβs\>s\_\{\\beta\}, then there exists a finiteβ\\betasuch that

𝒰s​\(1/2,β\)\>𝒰s​\(1/2,∞\)=𝒰s​\(ν1/2\)\.\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\>\\mathcal\{U\}\_\{s\}\(1/2,\\infty\)=\\mathcal\{U\}\_\{s\}\(\\nu\_\{1/2\}\)\.

###### Proof\.

We instantiate Proposition[A\.5](https://arxiv.org/html/2608.19748#A1.Thmtheorem5)at the fixed thresholdλ=1/2\\lambda=1/2\. On the retained interval\[1/2,1\]\[1/2,1\], the local log\-odds tilt is

h⁡\(w\):=log⁡\(w−1232−w\),w∈\(1/2,1\)\.h\(w\):=\\log\\\!\\left\(\\frac\{w\-\\frac\{1\}\{2\}\}\{\\frac\{3\}\{2\}\-w\}\\right\),\\qquad w\\in\(1/2,1\)\.Proposition[A\.5](https://arxiv.org/html/2608.19748#A1.Thmtheorem5)gives

∂∂\(β−1\)​𝒰s​\(1/2,β\)\|β=∞=CovWr∼Unif⁡\[1/2,1\]⁡\(ms​\(Wr\),h⁡\(Wr\)\)\.\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\\right\|\_\{\\beta=\\infty\}=\\operatorname\{Cov\}\_\{W\_\{r\}\\sim\\operatorname\{Unif\}\[1/2,1\]\}\\\!\\bigl\(m\_\{s\}\(W\_\{r\}\),h\(W\_\{r\}\)\\bigr\)\.Sincems​\(Wr\)=1−\(Wr−s\)2m\_\{s\}\(W\_\{r\}\)=1\-\(W\_\{r\}\-s\)^\{2\}, this is

∂∂\(β−1\)​𝒰s​\(1/2,β\)\|β=∞=−CovWr∼Unif⁡\[1/2,1\]⁡\(\(Wr−s\)2,h⁡\(Wr\)\)\.\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\\right\|\_\{\\beta=\\infty\}=\-\\operatorname\{Cov\}\_\{W\_\{r\}\\sim\\operatorname\{Unif\}\[1/2,1\]\}\\\!\\bigl\(\(W\_\{r\}\-s\)^\{2\},h\(W\_\{r\}\)\\bigr\)\.
The required covariance is explicit\. UnderWr∼Unif⁡\[1/2,1\]W\_\{r\}\\sim\\operatorname\{Unif\}\[1/2,1\],

𝔼⁡\[h⁡\(Wr\)\]=2​∫1/21log⁡\(w−1232−w\)​𝑑w=−log⁡4,\\mathbb\{E\}\[h\(W\_\{r\}\)\]=2\\int\_\{1/2\}^\{1\}\\log\\\!\\left\(\\frac\{w\-\\frac\{1\}\{2\}\}\{\\frac\{3\}\{2\}\-w\}\\right\)dw=\-\\log 4,and

𝔼⁡\[\(Wr−s\)2\]=\(s−34\)2\+148=s2−32​s\+712\.\\mathbb\{E\}\[\(W\_\{r\}\-s\)^\{2\}\]=\\left\(s\-\\frac\{3\}\{4\}\\right\)^\{2\}\+\\frac\{1\}\{48\}=s^\{2\}\-\\frac\{3\}\{2\}s\+\\frac\{7\}\{12\}\.A direct integration also gives

𝔼⁡\[\(Wr−s\)2​h​\(Wr\)\]=1112−s\+\(−2​s2\+4​s−136\)​log⁡2\.\\mathbb\{E\}\[\(W\_\{r\}\-s\)^\{2\}h\(W\_\{r\}\)\]=\\frac\{11\}\{12\}\-s\+\\left\(\-2s^\{2\}\+4s\-\\frac\{13\}\{6\}\\right\)\\log 2\.Subtracting the product of the first two moments yields

Cov⁡\(\(Wr−s\)2,h⁡\(Wr\)\)=1112−log⁡2−\(1−log⁡2\)​s\.\\operatorname\{Cov\}\\\!\\bigl\(\(W\_\{r\}\-s\)^\{2\},h\(W\_\{r\}\)\\bigr\)=\\frac\{11\}\{12\}\-\\log 2\-\(1\-\\log 2\)s\.Therefore

∂∂\(β−1\)​𝒰s​\(1/2,β\)\|β=∞=\(1−log⁡2\)​s−\(1112−log⁡2\)\.\\left\.\\frac\{\\partial\}\{\\partial\(\\beta^\{\-1\}\)\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\\right\|\_\{\\beta=\\infty\}=\(1\-\\log 2\)s\-\\left\(\\frac\{11\}\{12\}\-\\log 2\\right\)\.The thresholdsβs\_\{\\beta\}is the unique zero of this affine function\. Hence, for everys\>sβs\>s\_\{\\beta\}, the derivative in theβ−1\\beta^\{\-1\}direction is positive atβ=∞\\beta=\\infty, so for sufficiently small positiveβ−1\\beta^\{\-1\}, equivalently for sufficiently large finiteβ\\beta, we have

𝒰s​\(1/2,β\)\>𝒰s​\(1/2,∞\)\.\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)\>\\mathcal\{U\}\_\{s\}\(1/2,\\infty\)\.∎

Figure 5:Optimal finite sharpness in the fixed\-threshold quadratic illustration\. Forλ=1/2\\lambda=1/2andms​\(w\)=1−\(w−s\)2m\_\{s\}\(w\)=1\-\(w\-s\)^\{2\}, we numerically maximize𝒰s​\(1/2,β\)\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)over finiteβ\\betafor each oracle peak locations\>sβs\>s\_\{\\beta\}\. The dashed vertical line markssβ≈0\.7284s\_\{\\beta\}\\approx 0\.7284, below which pure truncation is locally optimal and the optimizer isβ=∞\\beta=\\infty\. Just abovesβs\_\{\\beta\}, the optimal finite tilt is arbitrarily weak, soβ⋆​\(s\)\\beta^\{\\star\}\(s\)is large; asssapproaches one, the optimal policy concentrates more strongly near the top proxy ranks, andβ⋆​\(s\)\\beta^\{\\star\}\(s\)decreases\.The proposition identifies the point at which pure truncation becomes locally suboptimal, but it does not assign a single universal value ofβ\\beta\. The optimal value depends on the oracle peak locationss\. To compute it, writeα=1/β\\alpha=1/\\betaand view𝒰s​\(1/2,β\)\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)as a one\-dimensional function ofα≥0\\alpha\\geq 0\. Differentiating the tilted expectation gives

∂∂α​𝒰s​\(1/2,β\)=Covf1/2,β⁡\(ms​\(W\),log⁡\(W−1232−W\)\)\.\\frac\{\\partial\}\{\\partial\\alpha\}\\mathcal\{U\}\_\{s\}\(1/2,\\beta\)=\\operatorname\{Cov\}\_\{f\_\{1/2,\\beta\}\}\\left\(m\_\{s\}\(W\),\\log\\left\(\\frac\{W\-\\frac\{1\}\{2\}\}\{\\frac\{3\}\{2\}\-W\}\\right\)\\right\)\.Thus an interior optimumβ⋆​\(s\)\\beta^\{\\star\}\(s\)is obtained by solving this scalar covariance equation, while fors≤sβs\\leq s\_\{\\beta\}the optimum remains the boundary valueβ=∞\\beta=\\infty, corresponding to pure truncation\. Figure[5](https://arxiv.org/html/2608.19748#A1.F5)plots the resulting finite optimizer fors\>sβs\>s\_\{\\beta\}:β⋆​\(s\)\\beta^\{\\star\}\(s\)diverges asssapproachessβs\_\{\\beta\}from above, because the beneficial tilt is infinitesimal at the threshold, and decreases asssapproaches one, where stronger concentration near the top proxy ranks becomes optimal\. When the proxy is highly reliables→1s\\to 1and thusβ⋆→0\\beta^\{\\star\}\\to 0\.

## Appendix BAdditional experiments and results

### B\.1Raw rewards

Table 4:Raw reward for Llama 8B Tülu 3 SFT on UltraFeedback eval split \(UF\) and AlpacaEval \(AE\), with reporting token length\.Table 5:Raw reward for Llama 8B Tülu 3 SFT on Magpie Air eval split \(MA\) and AlpacaEval \(AE\), with reporting token length\.Table 6:Raw reward for Mistral 7B SFT on Magpie Air eval split \(MA\) and AlpacaEval \(AE\), with reporting token length\.Tables[4](https://arxiv.org/html/2608.19748#A2.T4)and[5](https://arxiv.org/html/2608.19748#A2.T5)report the corresponding raw reward scores and generated lengths\. On UltraFeedback, TUP remains strongest under both independent Skywork reward models, including on AlpacaEval\. On Magpie Air, TUP remains competitive with the strongest baselines and improves over QRPO and BoNBoN under the independent Skywork evaluators\. These raw results should be read together with the LC tables because several methods, including TUP and QRPO, tend to produce longer responses\.

Figure 6:Additional cross\-model agreement rates between ArmoRM and independent Skywork reward models\. For each selected fractionpp, the plots show the fraction of completions placed by ArmoRM in the top or bottomppregion that the Skywork evaluator also places in the corresponding region\.Left:UltraFeedback with Skywork\-v2\-Qwen\.Center:Magpie Air with Skywork\-v2\-Llama\.Right:Magpie Air with Skywork\-v2\-Qwen\.Figure 7:Additional cost–benefit curves forλ\\lambda\-truncation\. Each plot compares the probability that truncation discards a completion ranked in the top25%25\\%by the independent Skywork evaluator with the probability that it retains a completion ranked in the bottom25%25\\%by that evaluator\.Left:UltraFeedback with Skywork\-v2\-Qwen\.Center:Magpie Air with Skywork\-v2\-Llama\.Right:Magpie Air with Skywork\-v2\-Qwen\.Figures[6](https://arxiv.org/html/2608.19748#A2.F6)and[7](https://arxiv.org/html/2608.19748#A2.F7)extend the motivating observation in Figure[2](https://arxiv.org/html/2608.19748#S3.F2)beyond the UltraFeedback/Skywork\-v2\-Llama setting\. Across the additional dataset–evaluator combinations, ArmoRM\-defined bottom regions are more consistently recovered by the independent Skywork evaluator than ArmoRM\-defined top regions\. The correspondingλ\\lambdacurves show the same truncation tradeoff: increasingλ\\lambdareduces the chance of retaining completions that the independent evaluator ranks poorly, but also increases the chance of discarding completions it ranks highly\. This supports using moderate truncation rather than an aggressive threshold\.

We also estimate the effective sharpness learned by the trained TUP checkpoint by reversing the construction used during training\. During training, the truncated win\-rate labelwλw\_\{\\lambda\}determines the target policy\-reference log\-ratio through the configured sharpnessβ\\beta\. After training, we instead observe the checkpoint log\-ratio

Δθ​\(y∣x\)=log⁡πθ​\(y∣x\)−log⁡πref​\(y∣x\),\\Delta\_\{\\theta\}\(y\\mid x\)=\\log\\pi\_\{\\theta\}\(y\\mid x\)\-\\log\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\),and ask which value ofβ\\betabest explains its dependence on the reward\-derived truncated win\-rate\. On thetestsplit, we constructwλw\_\{\\lambda\}by ranking each response reward against the 12 reference completions for the same prompt and applying the same thresholdλ=0\.5\\lambda=0\.5used in training\. We then fit the linear relation betweenΔθ​\(y∣x\)\\Delta\_\{\\theta\}\(y\\mid x\)andz=logit⁡\(wλ\)z=\\operatorname\{logit\}\(w\_\{\\lambda\}\)on the finite, unclipped subset\. This gives an estimate ofβeff≈0\.0147\\beta\_\{\\mathrm\{eff\}\}\\approx 0\.0147\. Thus, although training used the nominal valueβ=0\.01\\beta=0\.01, the learned policy behaves on this diagnostic as if its effective sharpness is roughly1\.51\.5times larger\.

### B\.2Length\-matched rewards

Table 7:TUP \(withλ=0\.5\\lambda=0\.5\) win rates against each benchmark method on similar\-length AlpacaEval responses for Llama 8B Tülu 3 SFT trained on Magpie Air\. For each comparison, we retain response pairs whose lengths differ by at most10%10\\%\. Values above50%50\\%favor TUP and are shown in bold\. The final column gives the number of retained pairs out of2,4152\{,\}415\.While the LC\-reward results in Tables[1](https://arxiv.org/html/2608.19748#S4.T1)and[2](https://arxiv.org/html/2608.19748#S4.T2)already account for response length at the metric level, we seek further evidence that verbosity is not the sole contributor to TUP’s strong performance\. To this end, Table[7](https://arxiv.org/html/2608.19748#A2.T7)compares TUP with each baseline on responses of similar length\. TUP is preferred over all benchmark methods under ArmoRM and Skywork\-Llama\. The results under Skywork\-Qwen are more mixed across methods, although TUP’s average win rate remains above50%50\\%under all three reward models\. Importantly, the matching criterion retains only a fraction of the response pairs and produces a different subset for each baseline\. These results should therefore be viewed as complementary evidence rather than as an estimate of performance on the full test set\. Together with the LC\-reward results, they further suggest that TUP’s strong performance is not solely due to its longer responses\.

### B\.3Global vs\. per\-prompt truncation

To estimate the potential headroom of prompt\-adaptive truncation over a fixed global thresholdλ\\lambda, we consider an idealized empirical setting in which Skywork\-Llama serves as the “oracle” evaluator\. For each UltraFeedback promptii, let\(yi\(1\),yi\(2\),…,yi\(N\)\)\(y\_\{i\}^\{\(1\)\},y\_\{i\}^\{\(2\)\},\\ldots,y\_\{i\}^\{\(N\)\}\)denote the responses ranked by ArmoRM\. We then:

1. 1\.Compute the ideal prompt\-specific threshold λi∗=arg​maxj⁡rSkywork​\(yi\(j\)\)−1N,\\lambda\_\{i\}^\{\*\}=\\frac\{\\argmax\_\{j\}r\_\{\\text\{Skywork\}\}\(y\_\{i\}^\{\(j\)\}\)\-1\}\{N\},which discards all responses ranked below the Skywork\-best response by ArmoRM while retaining that response\.
2. 2\.Select the single global thresholdλ∗\\lambda^\{\*\}that maximizes the average Skywork\-Llama win rate across all prompts\.

To isolate the effect of the truncation threshold, we apply no within\-tail reweighting\. Under this idealized setting, the prompt\-specific thresholds achieve an empirical Skywork\-based win rate of0\.93250\.9325, compared with0\.88200\.8820for the best fixed global threshold\. Thus, although prompt\-specific truncation offers some additional headroom, the fixed global threshold captures most of the attainable win rate in this analysis\.

## Appendix CImplementation details

### C\.1Codebase, models, and data

The experiments in Tables[1](https://arxiv.org/html/2608.19748#S4.T1)–[6](https://arxiv.org/html/2608.19748#A2.T6)use the released QRPO benchmark data rather than regenerating reference completions or reward annotations\. These data contain prompt\-specific reference\-completion pools scored by ArmoRM\.

All experiments are implemented using the official QRPO repository222[https://github\.com/CLAIRE\-Labo/quantile\-reward\-policy\-optimization\.git](https://github.com/CLAIRE-Labo/quantile-reward-policy-optimization.git), together with its released preprocessed reference datasets333[https://huggingface\.co/collections/skandermoalla/qrpo\-reference\-datasets](https://huggingface.co/collections/skandermoalla/qrpo-reference-datasets)\. We use Llama 8B Tülu 3 SFT as the initial policy\. On Mistral 7B we run a dedicated supervised fine\-tuning \(STF\) on the original model \(Mistral\-7B\-Instruct\-v0\.2\), we closely followed QRPO code, using the prompt–chosen\-response from Magpie\-Air\-DPO\-100K\-v0\.1\. After filtering sequences to at most 2,048 tokens, the training set contained 97,812 examples, with 1,992 held out for evaluation\. We trained for one epoch inbfloat16using an effective batch size of 128 across four GPUs, a learning rate of5×10−75\\times 10^\{\-7\}, 10% warm\-up, and cosine decay, without LoRA or other parameter\-efficient tuning methods\. To reproduce, please refer to our or QRPO GitHub repository\.

The training reward model in the released data isRLHFlow/ArmoRM\-Llama3\-8B\-v0\.1\. For transfer evaluation, we rescore generated completions withSkywork/Skywork\-Reward\-V2\-Llama\-3\.1\-8Band Skywork/Skywork\-Reward\-V2\-Qwen3\-8B\.

For AlpacaEval, we report the win rate and length\-controlled win rate against the defaultgpt4\-turboreference outputs, which correspond togpt\-4\-1106\-preview;gpt\-4ois used only as the judge due to the deprecation ofgpt\-4\-1106\-preview\.

The UltraFeedback setting contains61,02461\{,\}024training prompts and a held\-out set of1,9951\{,\}995prompts, split into998998checkpoint\-selection prompts and997997reward\-reporting prompts\. The Magpie Air setting contains97,81297\{,\}812training prompts and a held\-out set of1,9921\{,\}992prompts, split into996996checkpoint\-selection prompts and996996reward\-reporting prompts\. AlpacaEval contains805805evaluation prompts\. The QRPO benchmark reference pools contain5050reference completions per prompt\.

For DPO, REBEL, and QRPO, we use the best\-performing non\-SFT, off\-policy configurations reported in Tables 12 and 14 of[26](https://arxiv.org/html/2608.19748#bib.bib40)\. We use theoffpolicy2bestandoffpolicy2randomvariants as reported in Tables[9](https://arxiv.org/html/2608.19748#A3.T9)and[10](https://arxiv.org/html/2608.19748#A3.T10)\. BoNBoN and TUP are not included in the original QRPO benchmark, so we perform a separate hyperparameter search for these methods\.

### C\.2Reward models used for evaluation

Table[8](https://arxiv.org/html/2608.19748#A3.T8)summarizes the external reward models used in our experiments\. ArmoRM is used only as the offline proxy reward model for training labels, whereas the Skywork\-v2 models andgpt\-4oare used only for evaluation\. The RewardBench ranks are provided only as external context, since leaderboard positions can change as new models are added\.

Table 8:RewardBench standings \(regardless of public availability\) for the reward and judge models used in our experiments, retrieved in April 2026\. Global Rank denotes the model’s overall position on the RewardBench leaderboard, and Type Rank denotes its position within the corresponding model category\. The asterisk marksgpt\-4o; because AlpacaEval uses an API\-hosted judge, the exact served model version may not match the RewardBench entry\.
### C\.3TUP implementation

TUP uses the same precomputed completion pools and ArmoRM scores as the QRPO benchmark\. For each prompt, we convert the ArmoRM scores in the pool into empirical in\-pool win\-rates, apply the shifted truncationw^λ,r=max⁡\(w^r−λ,0\)\\hat\{w\}\_\{\\lambda,r\}=\\max\(\\hat\{w\}\_\{r\}\-\\lambda,0\), and train with the BCE objective in Algorithm[1](https://arxiv.org/html/2608.19748#alg1)\. Relative to the QRPO pipeline, the implementation only changes the scalar training label, the known intercept, and the loss used\. In the reported experiments the intercept uses the population normalizerβ​log⁡Zλ,β\\beta\\log Z\_\{\\lambda,\\beta\}; Appendix[A\.1](https://arxiv.org/html/2608.19748#A1.SS1)also gives the exact finite\-pool alternativeZK,λ,βZ\_\{K,\\lambda,\\beta\}for the empirical\-rank convention used in Algorithm[1](https://arxiv.org/html/2608.19748#alg1)\.

##### Numerical evaluation of the population normalizer\.

Although Proposition[3\.1](https://arxiv.org/html/2608.19748#S3.Thmtheorem1)writesZλ,βZ\_\{\\lambda,\\beta\}as an incomplete Beta function, the implementation evaluates the equivalent continuous integral directly with high\-precision arithmetic\. We do not use standard regularized incomplete\-Beta routines such asscipy\.special\.betainc, since these APIs normalize by the complete Beta function and assume positive shape parameters, whereas here the second parameter is1−1/β<01\-1/\\beta<0for theβ\\betavalues used in our sweeps\. We therefore compute

Zλ,β=∫01−λu1/β\(1−u\)−1/βduZ\_\{\\lambda,\\beta\}=\\int\_\{0\}^\{1\-\\lambda\}u^\{1/\\beta\}\(1\-u\)^\{\-1/\\beta\}\\,duusingmpmathwithmp\.workdps\(256\)\. This avoids the underflow that can occur in standard double\-precision quadrature for smallβ\\betaand makes the BCE interceptβ​log⁡Zλ,β\\beta\\log Z\_\{\\lambda,\\beta\}reproducible\.

### C\.4Hyperparameter search and model selection

Table 9:Training hyperparameters Llama 8B Tülu 3 SFT on UltraFeedback reported in Table[1](https://arxiv.org/html/2608.19748#S4.T1), Table[2](https://arxiv.org/html/2608.19748#S4.T2)and Table[4](https://arxiv.org/html/2608.19748#A2.T4)\.MethodDataset variantβ\\betal​rlrNum refλ\\lambdaCheckpointDPOoffpolicy2best0\.013e\-076\-300REBELoffpolicy2best1e\-063e\-076\-200REBEL \(rand\.\)offpolicy2random1e\-061e\-066\-100QRPOoffpolicy2best3e\-043e\-073\-100QRPO \(rand\.\)offpolicy2random3e\-043e\-073\-100BoNBoNoffpolicy2best1e\-031e\-066\-476TUP mildoffpolicy2random0\.011e\-0760\.2400TUP mid\.offpolicy2random0\.011e\-0760\.5400TUP aggressiveoffpolicy2random0\.011e\-0760\.8476

Table 10:Training hyperparameters Llama 8B Tülu 3 SFT on Magpie Air reported in Table[1](https://arxiv.org/html/2608.19748#S4.T1), Table[2](https://arxiv.org/html/2608.19748#S4.T2)and Table[5](https://arxiv.org/html/2608.19748#A2.T5)\.MethodDataset variantβ\\betal​rlrNum refλ\\lambdaCheckpointDPOoffpolicy2random0\.011e\-066\-480REBELoffpolicy2best1e\-061e\-076\-640REBEL \(rand\.\)offpolicy2random1e\-041e\-066\-320QRPOoffpolicy2best1e\-031e\-071\-320QRPO \(rand\.\)offpolicy2random1e\-031e\-071\-320BoNBoNoffpolicy2best1e\-031e\-066\-764TUP mildoffpolicy2random0\.011e\-0760\.2320TUP mid\.offpolicy2random0\.011e\-0760\.5320TUP aggressiveoffpolicy2random0\.011e\-0760\.8320

Table 11:Training hyperparameters Mistral 7B SFT on Magpie Air reported in Table[3](https://arxiv.org/html/2608.19748#S4.T3)and Table[6](https://arxiv.org/html/2608.19748#A2.T6)\.MethodDataset variantβ\\betal​rlrNum refλ\\lambdaCheckpointDPOoffpolicy2random0\.033e\-076\-480REBELoffpolicy2best1e\-041e\-076\-480REBEL \(rand\.\)offpolicy2random1e\-043e\-076\-320QRPOoffpolicy2best3e\-031e\-071\-764QRPO \(rand\.\)offpolicy2random3e\-031e\-061\-764BoNBoNoffpolicy2best1e\-031e\-076\-480TUP mildoffpolicy2random0\.011e\-0660\.2764TUP mid\.offpolicy2random0\.011e\-0660\.5764TUP aggressiveoffpolicy2random0\.011e\-0660\.8764

For BoNBoN, we use only the IPO\-BoN component of the original objective\. This gives the closest functional comparison in our setting, which is offline BoN\-style distillation without an additional SFT loss\. We tune only the learning rate:

lr∈\{1​e−6,3​e−7,1​e−7\}\.\\text\{lr\}\\in\\\{1\\mathrm\{e\}\{\-6\},3\\mathrm\{e\}\{\-7\},1\\mathrm\{e\}\{\-7\}\\\}\.
For TUP using Llama model on UltraFeedback and Magpie Air, we tune the learning rate, preference sharpness, and truncation threshold over

lr∈\{1​e−6,3​e−7,1​e−7\},β∈\{3​e−3,1​e−2,3​e−2\},λ∈\{0\.2,0\.5,0\.8\}\.\\text\{lr\}\\in\\\{1\\mathrm\{e\}\{\-6\},3\\mathrm\{e\}\{\-7\},1\\mathrm\{e\}\{\-7\}\\\},\\quad\\beta\\in\\\{3\\mathrm\{e\}\{\-3\},1\\mathrm\{e\}\{\-2\},3\\mathrm\{e\}\{\-2\}\\\},\\quad\\lambda\\in\\\{0\.2,0\.5,0\.8\\\}\.
For Mistral 7B on Magpie Air, due to computational constraints, we ran the search on a narrower set of hyper parameters:

lr∈\{1​e−6,3​e−7\},β∈\{1​e−2,3​e−2\},λ∈\{0\.2,0\.5,0\.8\}\.\\text\{lr\}\\in\\\{1\\mathrm\{e\}\{\-6\},3\\mathrm\{e\}\{\-7\}\\\},\\quad\\beta\\in\\\{1\\mathrm\{e\}\{\-2\},3\\mathrm\{e\}\{\-2\}\\\},\\quad\\lambda\\in\\\{0\.2,0\.5,0\.8\\\}\.
Checkpoint selection uses only the validation split\. We select checkpoints by length\-controlled reward under ArmoRM\. For both model families, UltraFeedback checkpoints are saved and evaluated every100100steps and Magpie Air checkpoints are saved and evaluated every160160steps\. For UltraFeedback, we only consider checkpoints with mean generated length below650650tokens\. For Magpie Air, we do not impose an analogous length threshold\. Tables[9](https://arxiv.org/html/2608.19748#A3.T9)and[10](https://arxiv.org/html/2608.19748#A3.T10)report the selected hyperparameters and checkpoints for each method\.

### C\.5Training settings

Unless stated otherwise, training settings follow the QRPO benchmark configuration\. Training usesbfloat16precision with Accelerate and DeepSpeed ZeRO\-1\. We use a cosine learning\-rate schedule with warmup ratio0\.10\.1\. Gradient clipping is effectively disabled in the Llama benchmark runs by settingmax\_grad\_normto10810^\{8\}\.

The per\-device training batch size is22, the per\-device evaluation batch size is44, and gradient accumulation is1616\. With44GPUs, this gives an effective training batch size of128128\.

The maximum prompt length, maximum completion length, and maximum total sequence length are all20482048\. The released reference completions were generated with temperature1\.01\.0and top\-p=1\.0p=1\.0\. For online evaluation, we generate one completion per prompt using temperature0\.60\.6, top\-p=0\.9p=0\.9, and a maximum of20482048new tokens\.

### C\.6Evaluation and uncertainty

For reward\-model evaluation, we follow the QRPO evaluation pipeline\. For each generated completion, the evaluator computes both raw reward and length\-controlled reward using the reference\-completion pool for the same prompt\. For UltraFeedback and Magpie Air, we use the released QRPO reference pools\. For AlpacaEval, which is not part of the released QRPO reference data, we construct the reference pool by generating1616completions per prompt from the base model using vLLM\. To compute length\-controlled metrics under a given reward model, we score both the generated completions and the corresponding prompt\-specific reference pools with that reward model\. The length\-controlled score normalizes reward and length relative to the prompt\-specific reference pool, then removes the estimated linear effect of length\. Prompts with degenerate reference\-length variance are masked by the evaluation code\.

For reward\-model metrics, we report uncertainty as the sample standard deviation across three generation seeds\. For AlpacaEval, we replace the recentlygpt\-4\-1106\-previewjudge used in earlier benchmark configurations withgpt\-4o\. This change may make the AlpacaEval scores not directly comparable to results computed with the older judge\.

### C\.7Deviation from the QRPO implementation

The QRPO codebase constructs training examples using both chosen and rejected completions by default\. For TUP, we instead use a single randomly selected response per prompt\. This matches the scalar\-label formulation of Algorithm[1](https://arxiv.org/html/2608.19748#alg1), where each completion receives one shifted\-truncated win\-rate label\. It also means that TUP receives less per\-prompt training signal than methods that use both responses\.

### C\.8Compute resources

The reported training runs were performed on a machine with44H200 GPUs and6464CPU cores\.

### C\.9Compliance and asset licenses

Table[12](https://arxiv.org/html/2608.19748#A3.T12)summarizes the external code, model, API, and dataset assets used in the experiments\. License and terms information was retrieved from the official repository, model cards, dataset cards, and provider documentation on April 30, 2026\. We use these assets only for offline training or evaluation as described in Appendix[C](https://arxiv.org/html/2608.19748#A3), and we do not redistribute the underlying external model checkpoints, API\-hosted judge, or source datasets as part of this submission\.

Table 12:External assets used in the experiments and their reported licenses or terms\. For the QRPO reference data, the table lists the released preprocessed datasets actually loaded by our experiments; these are derived reference\-completion pools and rewards rather than newly collected data in this work\.