Tail-Likelihood Reinforcement Learning

# Tail-Likelihood Reinforcement Learning
Source: [https://arxiv.org/html/2609.02987](https://arxiv.org/html/2609.02987)
Daman AroraFahim TajwarGuanning ZengQingyang WuZhongzhu ZhouChenfeng XuHaiwen FengAffiliation:Carnegie Mellon University University of California, Berkeley Impossible, Inc\. Yuda SongAarti SinghRuslan SalakhutdinovJ\. Andrew BagnellAffiliation:Together AI Aurora Innovation\{shrinivr, jeff4, azanette\}@andrew\.cmu\.eduJeff SchneiderAndrea Zanette

###### Abstract

Reinforcement learning typically optimizes average reward\. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high\-reward rollout\. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high\-reward outcomes\. We propose to optimize this coverage directly\. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events\. We introduceTail\-Likelihood Reinforcement Learning \(TailRL\), which maximizes the log\-probability of exceeding a randomly chosen reward threshold\. Its gradient gives more weight to rare, high\-reward rollouts and can be interpreted as a mixture of Best\-of\-\(k\) gradients\. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines\. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high\-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time\.

††footnotetext:†Joint advising\.††footnotetext:Project website, code, and other assets:[https://zanette\-labs\.github\.io/TailRL\-website/](https://zanette-labs.github.io/TailRL-website/)## 1Introduction

Reinforcement learning \(RL\) typically optimizes the expected reward of a policy\([Williams, 1992](https://arxiv.org/html/2609.02987#bib.bib48);[Shao et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib36);[Ahmadian et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib37);[Yu et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib38);[Zheng et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib11)\)\. For generative policies, however, the mean reward does not fully characterize performance: two policies with similar mean reward can have very different probabilities of producing rare but exceptionally good rollouts\. This distinction matters whenever additional samples can be drawn, both during training and at inference time\.

Recent work has exposed this problem directly: standard RL training can progressively lose coverage over rare, high\-reward rollouts, often visible as a degradation in Best\-of\-kkperformance\([Cui et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib80);[Yue et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib8);[Wu et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib40);[Dang et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib12);[Kirk et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib81)\)\. Once these rollouts become sufficiently unlikely, they are rarely sampled again, making further policy improvement increasingly difficult\. The same loss of coverage limits inference\-time scaling: a policy may perform well with a single sample while gaining little from drawing many\([Walder and Karkhanis, 2025](https://arxiv.org/html/2609.02987#bib.bib59);[Chen et al\., 2025b](https://arxiv.org/html/2609.02987#bib.bib5);[Yang et al\., 2025b](https://arxiv.org/html/2609.02987#bib.bib41)\)\. Thus, optimizing only the mean reward can discard information about the upper tail of the reward distribution that is crucial for both training and inference scaling\.

![Refer to caption](https://arxiv.org/html/2609.02987v1/figures/teaser_tugofwar.png)Figure 1:Expected reward pulls one rope; TailRL pulls them all\.Left: RL post\-training reshapes the reward distribution of the base policy\. Expected\-reward methods \(GRPO, RLOO\) shift the distribution and sharpen it around the mean, while TailRL shifts it further and grows a heavy high\-reward tail\. Middle: the expected\-reward objectiveJRLJ\_\{\\mathrm\{RL\}\}improves the distribution through a single handle, its mean\. Right: TailRL increases the tail probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)across reward thresholdsτ\\tau, placing greater emphasis on rarer, higher\-reward outcomes\.For binary rewards, MaxRL\([Tajwar et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib39)\)offers one way to address this problem\. Rather than maximizing the probability of success, MaxRL maximizes its log\-probability, placing greater emphasis on rare successes\. Its gradient decomposes into a harmonic mixture of Pass@kkgradients, directly connecting likelihood maximization to a mixture of Pass@kkobjectives\.

How should this principle extend tocontinuous rewards? Our starting point is simple: every reward threshold defines a binary event\. Given an inputxx, a rolloutz∼πθ\(⋅∣x\)z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\), and a thresholdτ\\tau, we ask whether the reward exceedsτ\\tau\. The corresponding*tail probability*is

pθ\(x,τ\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)\>τ\)\.p\_\{\\theta\}\(x,\\tau\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigl\(r\(x,z\)\>\\tau\\bigr\)\.\(1\)A continuous reward can be viewed as a family of binary success events, one for every thresholdτ\\tau\. This perspective leads toTail\-Likelihood Reinforcement Learning \(TailRL\)\. For rewards in\[0,1\]\[0,1\], TailRL maximizes the expected log\-likelihood of exceeding a uniformly chosen reward threshold:

JTailRL​\(θ,x\)=∫01log⁡pθ​\(x,τ\),𝑑τ\.J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\),d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q\}\(2\)Unlike expected\-reward maximization, which acts on a single summary of the reward distribution, TailRL explicitly optimizes upper\-tail probabilities across reward levels\. We show that the gradient of TailRL decomposes into a harmonic mixture of Best\-of\-kkgradients \([Section3\.1](https://arxiv.org/html/2609.02987#S3.SS1)\), directly connecting the training objective to coverage of high\-reward rollouts and inference\-time scaling\. MaxRL emerges as the binary\-reward special case\. Despite its different objective, TailRL admits a simple critic\-free policy\-gradient estimator that differs from standard RL only in its advantage calculation, allowing it to be implemented by swapping the advantage function in an existing RL pipeline\. Across object localization \([Section6\.1](https://arxiv.org/html/2609.02987#S6.SS1)\), maze navigation \([Section6\.2](https://arxiv.org/html/2609.02987#S6.SS2)\), GUI grounding \([Section6\.3](https://arxiv.org/html/2609.02987#S6.SS3)\), and code optimization \([Section6\.4](https://arxiv.org/html/2609.02987#S6.SS4)\), TailRL leverages rare high\-reward samples during training to avoid suboptimal solutions and produces policies that benefit more from additional samples at inference time\.

Our contributions are as follows\.

1. 1\.A likelihood objective for continuous rewards\.TailRL maximizes the log\-probability of exceeding a uniformly drawn reward threshold, which weights each reward level by the inverse of how often the policy reaches it\. It introduces no threshold or weighting hyperparameter and reduces exactly to MaxRL for binary rewards\.
2. 2\.Alignment with inference\-time scaling\.The gradient of TailRL decomposes harmonically over Best\-of\-kkgradients \([Section3\.1](https://arxiv.org/html/2609.02987#S3.SS1)\), ∇θJT​a​i​l​R​L​\(θ,x\)=∑k=1∞1k​∇θBest\-of\-​k​\(θ,x\),\\nabla\_\{\\theta\}J\_\{TailRL\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{1\}\{k\}\\,\\nabla\_\{\\theta\}\\,\\text\{Best\-of\-\}k\(\\theta;x\),so TailRL improves Best\-of\-kkat every inference budget without choosing one in advance\.
3. 3\.An unbiased finite\-rollout estimator\.A group ofNNrollouts defines an order\-NNtruncation of TailRL that interpolates from expected reward \(N=1N=1\) to the population objective \(N→∞N\\to\\infty\), and we derive closed\-form rollout weights that estimate its gradient without bias \([Section4](https://arxiv.org/html/2609.02987#S4)\)\. Unlike REINFORCE, where more rollouts only reduce variance, here the rollout budget selects the objective being optimized\.
4. 4\.Strong empirical results\.TailRL matches supervised objectives that observe the ground truth on object localization \([Section6\.1](https://arxiv.org/html/2609.02987#S6.SS1)\), learns from initial policies with0\.01%0\.01\\%success where expected\-reward baselines fail on maze navigation \([Section6\.2](https://arxiv.org/html/2609.02987#S6.SS2)\), matches RLOO’s Pass@1024 on GUI grounding with128128–256×256\\timesfewer inference rollouts \([Section6\.3](https://arxiv.org/html/2609.02987#S6.SS3)\), and reaches a7\.7×7\.7\\timesBest\-of\-1024 speedup on code optimization where GRPO and RLOO collapse onto copying the input \([Section6\.4](https://arxiv.org/html/2609.02987#S6.SS4)\)\.

## 2Preliminaries

We primarily focus on optimizing continuous reward reinforcement learning, where for each inputxx, a rolloutzzis generated by the policyπθ\\pi\_\{\\theta\}\. A rollout may be a generated response, program, or trajectory\. A deterministic reward function provides a scalar feedback for an input rollout pairr⁡\(x,z\)∈\[0,1\]r\(x,z\)\\in\[0,1\]\.[AppendixD](https://arxiv.org/html/2609.02987#A4)treats a more general bounded reward range\.

Training ultimately averages overx∼ρx\\sim\\rho\. To keep the notation light, we write each objective for a fixed inputxxand leave the outer average over inputs implicit\. We define the policy’s score\-function asS⁡\(x,z\):=∇θ​log​πθ​\(z∣x\)S\(x,z\):=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(z\\mid x\)\. Standard reinforcement learning maximizes expected reward and the score\-function identity\([Williams, 1992](https://arxiv.org/html/2609.02987#bib.bib48)\)gives its policy gradient as follows,

JRL\(θ;x\):=𝔼z∼πθ\(⋅∣x\)\[r\(x,z\)\],∇θJRL\(θ;x\)=𝔼z∼πθ\(⋅∣x\)\[r\(x,z\)S\(x,z\)\]\.\\begin\{gathered\}J\_\{\\mathrm\{RL\}\}\(\\theta;x\):=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[r\(x,z\)\\right\],\\\\\[3\.0pt\] \\nabla\_\{\\theta\}J\_\{\\mathrm\{RL\}\}\(\\theta;x\)=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[r\(x,z\)S\(x,z\)\\right\]\.\\end\{gathered\}\\addcontentsline\{lla\}\{section\}\{\\numberline q:jrl\}\(3\)
Thus, the policy gradient is a weighted combination of the score\-function and the weights are determined by the rollout’s reward\. During training, critic\-free rollout based policy gradient methods drawNNindependent rolloutsz1,…,zN∼πθ\(⋅∣x\)z\_\{1\},\\ldots,z\_\{N\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\. Critic\-free methods such as GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib36)\)and RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib37)\)compute a finite rollout estimate of[Eq\.3](https://arxiv.org/html/2609.02987#S2.E3)\. PKPO\([Walder and Karkhanis, 2025](https://arxiv.org/html/2609.02987#bib.bib59)\)optimizes Pass@kkand Best\-of\-kkfor binary and continuous rewards using a similar finite\-rollout based estimation framework\.[Section4](https://arxiv.org/html/2609.02987#S4)shows that TailRL uses the same template and changes only how these advantages are computed, and[AppendixF](https://arxiv.org/html/2609.02987#A6)places the three advantage functions side by side\.

### 2\.1Inference\-Time Selection

At deployment, additional inference compute can be used to sample several rollouts and select the one with the highest reward\. Forkkindependent rollouts, this performance is measured by probability of at\-least one success \(Pass@kk\) when rewards are binary and expected maximum reward among \(Best\-of\-kk\) when rewards are continuous:

Pass@k\(θ;x\):=1−\(Prz∼πθ\(⋅∣x\)\(r\(x,z\)=0\)\)k,\\displaystyle\\text\{Pass@\}k\(\\theta;x\):=1\-\\left\(\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigl\(r\(x,z\)=0\\bigr\)\\right\)^\{k\},\(binary reward\)\(4\)Best\-of\-k\(θ;x\):=𝔼\{zi\}i=1k∼πθ\(⋅∣x\)\[max1≤i≤kr\(x,zi\)\],\\displaystyle\\text\{Best\-of\-\}k\(\\theta;x\):=\\mathbb\{E\}\_\{\\\{z\_\{i\}\\\}\_\{i=1\}^\{k\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\max\_\{1\\leq i\\leq k\}r\(x,z\_\{i\}\)\\right\],\(continuous reward\)\(5\)Atk=1k=1, both reduce to the mean reward,JRL​\(θ,x\)J\_\{\\mathrm\{RL\}\}\(\\theta;x\)\.[AppendixG](https://arxiv.org/html/2609.02987#A7)gives the estimators we typically use to compute both empirically\. Both Pass@kkand Best\-of\-kkare non\-decreasing inkk, and ask→∞k\\to\\inftyeach approaches the maximum reward in their support\.

Unlike mean reward,Best\-of\-​k\\text\{Best\-of\-\}kdepends strongly on the upper part of the reward distribution\. Two policies with the same mean can scale differently with additional training or inference compute if one assigns more probability to high\-reward rollouts\.

### 2\.2Policy as a Generative Model of Rewards

A policy and reward function together define a distribution over rewards\. For an inputxx, the policy samples a rolloutzzand the reward function assigns its reward:

z∼πθ\(⋅∣x\),r=r\(x,z\)∈\[0,1\]\.z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\),\\qquad r=r\(x,z\)\\in\[0,1\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:reward\-generative\}\(6\)Because the reward function is deterministic, all randomness in the reward comes from the policy\. The induced reward distribution for any eventAA\(A⊆\[0,1\]A\\subseteq\[0,1\]\) over the support of rewards is defined as,

Prz∼πθ\(⋅∣x\)\(r\(x,z\)∈A\)=𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)∈A\}\]\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\(r\(x,z\)\\in A\\right\)=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\\in A\\\}\}\]\\addcontentsline\{lla\}\{section\}\{\\numberline q:reward\-mass\}\(7\)In this view, the policy is a generative model over rewards, and training manipulates the probability mass over reward values\. Expected reward reinforcement learning uses only the mean reward to shape this reward distribution\.

Different policies can have the same mean reward while assigning very different probabilities to high\-reward outcomes\. This becomes visible in their difference in Best\-of\-kkand Pass@kkperformance\. This motivates objectives that act on the entirety of the reward distribution rather than only its mean\. We begin with binary rewards, where the distribution is completely determined by a single success probability\.

### 2\.3MaxRL for Binary Rewards

For a binary rewardr⁡\(x,z\)∈\{0,1\}r\(x,z\)\\in\\\{0,1\\\}, expected reward equals the probability of success\.

qθ\(x\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)=1\)\.q\_\{\\theta\}\(x\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\(r\(x,z\)=1\\right\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:binary\-p\}\(8\)The reward distribution is therefore a Bernoulli distribution over success and failures\. The reward distribution is fully determined byqθ​\(x\)q\_\{\\theta\}\(x\)\. Standard reinforcement learning maximizesqθ​\(x\)q\_\{\\theta\}\(x\), whereas MaxRL\([Tajwar et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib39)\)maximizesJMaxRL​\(θ,x\)=log⁡qθ​\(x\)J\_\{\\mathrm\{MaxRL\}\}\(\\theta;x\)=\\log q\_\{\\theta\}\(x\)and its gradient is,

∇θJMaxRL\(θ;x\)=1qθ​\(x\)𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)=1\}S\(x,z\)\]\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{MaxRL\}\}\(\\theta;x\)=\\frac\{1\}\{q\_\{\\theta\}\(x\)\}\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)=1\\\}\}S\(x,z\)\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:maxrl\-gradient\}\(9\)The factor1/qθ​\(x\)1/q\_\{\\theta\}\(x\)gives greater weight to inputs on which success is rare\. MaxRL also connects to inference\-time sampling\.∇θJMaxRL​\(θ,x\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{MaxRL\}\}\(\\theta;x\)decomposes as a harmonic mixture of Pass@kkgradients\.

∇θJMaxRL​\(θ,x\)=∑k=1∞1k​∇θPass​@​k​\(θ,x\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{MaxRL\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{1\}\{k\}\\nabla\_\{\\theta\}\\mathrm\{Pass@\}k\(\\theta;x\)\\addcontentsline\{lla\}\{section\}\{\\numberline q:binary\-harmonic\}\(10\)

## 3Tail\-Likelihood Reinforcement Learning

![Refer to caption](https://arxiv.org/html/2609.02987v1/figures/teaser_decomposition.png)Figure 2:A continuous reward decomposes into threshold events\. A rollout with rewardrir\_\{i\}clears every threshold belowrir\_\{i\}\. Across a group of rollouts, these binary outcomes estimate the tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)at every reward threshold\. TailRL maximizes the average log\-probability along this curve\.We motivate TailRL by expressing expected reward in terms of the policy’s upper\-tail probabilities\. For rewards in\[0,1\]\[0,1\], define the tail probability at thresholdτ\\tauas

pθ\(x,τ\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)\>τ\)\.p\_\{\\theta\}\(x,\\tau\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\(r\(x,z\)\>\\tau\\right\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-tail\-probability\}\(11\)The expected reward is exactly the area under this tail\-probability curve:

JRL​\(θ,x\)=𝔼⁡\[r⁡\(x,z\)\]=∫01pθ​\(x,τ\)​𝑑τ\.J\_\{\\mathrm\{RL\}\}\(\\theta;x\)=\\mathbb\{E\}\[r\(x,z\)\]=\\int\_\{0\}^\{1\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:expected\-reward\-tail\}\(12\)
Tail\-Likelihood Reinforcement Learning \(TailRL\)instead applies the MaxRL likelihood principle at every reward threshold\. It maximizes the average log\-probability of exceeding a uniformly sampled threshold:

JTailRL​\(θ,x\):=∫01log⁡pθ​\(x,τ\)​𝑑τ=𝔼τ∼Unif⁡\[0,1\]​\[log⁡pθ​\(x,τ\)\]\.J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\):=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau=\\mathbb\{E\}\_\{\\tau\\sim\\mathrm\{Unif\}\[0,1\]\}\\left\[\\log p\_\{\\theta\}\(x,\\tau\)\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-objective\}\(13\)Equivalently, expected\-reward RL aggregates tail probabilities arithmetically, whereas TailRL aggregates them geometrically\. This makes small tail probabilities more influential, as seen directly from its gradient

∇θJTailRL​\(θ,x\)=∫011pθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{1\}\{p\_\{\\theta\}\(x,\\tau\)\}\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-threshold\-gradient\}\(14\)The higher the reward threshold, the smaller the tail\-probability and higher is the weight for the gradient\. Thus reward thresholds that are difficult to reach receive larger weight during optimization\.

### 3\.1Harmonic Decomposition over Best\-of\-kk

Although TailRL is defined through reward thresholds, it has an exact interpretation in terms of inference\-time sampling\. Recall thatBest​\-​of​\-​k​\(θ,x\)\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}\(\\theta;x\)is the expected maximum reward amongkkindependent rollouts fromπθ\(⋅∣x\)\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.

###### Theorem 1\(Best\-of\-kkdecomposition\)\.

The TailRL objective decomposes as

JTailRL​\(θ,x\)=∑k=1∞Best​\-​of​\-​k​\(θ,x\)−1k,J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}\(\\theta;x\)\-1\}\{k\},\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-harmonic\-objective\-limit\}\(15\)and, under the regularity conditions of[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6), its gradient satisfies

∇θJTailRL​\(θ,x\)=∑k=1∞1k​∇θBest​\-​of​\-​k​\(θ,x\)\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{1\}\{k\}\\nabla\_\{\\theta\}\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-harmonic\-gradient\-limit\}\(16\)

Thus, TailRL combines Best\-of\-kklearning signals across all inference budgets\. The harmonic weights1/k1/karise automatically from the logarithm, so TailRL does not require selecting a target inference budget in advance\. The proof is given in[SectionC\.3\.3](https://arxiv.org/html/2609.02987#A3.SS3.SSS3)\.

### 3\.2Recovery of MaxRL for Binary Rewards

For binary rewards, every nontrivial reward threshold defines the same success event\. Let

qθ\(x\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)=1\)q\_\{\\theta\}\(x\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\(r\(x,z\)=1\\right\)denote the probability of success\. For everyτ∈\[0,1\)\\tau\\in\[0,1\),pθ​\(x,τ\)=qθ​\(x\)p\_\{\\theta\}\(x,\\tau\)=q\_\{\\theta\}\(x\), and therefore

JTailRL​\(θ,x\)=∫01log⁡qθ​\(x\)​𝑑τ=log⁡qθ​\(x\)=JMaxRL​\(θ,x\)\.J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log q\_\{\\theta\}\(x\)\\,d\\tau=\\log q\_\{\\theta\}\(x\)=J\_\{\\mathrm\{MaxRL\}\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-maxrl\-population\}\(17\)Moreover, for binary rewardsBest​\-​of​\-​k\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}coincides withPass​@​k\\mathrm\{Pass@\}k, so the harmonic Best\-of\-kkdecomposition above reduces exactly to the harmonic Pass@kkdecomposition of MaxRL\. Thus, TailRL directly generalizes MaxRL from binary to continuous rewards\.

### 3\.3Probabilistic Interpretation of TailRL

A direct extension of MaxRL would define onlyr⁡\(x,z\)=1r\(x,z\)=1as success and treat every lower reward as the same failure\. Exact success may be rare or unattainable\. Lowering the success threshold makes the event more common, but still treats all rewards on either side of the threshold as equivalent\. Once a rollout crosses the threshold, the objective has no preference for improving it further\. The resulting policy can perform well at the chosen threshold while remaining poor at higher reward levels \([SectionH\.4](https://arxiv.org/html/2609.02987#A8.SS4)\)\. We therefore need a likelihood event that preserves the continuous reward signal\.

A continuous reward defines such an event at every reward threshold\. Forτ∈\[0,1\)\\tau\\in\[0,1\), define the tail\-probability

pθ\(x,τ\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)\>τ\)\.p\_\{\\theta\}\(x,\\tau\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\(r\(x,z\)\>\\tau\\right\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-tail\-probability\-restated\}\(18\)
##### Independent quality audit

We combine the tail events by assigning each reward threshold an independent rollout\. The audit passes only if every rollout clears its assigned threshold\. Independent rollouts are necessary: reusing one rollout would collapse the nested events to the hardest threshold\.

For an audit withLLequally spaced thresholds, letτℓ=\(ℓ−1\)/L\\tau\_\{\\ell\}=\(\\ell\-1\)/Land drawz1,…,zL∼i\.i\.d\.πθ\(⋅∣x\)z\_\{1\},\\ldots,z\_\{L\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid x\)\. Define the event that the policy passes the audit as

EL:=⋂ℓ=1L\{r\(x,zℓ\)\>τℓ\}\.E\_\{L\}:=\\bigcap\_\{\\ell=1\}^\{L\}\\left\\\{r\(x,z\_\{\\ell\}\)\>\\tau\_\{\\ell\}\\right\\\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-audit\-event\}\(19\)Because the rollouts are independent, the audit probability factorizes:

Prz1…zL∼πθ\(⋅∣x\)\(EL∣x\)=∏ℓ=1Lpθ\(x,τℓ\)\.\\Pr\_\{z\_\{1\}\.\.\.z\_\{L\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\(E\_\{L\}\\mid x\)=\\prod\_\{\\ell=1\}^\{L\}p\_\{\\theta\}\(x,\\tau\_\{\\ell\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-audit\-probability\}\(20\)The number of thresholds controls only the resolution of the audit\. Requiring invariance to the change in scale of the objective due to increasing resolution of the audit, while agreeing with ordinary log\-likelihood whenL=1L=1, uniquely gives the normalization1/L1/L:

1L​log⁡Prθ​\(EL∣x\)=1L​∑ℓ=1Llog⁡pθ​\(x,τℓ\)\.\\frac\{1\}\{L\}\\log\\Pr\_\{\\theta\}\(E\_\{L\}\\mid x\)=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\log p\_\{\\theta\}\(x,\\tau\_\{\\ell\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-audit\-rate\}\(21\)
Each midpointτℓ\\tau\_\{\\ell\}represents an interval of width1/L1/L\. The right\-hand side of[Eq\.21](https://arxiv.org/html/2609.02987#S3.E21)is therefore a Riemann sum over the reward range\. LettingL→∞L\\to\\inftygives

JTailRL​\(θ,x\):=limL→∞1L​log⁡Prθ​\(EL∣x\)=∫01log⁡pθ​\(x,τ\)​𝑑τ,J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\):=\\lim\_\{L\\to\\infty\}\\frac\{1\}\{L\}\\log\\Pr\_\{\\theta\}\(E\_\{L\}\\mid x\)=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau,\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-objective\-audit\-limit\}\(22\)
We call[Eq\.13](https://arxiv.org/html/2609.02987#S3.E13)the population\-level TailRL objective\. Equivalently, TailRL maximizes the expected log\-probability of clearing a uniformly sampled reward threshold\.

TailRL uses the full support of the reward distribution rather than selecting an arbitrary cut\-off\. Uniform sampling of thresholdτ\\tauassigns equal importance to the log\-likelihood of their tail\-events\. It introduces no weighing hyperparameter\. More generally, one could sampleτ\\taufrom a non\-uniform distribution inducing a re\-weighting of the log\-likelihood terms\. Any strictly positive normalized weighting is exactly equivalent to applying TailRL after a monotone transformation of the rewards \([Proposition9](https://arxiv.org/html/2609.02987#Thmtheorem9)\)\. We use the uniform distribution throughout and leave task\-specific reward shaping to future work\. The audit defines the population objective; it does not yet prescribe the finite\-rollout estimator used for training \([Section4](https://arxiv.org/html/2609.02987#S4)\)\.

## 4Estimating the TailRL Gradient

Training observes only a finite group of rollouts during training\. The population\-level objective is inestimable from finite rollouts\. We show that a rollout budget ofNNnaturally defines an order\-NNtruncation of the TailRL objective and admits a simple unbiased policy\-gradient estimator\.

### 4\.1From Finite Rollouts to a Finite\-Order Objective

For an orderT≥1T\\geq 1, define

JTailRL\(T\)​\(θ,x\):=∑k=1TBest​\-​of​\-​k​\(θ,x\)−1k\.J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\):=\\sum\_\{k=1\}^\{T\}\\frac\{\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}\(\\theta;x\)\-1\}\{k\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-harmonic\-family\}\(23\)Its gradient is

∇θJTailRL\(T\)​\(θ,x\)=∑k=1T1k​∇θBest​\-​of​\-​k​\(θ,x\)\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\nabla\_\{\\theta\}\\mathrm\{Best\\text\{\-\}of\\text\{\-\}k\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-harmonic\-gradient\-main\}\(24\)AtT=1T=1, this has the standard expected\-reward gradient, whileJTailRL\(T\)​\(θ,x\)→JTailRL​\(θ,x\)J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)\\to J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)asT→∞T\\to\\infty\.

Equivalently, the finite\-order gradient can be written directly in terms of tail probabilities:

∇θJTailRL\(T\)​\(θ,x\)=∫011−\(1−pθ​\(x,τ\)\)Tpθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{1\-\(1\-p\_\{\\theta\}\(x,\\tau\)\)^\{T\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:finite\-tail\-weight\-gradient\}\(25\)The threshold weight equals11atT=1T=1and approaches1/pθ​\(x,τ\)1/p\_\{\\theta\}\(x,\\tau\)asT→∞T\\to\\infty\. Thus, increasingTTsmoothly moves the objective from expected\-reward RL toward population TailRL, progressively emphasizing reward levels that are harder to reach\.[Figure3](https://arxiv.org/html/2609.02987#S5.F3)visualizes this interpolation and compares it with PKPO\.

For binary rewards, the finite\-order family and its estimator reduce exactly to their MaxRL counterparts; see[SectionC\.8](https://arxiv.org/html/2609.02987#A3.SS8)\.

### 4\.2Finite\-Rollout Estimator

Critic\-free methods express policy gradients as weighted combination of the score\-function for different rollouts\. In this section we seek to express the exact weights that allows us to give an unbiased estimate of the gradient of the order\-NNtruncated objective\.

Suppose we drawNNindependent rolloutsz1,…,zN∼πθ\(⋅∣x\)z\_\{1\},\\ldots,z\_\{N\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\. At each reward threshold, we divide one unit of credit equally among the sampled rollouts that exceed that threshold\. A rollout accumulates this credit over every threshold below its reward:

ω⁡\(r⁡\(x,zi\)\):=∫0r⁡\(x,zi\)d​τ∑j=1N𝟙\{r\(x,zj\)\>τ\}\.\\omega\(r\(x,z\_\{i\}\)\):=\\int\_\{0\}^\{r\(x,z\_\{i\}\)\}\\frac\{d\\tau\}\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-finite\-weight\}\(26\)Thresholds cleared by fewer rollouts therefore contribute more weight\.

The weights can be computed exactly after sorting the rewards\. Letr\(1\)≤⋯≤r\(N\)r\_\{\(1\)\}\\leq\\cdots\\leq r\_\{\(N\)\}denote the sorted rewards and setr\(0\):=0r\_\{\(0\)\}:=0\. Then

ω⁡\(r\(i\)\)=ω⁡\(r\(i−1\)\)\+r\(i\)−r\(i−1\)N−i\+1,ω⁡\(r\(0\)\)=0\.\\omega\(r\_\{\(i\)\}\)=\\omega\(r\_\{\(i\-1\)\}\)\+\\frac\{r\_\{\(i\)\}\-r\_\{\(i\-1\)\}\}\{N\-i\+1\},\\qquad\\omega\(r\_\{\(0\)\}\)=0\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-weight\-recurrence\}\(27\)
The resulting policy\-gradient estimator has the standard score\-function form:

gTailRL\(N\)​\(x\):=∑i=1Nω⁡\(r⁡\(x,zi\)\)​S​\(x,zi\)\.g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\):=\\sum\_\{i=1\}^\{N\}\\omega\(r\(x,z\_\{i\}\)\)\\,S\(x,z\_\{i\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-finite\-estimator\}\(28\)Thus, TailRL differs from a standard critic\-free policy\-gradient method only in how sampled rewards are converted into rollout weights\.

###### Theorem 2\(Unbiased finite\-rollout estimator\)\.

ForNNindependent rollouts,

𝔼⁡\[gTailRL\(N\)​\(x\)\]=∇θJTailRL\(N\)​\(θ,x\)\.\\mathbb\{E\}\[g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\]=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-finite\-unbiasedness\}\(29\)

Hence, the rollout budget determines which member of the TailRL family is optimized: one rollout recovers the expected\-reward gradient, while larger rollout groups incorporate progressively higher Best\-of\-kklearning signals\. This differs from REINFORCE, where increasing the rollout count reduces estimation variance without changing the underlying expected\-reward objective\.

##### Centered advantages\.

In practice, we center the rollout weights within each group:

Ai:=ω⁡\(r⁡\(x,zi\)\)−ω¯,ω¯:=1N​∑j=1Nω⁡\(r⁡\(x,zj\)\)\.A\_\{i\}:=\\omega\(r\(x,z\_\{i\}\)\)\-\\bar\{\\omega\},\\qquad\\bar\{\\omega\}:=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\omega\(r\(x,z\_\{j\}\)\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-centered\-advantage\}\(30\)Centering reduces variance and allows TailRL to be used as a drop\-in replacement for the advantage calculation in standard policy\-gradient implementations\. Because the baseline is estimated from the same rollout group, the centered estimator is unbiased for∇θJTailRL\(N−1\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\-1\)\}rather than∇θJTailRL\(N\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\. Complete derivations and proofs are given in[AppendixC](https://arxiv.org/html/2609.02987#A3)\.

## 5Unifying Gradient Weight View

Figure 3:Gradient weight assigned to tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)\. Order\-TTTailRL approaches the population weight\. PKPO’s weight is capped atkoptk\_\{\\mathrm\{opt\}\}and vanishes aspθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)grows\.To compare population\-level TailRL with expected reward maximization, we look one step earlier\. We express the gradients of TailRL and expected reward maximization under a unified view and ask how each objective up\-weights its gradients\.TailRL and expected reward maximization belong to a common family of objectives\.

Consider a fixed inputxxand a fixed parametersθ\\thetaof our policyπθ\(⋅∣x\)\\pi\_\{\\theta\}\(\\cdot\\mid x\)\. Recall that we can express the tail\-probability as,

pθ\(x,τ\):=Prz∼πθ\(⋅∣x\)\(r\(x,z\)\>τ\)p\_\{\\theta\}\(x,\\tau\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\(r\(x,z\)\>\\tau\\right\)\(31\)
Its gradient∇θpθ​\(x,τ\)\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)points in the direction that makes the tail\-event more likely\. Up to constants independent ofθ\\theta, they can be written in the common form they can be expressed as,

Jϕ​\(θ,x\):=∫01ϕ⁡\(pθ​\(x,τ\)\)​𝑑τ,J\_\{\\phi\}\(\\theta;x\):=\\int\_\{0\}^\{1\}\\phi\\\!\\left\(p\_\{\\theta\}\(x,\\tau\)\\right\)d\\tau,\\addcontentsline\{lla\}\{section\}\{\\numberline q:level\-utility\-view\}\(32\)Here,ϕ⁡\(p\)\\phi\(p\)specifies how the objective values a tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)\. Differentiating w\.r\.tθ\\thetagives

∇θJϕ​\(θ,x\)=∫01ϕ′​\(pθ​\(x,τ\)\)​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J\_\{\\phi\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\phi^\{\\prime\}\\\!\\left\(p\_\{\\theta\}\(x,\\tau\)\\right\)\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:weight\-view\}\(33\)
Table 1:Scalar functionϕ\\phiand gradient weightsϕ′\\phi^\{\\prime\}\. Standard RL gives equal weight to gradients while TailRL weights the gradients inverse tail\-probability\.The derivativeϕ′\\phi^\{\\prime\}is therefore the gradient weight of the derivative of a tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\), and it does not introduce a new objective\. It is the marginal value that the objective assigns to making that level more likely\. The three objectives differ only in this weight \([Figs\.3](https://arxiv.org/html/2609.02987#S5.F3)and[1](https://arxiv.org/html/2609.02987#S5.T1)\)\.Expected reward maximization usesϕ⁡\(p\)=p\\phi\(p\)=p, soϕ′​\(p\)=1\\phi^\{\\prime\}\(p\)=1: every tail\-probability receives the same weight, regardless of how often the policy reaches it\. TailRL usesϕ⁡\(p\)=log⁡p\\phi\(p\)=\\log p, soϕ′​\(p\)=1/p\\phi^\{\\prime\}\(p\)=1/p\. Becausepθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)is non\-increasing inτ\\tau, this places greater weight on higher reward thresholds that the policy reaches rarely\. The gradient of order\-TTmember of the truncated family is,

∇θJTailRL\(T\)​\(θ,x\)=∫011−\(1−pθ​\(x,τ\)\)Tpθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{T\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:truncated\-weight\-function\}\(34\)It up\-weights the gradient of the tail probabilities asϕ′​\(p\)=1−\(1−p\)Tp\\phi^\{\\prime\}\(p\)=\\frac\{1\-\\left\(1\-p\\right\)^\{T\}\}\{p\}, i\.e\. the multiplier in[Eq\.34](https://arxiv.org/html/2609.02987#S5.E34)\. AtT=1T=1, this multiplier equals11, recovering the expected reward gradient\. AsT→∞T\\to\\infty, it approaches the tail\-likelihood weight1/p1/p\. Expected reward maximization gives equal weight to the gradient of all tail\-probabilities\.

## 6Experiments

We organize the experiments around four questions implied by the theory\.\[ys: quickly recap the theory and itemize the 4 qs\]

1. 1\.First, is the population\-level TailRL objective a useful learning target, and does its finite\-rollout estimator approach it as number of rollouts increases? \([Section4\.1](https://arxiv.org/html/2609.02987#S4.SS1)\)
2. 2\.Second, does TailRL help specifically when high\-reward rollouts are attainable but rare? \([Section3](https://arxiv.org/html/2609.02987#S3)\)
3. 3\.Third, does its alignment with Best\-of\-kkyield stronger inference\-time scaling? \([Section3\.1](https://arxiv.org/html/2609.02987#S3.SS1)\)
4. 4\.Finally, can TailRL prevent a common moderate\-reward behavior from displacing rarer, better outcomes? \([Section3\.1](https://arxiv.org/html/2609.02987#S3.SS1)and[Section3](https://arxiv.org/html/2609.02987#S3)\)

To answer these questions, we devise 4 experimental settings\. We test the population level objective and its finite\-rollouts approximation on a localization task in theImageNetdataset \([Section6\.1](https://arxiv.org/html/2609.02987#S6.SS1)\) \. To understand the behavior of TailRL under rare, high reward rollouts, we consider aText\-Mazesetting \([Section6\.2](https://arxiv.org/html/2609.02987#S6.SS2)\)\.GUI groundingallows us to study the scaling behavior of models trained with TailRL as more inference compute in poured into the problem \([Section6\.3](https://arxiv.org/html/2609.02987#S6.SS3)\) and finally,Code Optimizationtests resistance to a safe but suboptimal shortcut \([Section6\.4](https://arxiv.org/html/2609.02987#S6.SS4)\)\. Together, they test whether TailRL works for the reasons predicted by the theory, from the population objective to the behavior of the learned policy\.

We compare TailRL with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib36)\), RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib37)\), two popular group\-based policy optimization algorithms, and PKPO[Walder and Karkhanis \(2025\)](https://arxiv.org/html/2609.02987#bib.bib59)that maximizes Best\-of\-koptk\_\{\\mathrm\{opt\}\}for an inference budgetkoptk\_\{\\mathrm\{opt\}\}under pure on\-policy policy gradient setup to avoid confounders\.

### 6\.1ImageNet Object Localization

![Refer to caption](https://arxiv.org/html/2609.02987v1/figures/imagenet-overall-legend.png)Figure 4:\(ImageNet object localization\)An overview of the ImageNet object localization task\. A ResNet\-50 backbone\([He et al\., 2016](https://arxiv.org/html/2609.02987#bib.bib33)\)encodes the image, and four categorical heads parameterize the policy\. A rollout samples the box center coordinatesx^c,y^c\\hat\{x\}\_\{c\},\\hat\{y\}\_\{c\}, widthw^\\hat\{w\}, and heighth^\\hat\{h\}\. The sampled box is rewarded by its IoU with the matched ground\-truth box\.ImageNet object localization requires predicting a bounding box around an object of interest without classifying the object\([Russakovsky et al\., 2015](https://arxiv.org/html/2609.02987#bib.bib32)\)\. This setting addresses three questions\. First, how does reinforcement learning from a scalar reward compare with supervised objectives that directly observe the ground\-truth bounding box? Second, does the finite rollout TailRL gradient estimatorgTailRL\(N\)g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}approach the population\-level gradient∇θJTailRL\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}asNNincreases? Third, how does TailRL compare with the expected reward baselines at matched and smaller training rollout budgets?

##### Task and setup

For an input imagexx, a rolloutzzis a bounding box drawn from the categorical policy of[Fig\.4](https://arxiv.org/html/2609.02987#S6.F4), and the continuous rewardr⁡\(x,z\)∈\[0,1\]r\(x,z\)\\in\[0,1\]is its intersection\-over\-union \(IoU\) with the ground\-truth box\. We report CorLoc@δ\\delta, the fraction of input images for which the greedy prediction has IoU greater thanδ\\delta\([Deselaers et al\., 2010](https://arxiv.org/html/2609.02987#bib.bib60)\); mean IoU; and Best\-of\-kkIoU, the largest IoU amongkkinference rollouts\.

Its worthwhile to note that since the policy induces a categorical distribution over the finite set of possible bounding boxes, we can evaluate the probability and IoU reward of every box\. This gives us the exact reward distribution in closed form\. Using this we can directly compute and optimize the population\-level objectiveJTailRLJ\_\{\\mathrm\{TailRL\{\}\}\}\. We also train supervised baselines that directly regress the ground\-truth coordinates\. Training and evaluation details are provided in[AppendixH](https://arxiv.org/html/2609.02987#A8)\.

Figure 5:\(ImageNet object localization\)A comparison of population\-level TailRL versus direct supervision for the task of ImageNet Object Localization\. TailRL either outperforms or is competitive against task\-specific objectives\.
##### Comparison with direct supervision

Using only scalar IoU rewards, population\-level TailRL matches or exceeds objectives that directly supervise the ground\-truth coordinates \([Fig\.5](https://arxiv.org/html/2609.02987#S6.F5)\)\. Among the supervised objectives, the combined L1\+GIoU objective\([Rezatofighi et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib67)\)used by DETR\([Carion et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib66)\)is the strongest\. TailRL achieves higher CorLoc@0\.50\.5and CorLoc@0\.750\.75than L1\+GIoU while obtaining comparable mean IoU\. It also outperforms the individual L1 and GIoU baselines across all reported metrics throughout training\. These results show that under large training compute, optimizing a scalar continuous reward can compete with task\-specific supervised objectives\.

Takeaway 1When exact tail\-probabilities are estimable, the population\-level TailRL performs as well as or better than task\-specific supervised objectives that recieve ground truth information\.

Figure 6:\(ImageNet object localization\)Finite rollout TailRL gradients converge toward the population\-level gradient as training rollouts increase\.
##### Increasing training rollout budget

Increasing training rollouts,NNmoves the finite rollout TailRL curves toward the exact population\-level objective \([Fig\.7](https://arxiv.org/html/2609.02987#S6.F7)\)\. AtN=1024N=1024, TailRL closely tracks the population\-level objective, while smaller rollout budgets remain progressively farther away\. This ordering holds across CorLoc@0\.50\.5, mean IoU, and Best\-of\-10241024IoU and other performance measures that we test for this experiment\. We test this convergence directly at the gradient level in[Fig\.6](https://arxiv.org/html/2609.02987#S6.F6)\. AsNNincreases, the cosine similarity between the sampled finite rollout gradient and the population\-level gradient rises steadily toward exact agreement\. This increasing alignment is consistent with the convergence predicted in[Section4\.1](https://arxiv.org/html/2609.02987#S4.SS1)\. Increasing rollouts not only reduce the variance of the objective we are trying to estimate, but it also approximates a higher order truncated objective\.

Takeaway 2As training rollouts increase, the gradient of order\-NNtruncated objective computed from finite rollout TailRL converges to the gradient of population\-level TailRL\.

##### Comparison with RL baselines

At a matched budget ofN=1024N=1024, TailRL outperforms GRPO and RLOO across all three metrics, despite every method receiving the same continuous IoU feedback\. The gap does not close at smaller budgets: even atN=16N=16, TailRL exceeds both baselines trained atN=1024N=1024, using164\\tfrac\{1\}\{64\}as many training rollouts per input\. TailRL also pareto dominates PKPO which maximizes expected maximum reward\. With justN=16N=16training rollouts, it outperforms at CorLoc@0\.50\.5and mean IoU\. At matched training compute, TailRL matches PKPO at Best\-of\-10241024reward, while significantly outperforming at mean IoU and CorLoc@0\.50\.5\.

Figure 7:\(ImageNet object localization\)Comparing performance of expected\-reward maximization baselines with TailRL on held\-out validation set\. We report CorLoc@0\.50\.5, mean IoU, and Best\-of\-10241024IoU\. TailRL uses training rolloutsN∈\{16,64,256,1024\}N\\in\\\{16,64,256,1024\\\}, with darker curves indicating largerNN; the dashed curve optimizes the exactly computed TailRL population\-level objective\. GRPO and RLOO useN=1024N=1024\([AppendixH](https://arxiv.org/html/2609.02987#A8)\)\.Figure 8:\(Text\-Maze\)Four rollouts on one Text\-Maze and the continuous reward each receives\. The two left paths fail to reach the goal and still receive continuous credit for progress\. The third reaches the goal but wanders, so it gets rewarded below a shortest path\. Only the rightmost, a shortest path \(right\), earns reward11\.

### 6\.2Text\-Maze Navigation from Low\-Success Initial Policies

Next we examine how TailRL and other baselines compare when we vary the quality of the initialization policy\. By varying the amount of supervised pretraining before doing RL post\-training, we obtain a controlled range of initial policies with varying coverage over high reward attaining rollouts\. We ask whether TailRL can learn from poor initialization of policies and how it compares against expected reward maximization baselines\.

##### Task and setup

For an input17×1717\\times 17maze represented as text, a rollout is a token sequence describing a path\. The continuous rewardr⁡\(x,z\)∈\[0,1\]r\(x,z\)\\in\[0,1\]measures proximity to the goal and path length relative to the shortest path \([Fig\.8](https://arxiv.org/html/2609.02987#S6.F8)\)\. A rollout receives reward11only when it reaches the goal along a shortest path, an event we call*shortest\-path success*\. Unsuccessful rollouts receive partial credit for ending closer to the goal, while successful rollouts receive more reward for shorter paths\. We evaluate the post\-trained policies on a held\-out validation set of10241024mazes\.

By varying the amount of supervised pretraining on goal\-reaching trajectories, we obtain initial policies with shortest\-path success rates ranging from approximately1%1\\%down to0\.01%0\.01\\%\. Starting from each checkpoint, we separately post\-train policies with TailRL, GRPO, and PKPO \(koptk\_\{\\mathrm\{opt\}\}\)\. All methods use the same initial policy andN=16N=16unless stated otherwise\. Model, reward, training, and evaluation details are provided in[AppendixI](https://arxiv.org/html/2609.02987#A9)\.

Figure 9:\(Text\-Maze\)Pass@11before and after RL post\-training for the task of Text\-Maze navigation\. Shaded area marks the regime where initialization policy is poor\. After RL post\-training, RLOO and GRPO fail to reliably improve Pass@kkin this regime\.
##### Learning from low\-success initial policies

[Figure9](https://arxiv.org/html/2609.02987#S6.F9)shows that RLOO, GRPO and TailRL behave similarly when the initial policy produces shortest\-path successes frequently, but diverge as the initial success rate falls\. Despite higher coverage of initial policy, PKPO underperforms TailRL at Pass@11\. The expected reward baselines fail to learn reliably in the low initial success regime\. PKPO consistently improves Pass@11across the spectrum of initialization policy yet still under\-performs TailRL at Pass@11even at the regime of poor policy initalization\. Once the initial success rate rises beyond this regime, TailRL, RLOO and GRPO learn well and their differences narrow\. GRPO’s performance improves before RLOO, although the performance becomes seed dependent before it consistently starts to navigate the maze\. The advantage of TailRL is therefore concentrated where high\-reward rollouts are attainable but rare\. Inference and training\-rollout budget sweeps are reported in[SectionI\.5](https://arxiv.org/html/2609.02987#A9.SS5)\.

Takeaway 3TailRL upweighs gradients for rare exceptional rollouts\. This results in TailRL outperforming other methods when initial policy has poor coverage over rare excellent rollouts\.

### 6\.3GUI Grounding with Vision\-Language Models

We next demonstrate the efficacy of TailRL on Vision Language Models \(VLM\)\. We consider a visual grounding task with verifiable rewards\. Through this task, we show that models trained with TailRL benefit from inference\-time sampling\.

Task and setup:Given a screenshot and a natural\-language instruction to perform a click, the VLM policy generates the coordinates of the click location\. The continuous rewardr⁡\(x,z\)∈\[0,2\.5\]r\(x,z\)\\in\[0,2\.5\]combines proximity to the target, a bonus for clicking inside the target element, and a format bonus\([Yuan et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib70)\)\. We fine\-tune Qwen2\.5\-VL\-3B and Qwen2\.5\-VL\-7B\([Bai et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib68)\)on the GTA1 grounding corpus\([Yang et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib69)\)\. The training configuration is identical across TailRL, GRPO, and RLOO except for the advantage estimator\. We evaluate the resulting policies on ScreenSpot\-Pro\([Li et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib71)\)\. Here, Pass@kkis the probability that at least one ofkkinference rollouts clicks inside the target element\([Chen et al\., 2021](https://arxiv.org/html/2609.02987#bib.bib51)\), while Best\-of\-kkreward is the largest continuous reward among those rollouts\. The reward construction, training protocol and evaluation procedure are detailed in[AppendixJ](https://arxiv.org/html/2609.02987#A10)\.

Results:At both model scales, TailRL and RLOO achieve similar Pass@11, but they substantially differ in how the learned policies respond to additional inference sampling \([Fig\.10](https://arxiv.org/html/2609.02987#S6.F10)\)\. As inference rolloutskkincreases, TailRL’s Pass@kkcontinues to rise, whereas RLOO plateaus considerably earlier; GRPO’s Pass@kkis inferior to TailRL and RLOO\. TailRL leaves more inputs with a non\-negligible probability of producing a successful click, so additional samples continue to reveal useful candidates\.

We define the*matching budget*as the smallest evaluated value ofkkat which TailRL reaches or exceeds a baseline’s mean Pass@10241024\. On ScreenSpot\-Pro, at 3B, TailRL reaches RLOO’s Pass@10241024with88rather than10241024inference rollouts, a128128\-fold reduction; at 7B it does so with44rollouts, a256256\-fold reduction\. A single TailRL rollout exceeds GRPO’s mean Pass@10241024at both model scales\.

Takeaway 4On GUI grounding, TailRL shows superior scaling of inference compute when compared against expected reward maximization baselines\.

### 6\.4Code Runtime Optimization

Figure 10:\(GUI\-grounding\)Evaluation results for GUI grounding task on ScreenSpot\-Pro benchmark for Qwen2\.5\-VL\-3B \(top\) and 7B \(bottom\)\. Columns show Pass@kk, Best\-of\-kkreward, and exponentially smoothed training\-batch accuracy\. TailRL matches RLOO’s Pass@10241024using88rollouts at 3B and44rollouts at 7B scale, a reduction of128×128\\timesand256×256\\timesin test\-time compute respectively\.We now ask a qualitatively different question: what happens when training is attracted to a safe but systematically suboptimal behavior hurting further exploration? Such behaviors provide a reliable moderate reward and can become stable solutions, even when rarer and riskier behaviors offer substantially better outcomes\([Skalse et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib103);[Baronio et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib102)\)\. This creates a stress test for TailRL: whether it concentrates the policy on the dependable yet suboptimal shortcuts or improves the policy’s coverage over high\-reward tail\.

Figure 11:\(Code runtime optimization\)Initial reward distribution over all test problems of PIE dataset for Qwen3\-1\.7B\.74\.4%74\.4\\%of rollouts are incorrect,23\.5%23\.5\\%are correct but not faster than the input, concentrating in a spike at1×1\\times, and2\.1%2\.1\\%are correct and faster\.##### Task and setup

Each input is a slow C\+\+ program from the PIE\([Shypula et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib72)\)corpus of competitive\-programming\. The LLM is prompted to write a faster yet correct version of the input program\. A rollout is a rewritten program and is compiled and executed against the problem’s test suite\. Incorrect outputs receives zero reward otherwise receives a reward equal to the speedup over the input program\. We measure speedup using gem5 time so that timing noise cannot create spurious improvements\([Binkert et al\., 2011](https://arxiv.org/html/2609.02987#bib.bib88);[Lowe\-Power et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib101)\)\. We post\-train Qwen3\-1\.7B\([Yang et al\., 2025a](https://arxiv.org/html/2609.02987#bib.bib2)\)with TailRL, GRPO, and RLOO usingN=16N=16rollouts per program\.

We report mean reward over all rollouts; density of Best\-of\-kkrewards atk=1024k=1024; and the fraction of rollouts that pass every test; and Best\-of\-kkreward density atk=1024k=1024\. A policy that always reproduces its input obtains a mean reward of11and passes all test cases\. Full task, training, and evaluation details are provided in[AppendixK](https://arxiv.org/html/2609.02987#A11)\.

Figure 12:\(Code runtime optimization\)\(Left\) Average reward during training for 1 epoch, EMA over three runs per objective\. \(Middle\) kernel density of the per\-problem Best\-of\-10241024reward on the test\-set problems of PIE\. GRPO and RLOO are drawn with alternating dashes because their densities coincide at a reward of 1\.0 indicating that they only echo the input program\. Right: fraction of training rollouts for a given batch passing every test\. In this task, it is better to explore and find the faster rewrite of the input program than to copy the input and be correct\. Correctness only gets a reward of1\.01\.0while correct and meaningfully faster programs receive much higher rewards\. Just copying the input is a degenerate soltuion exhibited by RLOO and GRPO[SectionK\.7](https://arxiv.org/html/2609.02987#A11.SS7)All experiments have been performed on 3 seeds per method\.
##### Results

GRPO and RLOO rapidly converge toward the copying shortcut \([Fig\.12](https://arxiv.org/html/2609.02987#S6.F12)\) as evidenced by sample rollouts \([SectionK\.7](https://arxiv.org/html/2609.02987#A11.SS7)\)\. Their correctness rise above98%98\\%, while their mean rewards settle just below1\.01\.0\. Their policy entropy collapses by one to two orders of magnitude over the same interval \([Fig\.26](https://arxiv.org/html/2609.02987#A11.F26)\)\. Expected\-reward training therefore converges to the most reliable mode of the reward distribution while eliminating the behavior to maximize coverage over excellent outputs\.

TailRL follows a different trajectory\. Rather than collapsing onto the guaranteed reward from copying the input, it maintains substantially higher entropy and continues producing risky rewrites\. This lowers single\-rollout correctness, but places more probability on programs that are both correct and meaningfully faster, raising mean reward to2\.922\.92, nearly three times the copying value\. All methods use the same one\-epoch training budget, and TailRL is still improving at step300300; these are therefore matched\-compute results rather than converged endpoints\. On the held\-out test set the trained policies separate sharply: TailRL’s mean Best\-of\-10241024speedup is7\.7×7\.7\\timesagainst0\.98×0\.98\\timesfor GRPO and0\.96×0\.96\\timesfor RLOO, whose best rollouts never beat the input they reproduce \([Fig\.12](https://arxiv.org/html/2609.02987#S6.F12), middle;[Fig\.26](https://arxiv.org/html/2609.02987#A11.F26)\)\.

Takeaway 5When an easy suboptimal solution exists, expected reward maximization methods settle for it\. TailRL keeps searching and learns to produce rarer and better outcomes\.

## 7Related Work

##### Objectives beyond expected reward

Reinforcement\-learning post\-training for reasoning and agentic models still largely maximizes expected reward\([Ouyang et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib6);[Guo et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib30)\)\. GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib36)\)and RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib37)\)are practical, critic\-free variants of the score\-function policy gradient\([Williams, 1992](https://arxiv.org/html/2609.02987#bib.bib48)\)\. RLOO changes the baseline, while GRPO applies a common normalization to the update for each input; neither changes the relative weighting of reward levels within that input \([Section5](https://arxiv.org/html/2609.02987#S5)\)\. TailRL changes the objective instead\. Closest to our work, MaxRL maximizes the log\-probability of success for binary rewards and decomposes its gradient into a harmonic sum of Pass@kkgradients\([Tajwar et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib39)\)\. PKPO derives unbiased estimators for a chosen Pass@kkobjective and its continuous counterpart, Best\-of\-kk\([Walder and Karkhanis, 2025](https://arxiv.org/html/2609.02987#bib.bib59)\); related methods directly optimize a selected inference\-time metric\([Tang et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib65);[Chen et al\., 2025b](https://arxiv.org/html/2609.02987#bib.bib5)\)or study objectives defined by monotone transforms of success probability\([Davis and Recht, 2025](https://arxiv.org/html/2609.02987#bib.bib19)\)\. These approaches select a threshold, transform, or inference budget\. TailRL instead integrates log tail\-probability over all reward thresholds, yielding harmonic combinations of Best\-of\-kkobjectives and recovering MaxRL exactly for binary rewards\.

##### Coverage and tail\-sensitive reinforcement learning

Expected\-reward training can narrow the policy distribution and remove rare, high\-reward behavior, an effect studied mechanistically\([Cui et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib80)\)and observed at large sampling budgets\([Yue et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib8);[Kirk et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib81)\)\. Rather than adding a separate diversity regularizer, TailRL addresses this through the objective itself by weighting each reward level inversely by how often the policy reaches it\. TailRL is also distinct from distributional and risk\-sensitive reinforcement learning, which learn a return distribution through a critic or optimize a tail statistic at a chosen risk level\([Bellemare et al\., 2017](https://arxiv.org/html/2609.02987#bib.bib84);[Rockafellar and Uryasev, 2000](https://arxiv.org/html/2609.02987#bib.bib86)\)\. TailRL uses no learned critic: its policy\-gradient weights are computed directly from each rollout group, and it integrates over all reward thresholds rather than fixing one risk level, which can otherwise overlook rare successes\([Greenberg et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib98)\)\. We discuss further connections to exploration, sample allocation, threshold decompositions of continuous rewards, and the evaluated domains in[AppendixB](https://arxiv.org/html/2609.02987#A2)\.

## 8Conclusion

We introduced TailRL, a likelihood objective for continuous rewards that maximizes the expected log\-probability of exceeding a uniformly sampled reward threshold\. It recovers MaxRL exactly for binary rewards, decomposes harmonically over Best\-of\-kkgradients, and admits a simple, critic\-free finite\-rollout estimator\. Across four settings, its gains were largest when high\-reward rollouts were rare or when training was drawn toward a common but suboptimal behavior\. In these regimes, TailRL learned more reliably from rare outcomes, benefited more from additional training and inference samples, and avoided suboptimal collapse\. More broadly, a continuous reward is more than a scalar to average: it defines a family of success events, one at every reward level\. TailRL provides a practical way to optimize their likelihoods\.

## Acknowledgements

This research used the DeltaAI advanced computing and data resource\([Bode et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib105)\), which is supported by the National Science Foundation under award OAC\-2320345 and by the State of Illinois\. DeltaAI is a joint effort of the University of Illinois Urbana\-Champaign and its National Center for Supercomputing Applications\. These resources were used through the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program\([Boerner et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib104)\)\. Overall, this project used ACCESS allocations CIS250426, CIS260353, CIS260522, CIS260557, CIS260677, CIS260678, and CIS260679\. We are especially grateful to Brett Bode of the NCSA Delta Support team, whose assistance in effectively using the Delta cluster was critical to completing this work on schedule\. We would like to extend our sincere gratitude to Sooth Labs for their generous support by granting us their compute resources\. This work was partially supported by the National Science Foundation under Grants CCF\-2106778\. Fahim and Shrinivas were funded in part by Stack AV and Skylark Labs

We are grateful to Sumukh Aithal, Ben Freed, Surgan Jandial, Sreyas Venkatraman, Kartik Sharma, Elton Lobo, Parv Maheshwari, Rhea Basappa, Srinath Ravi, Rishubh Parihar, Harsh Rangwani, Chaitanya Chawla, and Mayank Mishra for carefully reviewing earlier drafts and providing valuable feedback\. We also thank Rohit Sonkar, Anoushka Alavilli, Jiayu Chen, and Mineui Hong from the CMU Auton Lab, and Lawrence Jang from Russ lab for their helpful feedback that improved the quality of the draft\. The authors would also like to extend their gratitude to Prof\. Yaser Sheikh and Prof\. Yonatan Bisk for helpful discussions and suggestions throughout this work\.

## References

- Agarwalet al\.\(2021\)A\. Agarwal, S\. M\. Kakade, J\. D\. Lee, and G\. MahajanOn the theory of policy gradient methods: optimality, approximation, and distribution shift\.Journal of Machine Learning Research22\(98\),pp\. 1–76\.Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p4.1)\.
- Ahmadianet al\.\(2024\)A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 12248–12267\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.662),2402\.14740Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p1.1),[§2](https://arxiv.org/html/2609.02987#S2.p3.1),[§6](https://arxiv.org/html/2609.02987#S6.p3.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. LinQwen2\.5\-VL technical report\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923)Cited by:[§J\.1](https://arxiv.org/html/2609.02987#A10.SS1.p1.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.3](https://arxiv.org/html/2609.02987#S6.SS3.p2.1)\.
- Baronioet al\.\(2025\)C\. Baronio, P\. Marsella, B\. Pan, S\. Guo, and S\. AlbertiKevin: multi\-turn RL for generating CUDA kernels\.External Links:2507\.11948,[Link](https://arxiv.org/abs/2507.11948)Cited by:[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.p1.1)\.
- Bellemareet al\.\(2017\)M\. G\. Bellemare, W\. Dabney, and R\. MunosA distributional perspective on reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.70,pp\. 449–458\.External Links:1707\.06887Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Bellemareet al\.\(2023\)M\. G\. Bellemare, W\. Dabney, and M\. RowlandDistributional reinforcement learning\.MIT Press\.External Links:[Document](https://dx.doi.org/10.7551/mitpress/14207.001.0001)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Bellemareet al\.\(2016\)M\. G\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. MunosUnifying count\-based exploration and intrinsic motivation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 1471–1479\.External Links:1606\.01868Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Binkertet al\.\(2011\)N\. Binkert, B\. Beckmann, G\. Black, S\. K\. Reinhardt, A\. Saidi, A\. Basu, J\. Hestness, D\. R\. Hower, T\. Krishna, S\. Sardashti, R\. Sen, K\. Sewell, M\. Shoaib, N\. Vaish, M\. D\. Hill, and D\. A\. WoodThe gem5 simulator\.ACM SIGARCH Computer Architecture News39\(2\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1145/2024716.2024718)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.SSS0.Px1.p1.1)\.
- Bodeet al\.\(2025\)B\. Bode, G\. Bauer, L\. Herriott, V\. Kindratenko, and W\. GroppDeltaAI: a national resource for ai/ml research\.InPractice and Experience in Advanced Research Computing 2025: The Power of Collaboration,PEARC ’25,New York, NY, USA\.External Links:ISBN 9798400713989,[Link](https://doi.org/10.1145/3708035.3736062),[Document](https://dx.doi.org/10.1145/3708035.3736062)Cited by:[Acknowledgements](https://arxiv.org/html/2609.02987#Sx1.p1.1)\.
- Boerneret al\.\(2023\)T\. J\. Boerner, S\. Deems, T\. R\. Furlani, S\. L\. Knuth, and J\. TownsACCESS: advancing innovation: nsf’s advanced cyberinfrastructure coordination ecosystem: services & support\.InPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good,PEARC ’23,New York, NY, USA,pp\. 173–176\.External Links:[Document](https://dx.doi.org/10.1145/3569951.3597559),[Link](https://doi.org/10.1145/3569951.3597559)Cited by:[Acknowledgements](https://arxiv.org/html/2609.02987#Sx1.p1.1)\.
- Brownet al\.\(2024\)B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. MirhoseiniLarge language monkeys: scaling inference compute with repeated sampling\.External Links:2407\.21787,[Link](https://arxiv.org/abs/2407.21787)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Burdaet al\.\(2019\)Y\. Burda, H\. Edwards, A\. Storkey, and O\. KlimovExploration by random network distillation\.InInternational Conference on Learning Representations \(ICLR\),External Links:1810\.12894Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Caoet al\.\(2020\)W\. Cao, V\. Mirjalili, and S\. RaschkaRank consistent ordinal regression for neural networks with application to age estimation\.Pattern Recognition Letters140,pp\. 325–331\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1),[Appendix E](https://arxiv.org/html/2609.02987#A5.p1.1)\.
- Carionet al\.\(2020\)N\. Carion, F\. Massa, G\. Synnaeve, N\. Usunier, A\. Kirillov, and S\. ZagoruykoEnd\-to\-end object detection with transformers\.InEuropean Conference on Computer Vision \(ECCV\),Vol\.12346,pp\. 213–229\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§H\.1](https://arxiv.org/html/2609.02987#A8.SS1.p1.3),[§6\.1](https://arxiv.org/html/2609.02987#S6.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§J\.6](https://arxiv.org/html/2609.02987#A10.SS6.p2.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1),[§G\.1](https://arxiv.org/html/2609.02987#A7.SS1.p1.1),[§6\.3](https://arxiv.org/html/2609.02987#S6.SS3.p2.1)\.
- Chenet al\.\(2025a\)X\. Chen, J\. Lu, M\. Kim, D\. Zhang, J\. Tang, A\. Piché, N\. Gontier, Y\. Bengio, and E\. KamallooSelf\-evolving curriculum for LLM reasoning\.External Links:2505\.14970,[Link](https://arxiv.org/abs/2505.14970)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Chenet al\.\(2025b\)Z\. Chen, X\. Qin, Y\. Wu, Y\. Ling, Q\. Ye, W\. X\. Zhao, and G\. ShiPass@k training for adaptively balancing exploration and exploitation of large reasoning models\.External Links:2508\.10751,[Link](https://arxiv.org/abs/2508.10751)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Chenget al\.\(2026\)D\. Cheng, S\. Huang, X\. Zhu, B\. Dai, W\. X\. Zhao, Z\. Zhang, and F\. WeiReasoning with exploration: an entropy perspective\.InAAAI Conference on Artificial Intelligence \(AAAI\),Vol\.40,pp\. 30377–30385\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i36.40290),2506\.14758Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Chowet al\.\(2018\)Y\. Chow, M\. Ghavamzadeh, L\. Janson, and M\. PavoneRisk\-constrained reinforcement learning with percentile risk criteria\.Journal of Machine Learning Research18\(167\),pp\. 1–51\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Clavieret al\.\(2022\)P\. Clavier, S\. Allassonnière, and E\. L\. PennecRobust reinforcement learning with distributional risk\-averse formulation\.External Links:2206\.06841,[Link](https://arxiv.org/abs/2206.06841)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen, Z\. Liu, H\. Peng, L\. Bai, W\. Ouyang, Y\. Cheng, B\. Zhou, and N\. DingThe entropy mechanism of reinforcement learning for reasoning language models\.External Links:2505\.22617,[Link](https://arxiv.org/abs/2505.22617)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Dabneyet al\.\(2018a\)W\. Dabney, G\. Ostrovski, D\. Silver, and R\. MunosImplicit quantile networks for distributional reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1096–1105\.External Links:1806\.06923Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Dabneyet al\.\(2018b\)W\. Dabney, M\. Rowland, M\. G\. Bellemare, and R\. MunosDistributional reinforcement learning with quantile regression\.InAAAI Conference on Artificial Intelligence \(AAAI\),pp\. 2892–2901\.External Links:1710\.10044Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Daiet al\.\(2025\)R\. Dai, L\. Song, H\. Liu, Z\. Liang, D\. Yu, H\. Mi, Z\. Tu, R\. Liu, T\. Zheng, H\. Zhu, and D\. YuCDE: curiosity\-driven exploration for efficient reinforcement learning in large language models\.External Links:2509\.09675,[Link](https://arxiv.org/abs/2509.09675)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Danget al\.\(2025\)X\. Dang, C\. Baek, J\. Z\. Kolter, and A\. RaghunathanAssessing diversity collapse in reasoning\.InScaling Self\-Improving Foundation Models without Human Supervision,External Links:[Link](https://openreview.net/forum?id=AMiKsHLjQh)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1)\.
- Davis and Recht \(2025\)D\. Davis and B\. RechtWhat is the objective of reasoning with reinforcement learning?\.External Links:2510\.13651,[Link](https://arxiv.org/abs/2510.13651)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Deselaerset al\.\(2010\)T\. Deselaers, B\. Alexe, and V\. FerrariLocalizing objects while learning their appearance\.InEuropean Conference on Computer Vision \(ECCV\),pp\. 452–466\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.1](https://arxiv.org/html/2609.02987#S6.SS1.SSS0.Px1.p1.1)\.
- Douet al\.\(2024\)S\. Dou, Y\. Liu, H\. Jia, E\. Zhou, L\. Xiong, J\. Shan, C\. Huang, X\. Wang, X\. Fan, Z\. Xi, Y\. Zhou, T\. Ji, R\. Zheng, Q\. Zhang, T\. Gui, and X\. HuangStepCoder: improving code generation with reinforcement learning from compiler feedback\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Bangkok, Thailand,pp\. 4571–4585\.Note:arXiv:2402\.01391External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.251)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Frank and Hall \(2001\)E\. Frank and M\. HallA simple approach to ordinal classification\.InMachine Learning: ECML 2001,Lecture Notes in Computer Science, Vol\.2167,pp\. 145–156\.External Links:[Document](https://dx.doi.org/10.1007/3-540-44795-4%5F13)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1),[Appendix E](https://arxiv.org/html/2609.02987#A5.p1.1)\.
- Gaiet al\.\(2025\)J\. Gai, G\. Zeng, H\. Zhang, and A\. RaghunathanDifferential smoothing mitigates sharpening and improves LLM reasoning\.External Links:2511\.19942,[Link](https://arxiv.org/abs/2511.19942)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational Conference on Machine Learning \(ICML\),pp\. 10835–10866\.External Links:2210\.10760Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Gehringet al\.\(2025\)J\. Gehring, K\. Zheng, J\. Copet, V\. Mella, T\. Cohen, and G\. SynnaeveRLEF: grounding code LLMs in execution feedback with reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 19034–19055\.External Links:2410\.02089Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Glynn \(1990\)P\. W\. GlynnLikelihood ratio gradient estimation for stochastic systems\.Communications of the ACM33\(10\),pp\. 75–84\.Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p2.1)\.
- Greenberget al\.\(2022\)I\. Greenberg, Y\. Chow, M\. Ghavamzadeh, and S\. MannorEfficient risk\-averse reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2205\.05138Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),2501\.12948Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- GX\-Chenet al\.\(2025\)A\. GX\-Chen, J\. Prakash, J\. Guo, R\. Fergus, and R\. RanganathKL\-regularized reinforcement learning is designed to mode collapse\.External Links:2510\.20817,[Link](https://arxiv.org/abs/2510.20817)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Hamidet al\.\(2026\)J\. I\. Hamid, I\. H\. Orney, E\. Xu, C\. Finn, and D\. SadighPolychromic objectives for reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),External Links:2509\.25424Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Haoet al\.\(2026\)Z\. Hao, H\. Wang, H\. Liu, J\. Luo, J\. Yu, H\. Dong, Q\. Lin, C\. Wang, and J\. ChenRethinking entropy interventions in RLVR: an entropy change perspective\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 31105–31133\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1436),2510\.10150Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90),1512\.03385Cited by:[Appendix H](https://arxiv.org/html/2609.02987#A8.p2.1),[Figure 4](https://arxiv.org/html/2609.02987#S6.F4),[Figure 4](https://arxiv.org/html/2609.02987#S6.F4.5.1)\.
- Huet al\.\(2026\)J\. Hu, M\. Liu, X\. Lu, F\. Wu, Z\. Harchaoui, S\. Diao, Y\. Choi, P\. Molchanov, J\. Yang, J\. Kautz, and Y\. DongBroRL: scaling reinforcement learning via broadened exploration\.InInternational Conference on Machine Learning \(ICML\),External Links:2510\.01180Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Jianget al\.\(2026\)Y\. Jiang, J\. Huang, Y\. Yuan, X\. Mao, Y\. Yue, Q\. Zhao, and L\. YanRisk\-sensitive reinforcement learning for alleviating exploration dilemmas in large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2509\.24261Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Kaddour \(2026\)J\. KaddourTarget policy optimization\.External Links:2604\.06159,[Link](https://arxiv.org/abs/2604.06159)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Kirket al\.\(2024\)R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina, E\. Hambro, E\. Grefenstette, and R\. RaileanuUnderstanding the effects of RLHF on LLM generalisation and diversity\.InInternational Conference on Learning Representations \(ICLR\),External Links:2310\.06452Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Koolet al\.\(2019\)W\. Kool, H\. van Hoof, and M\. WellingBuy 4 REINFORCE samples, get a baseline for free\!\.InDeep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop,External Links:[Link](https://openreview.net/forum?id=r1lgTGL5DE)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.InConference on Language Modeling \(COLM\),External Links:2411\.15124Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Leet al\.\(2022\)H\. Le, Y\. Wang, A\. D\. Gotmare, S\. Savarese, and S\. C\.H\. HoiCodeRL: mastering code generation through pretrained models and deep reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2207\.01780Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Liet al\.\(2025\)K\. Li, Z\. Meng, H\. Lin, Z\. Luo, Y\. Tian, J\. Ma, Z\. Huang, and T\. ChuaScreenSpot\-Pro: GUI grounding for professional high\-resolution computer use\.InProceedings of the 33rd ACM International Conference on Multimedia \(MM 2025\),pp\. 8778–8786\.External Links:[Document](https://dx.doi.org/10.1145/3746027.3755688),2504\.07981Cited by:[§J\.6](https://arxiv.org/html/2609.02987#A10.SS6.p1.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.3](https://arxiv.org/html/2609.02987#S6.SS3.p2.1)\.
- Li and Lin \(2006\)L\. Li and H\. LinOrdinal regression by extended binary classification\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.19,pp\. 865–872\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1),[Appendix E](https://arxiv.org/html/2609.02987#A5.p1.1)\.
- Liet al\.\(2022\)Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. Sutherland Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. VinyalsCompetition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Document](https://dx.doi.org/10.1126/science.abq1158)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, Y\. Zhu, K\. Xiao, Q\. Fu, X\. Han, W\. Yang, and D\. YeRLTF: reinforcement learning from unit test feedback\.Transactions on Machine Learning Research\.Note:arXiv:2307\.04349Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, Z\. Liu, S\. Zhu, P\. Li, C\. Xie, J\. Wang, X\. Hu, X\. Han, J\. Yuan, X\. Wang, S\. Zhang, H\. Yang, and F\. WuInfiGUI\-G1: advancing GUI grounding with adaptive exploration policy optimization\.InAAAI Conference on Artificial Intelligence \(AAAI\),pp\. 32267–32275\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i38.40500),2508\.05731Cited by:[§J\.5](https://arxiv.org/html/2609.02987#A10.SS5.p1.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding R1\-Zero\-like training: a critical perspective\.InConference on Language Modeling \(COLM\),External Links:2503\.20783Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Lowe\-Poweret al\.\(2020\)J\. Lowe\-Power, A\. M\. Ahmad, A\. Akram, M\. Alian, R\. Amslinger, M\. Andreozzi, A\. Armejach, N\. Asmussen, B\. Beckmann, S\. Bharadwaj, G\. Black, G\. Bloom, B\. R\. Bruce, D\. R\. Carvalho, J\. Castrillon, L\. Chen, N\. Derumigny, S\. Diestelhorst, W\. Elsasser, C\. Escuin, M\. Fariborz, A\. Farmahini\-Farahani, P\. Fotouhi, R\. Gambord, J\. Gandhi, D\. Gope, T\. Grass, A\. Gutierrez, B\. Hanindhito, A\. Hansson, S\. Haria, A\. Harris, T\. Hayes, A\. Herrera, M\. Horsnell, S\. A\. R\. Jafri, R\. Jagtap, H\. Jang, R\. Jeyapaul, T\. M\. Jones, M\. Jung, S\. Kannoth, H\. Khaleghzadeh, Y\. Kodama, T\. Krishna, T\. Marinelli, C\. Menard, A\. Mondelli, M\. Moreto, T\. Mück, O\. Naji, K\. Nathella, H\. Nguyen, N\. Nikoleris, L\. E\. Olson, M\. Orr, B\. Pham, P\. Prieto, T\. Reddy, A\. Roelke, M\. Samani, A\. Sandberg, J\. Setoain, B\. Shingarov, M\. D\. Sinclair, T\. Ta, R\. Thakur, G\. Travaglini, M\. Upton, N\. Vaish, I\. Vougioukas, W\. Wang, Z\. Wang, N\. Wehn, C\. Weis, D\. A\. Wood, H\. Yoon, and É\. F\. ZulianThe gem5 simulator: version 20\.0\+\.External Links:2007\.03152,[Link](https://arxiv.org/abs/2007.03152)Cited by:[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.SSS0.Px1.p1.1)\.
- L’Ecuyer \(1995\)P\. L’EcuyerNote: on the interchange of derivative and expectation for likelihood ratio derivative estimators\.Management Science41\(4\),pp\. 738–747\.Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p2.1)\.
- McCullagh \(1980\)P\. McCullaghRegression models for ordinal data\.Journal of the Royal Statistical Society: Series B \(Methodological\)42\(2\),pp\. 109–127\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1),[Appendix E](https://arxiv.org/html/2609.02987#A5.p1.1)\.
- Mohamedet al\.\(2020\)S\. Mohamed, M\. Rosca, M\. Figurnov, and A\. MnihMonte Carlo gradient estimation in machine learning\.Journal of Machine Learning Research21\(132\),pp\. 1–62\.Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p2.1)\.
- Nguyenet al\.\(2025\)P\. M\. Nguyen, C\. D\. La, D\. M\. H\. Nguyen, N\. V\. Chawla, B\. T\. Nguyen, and K\. D\. DoanThe reasoning boundary paradox: how reinforcement learning constrains language models\.External Links:2510\.02230,[Link](https://arxiv.org/abs/2510.02230)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Niuet al\.\(2016\)Z\. Niu, M\. Zhou, L\. Wang, X\. Gao, and G\. HuaOrdinal regression with multiple output CNN for age estimation\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4920–4928\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1),[Appendix E](https://arxiv.org/html/2609.02987#A5.p1.1)\.
- OpenAIet al\.\(2024\)OpenAI, A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Orneyet al\.\(2026\)I\. H\. Orney, J\. I\. Hamid, S\. S\. Ramanujam, S\. Wu, H\. Hu, N\. Goodman, D\. Sadigh, and C\. FinnPoly\-EPO: training exploratory reasoning models\.External Links:2604\.17654,[Link](https://arxiv.org/abs/2604.17654)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Osband \(2026\)I\. OsbandDelightful policy gradient\.External Links:2603\.14608,[Link](https://arxiv.org/abs/2603.14608)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35\.External Links:2203\.02155Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Papiniet al\.\(2018\)M\. Papini, D\. Binaghi, G\. Canonaco, M\. Pirotta, and M\. RestelliStochastic variance\-reduced policy gradient\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p4.1)\.
- Parmaset al\.\(2026\)P\. Parmas, Y\. Kim, K\. Matsutani, S\. Takashiro, S\. Nishimori, T\. Kojima, Y\. Iwasawa, and Y\. MatsuoOrderGrad: optimizing beyond the mean with order\-statistic policy gradient estimation\.External Links:2606\.06096,[Link](https://arxiv.org/abs/2606.06096)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Patelet al\.\(2026\)N\. Patel, F\. Wang, and I\. S\. DhillonODRPO: ordinal decompositions of discrete rewards for robust policy optimization\.External Links:2605\.12667,[Link](https://arxiv.org/abs/2605.12667)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1)\.
- Pathaket al\.\(2017\)D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. DarrellCuriosity\-driven exploration by self\-supervised prediction\.InInternational Conference on Machine Learning \(ICML\),pp\. 2778–2787\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Quet al\.\(2026\)Y\. Qu, A\. Setlur, V\. Smith, R\. Salakhutdinov, and A\. KumarPOPE: learning to reason on hard problems via privileged on\-policy exploration\.External Links:2601\.18779,[Link](https://arxiv.org/abs/2601.18779)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Rezatofighiet al\.\(2019\)H\. Rezatofighi, N\. Tsoi, J\. Gwak, A\. Sadeghian, I\. Reid, and S\. SavareseGeneralized intersection over union: a metric and a loss for bounding box regression\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§H\.1](https://arxiv.org/html/2609.02987#A8.SS1.p1.2),[Appendix H](https://arxiv.org/html/2609.02987#A8.p3.1),[§6\.1](https://arxiv.org/html/2609.02987#S6.SS1.SSS0.Px2.p1.1)\.
- Rockafellar and Uryasev \(2000\)R\. T\. Rockafellar and S\. UryasevOptimization of conditional value\-at\-risk\.Journal of Risk2\(3\),pp\. 21–41\.External Links:[Document](https://dx.doi.org/10.21314/JOR.2000.038)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Russakovskyet al\.\(2015\)O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein, A\. C\. Berg, and L\. Fei\-FeiImageNet large scale visual recognition challenge\.International Journal of Computer Vision115\(3\),pp\. 211–252\.External Links:[Document](https://dx.doi.org/10.1007/s11263-015-0816-y),1409\.0575Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[Appendix H](https://arxiv.org/html/2609.02987#A8.p1.1),[§6\.1](https://arxiv.org/html/2609.02987#S6.SS1.p1.1)\.
- Schaefferet al\.\(2025\)R\. Schaeffer, J\. Kazdan, J\. Hughes, J\. Juravsky, S\. Price, A\. Lynch, E\. Jones, R\. Kirk, A\. Mirhoseini, and S\. KoyejoHow do large language monkeys get their power \(laws\)?\.InInternational Conference on Machine Learning \(ICML\),Vol\.267,pp\. 53132–53176\.External Links:2502\.17578Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p1.1),[§2](https://arxiv.org/html/2609.02987#S2.p3.1),[§6](https://arxiv.org/html/2609.02987#S6.p3.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Shiet al\.\(2023\)X\. Shi, W\. Cao, and S\. RaschkaDeep neural networks for rank\-consistent ordinal regression based on conditional probabilities\.Pattern Analysis and Applications26,pp\. 941–955\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px7.p1.1)\.
- Shojaeeet al\.\(2023\)P\. Shojaee, A\. Jain, S\. Tipirneni, and C\. K\. ReddyExecution\-based code generation using deep reinforcement learning\.Transactions on Machine Learning Research\.Note:arXiv:2301\.13816Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Shypulaet al\.\(2024\)A\. Shypula, A\. Madaan, Y\. Zeng, U\. Alon, J\. Gardner, M\. Hashemi, G\. Neubig, P\. Ranganathan, O\. Bastani, and A\. YazdanbakhshLearning performance\-improving code edits\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/pdf?id=ix7rLVHXyY)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.SSS0.Px1.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward gaming\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 9460–9471\.Note:arXiv:2209\.13085Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.p1.1)\.
- Snellet al\.\(2025\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InInternational Conference on Learning Representations \(ICLR\),External Links:2408\.03314Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2025\)Y\. Song, J\. Kempe, and R\. MunosOutcome\-based exploration for LLM reasoning\.External Links:2509\.06941,[Link](https://arxiv.org/abs/2509.06941)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. ChristianoLearning to summarize from human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2009\.01325Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. A\. McAllester, S\. P\. Singh, and Y\. MansourPolicy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 1057–1063\.Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Tajwaret al\.\(2026\)F\. Tajwar, G\. Zeng, Y\. Zhou, Y\. Song, D\. Arora, Y\. Jiang, J\. Schneider, R\. Salakhutdinov, H\. Feng, and A\. ZanetteMaximum likelihood reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),External Links:2602\.02710Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.02987#S2.SS3.p1.2),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Tamaret al\.\(2015\)A\. Tamar, Y\. Glassner, and S\. MannorOptimizing the CVaR via sampling\.InAAAI Conference on Artificial Intelligence \(AAAI\),pp\. 2993–2999\.External Links:1404\.3862Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px6.p1.1)\.
- Tanget al\.\(2025\)Y\. Tang, K\. Zheng, G\. Synnaeve, and R\. MunosOptimizing language models for inference time objectives using reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 59066–59085\.External Links:2503\.19595,[Link](https://arxiv.org/abs/2503.19595)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Teamet al\.\(2025\)K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao, C\. Tang, C\. Wang, D\. Zhang, E\. Yuan, E\. Lu, F\. Tang, F\. Sung, G\. Wei, G\. Lai, H\. Guo,et al\.Kimi k1\.5: scaling reinforcement learning with LLMs\.External Links:2501\.12599,[Link](https://arxiv.org/abs/2501.12599)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1)\.
- Thrampoulidiset al\.\(2026\)C\. Thrampoulidis, S\. Mahdavi, and W\. DengAdvantage shaping as surrogate reward maximization: unifying Pass@K policy gradients\.Transactions on Machine Learning Research\.External Links:2510\.23049Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.
- Tuylset al\.\(2026\)J\. Tuyls, D\. J\. Foster, A\. Krishnamurthy, and J\. T\. AshRepresentation\-based exploration for language models: from test\-time to post\-training\.InInternational Conference on Learning Representations \(ICLR\),External Links:2510\.11686Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Walder and Karkhanis \(2025\)C\. Walder and D\. T\. KarkhanisPass@K policy optimization: solving harder reinforcement learning problems\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38\.External Links:2505\.15201Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1),[§2](https://arxiv.org/html/2609.02987#S2.p3.1),[§6](https://arxiv.org/html/2609.02987#S6.p3.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang, Y\. Liu, A\. Yang, A\. Zhao, Y\. Yue, S\. Song, B\. Yu, G\. Huang, and J\. LinBeyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for LLM reasoning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://openreview.net/forum?id=yfcpdY4gMP)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2203\.11171Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px3.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3–4\),pp\. 229–256\.External Links:[Document](https://dx.doi.org/10.1007/BF00992696)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p2.1),[§1](https://arxiv.org/html/2609.02987#S1.p1.1),[§2](https://arxiv.org/html/2609.02987#S2.p2.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2025\)F\. Wu, W\. Xuan, X\. Lu, M\. Liu, Y\. Dong, Z\. Harchaoui, and Y\. ChoiThe invisible leash: why RLVR may or may not escape its origin\.External Links:2507\.14843,[Link](https://arxiv.org/abs/2507.14843)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1)\.
- Xionget al\.\(2025\)W\. Xiong, C\. Ye, B\. Liao, H\. Dong, X\. Xu, C\. Monz, J\. Bian, N\. Jiang, and T\. ZhangReinforce\-Ada: an adaptive sampling framework under non\-linear RL objectives\.External Links:2510\.04996,[Link](https://arxiv.org/abs/2510.04996)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px5.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§6\.4](https://arxiv.org/html/2609.02987#S6.SS4.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, D\. Li, Y\. Dai, Y\. Yang, Z\. Luo, Z\. Zhao, Z\. Hu, J\. Huang, A\. Saha, Z\. Chen, R\. Xu, L\. Pan, S\. Savarese, C\. Xiong, and J\. LiGTA1: GUI test\-time scaling agent\.InInternational Conference on Learning Representations \(ICLR\),External Links:2507\.05791Cited by:[§J\.1](https://arxiv.org/html/2609.02987#A10.SS1.p1.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.3](https://arxiv.org/html/2609.02987#S6.SS3.p2.1)\.
- Yanget al\.\(2025b\)Z\. Yang, Z\. Guo, Y\. Huang, Y\. Wang, D\. Xie, H\. Li, Y\. Wang, X\. Liang, and J\. TangDepth\-breadth synergy in RLVR: unlocking LLM reasoning gains with adaptive exploration\.External Links:2508\.13755,[Link](https://arxiv.org/abs/2508.13755)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2503\.14476Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p1.1)\.
- Yuanet al\.\(2025\)X\. Yuan, J\. Zhang, K\. Li, Z\. Cai, L\. Yao, J\. Chen, E\. Wang, Q\. Hou, J\. Chen, P\. Jiang, and B\. LiSE\-GUI: enhancing visual grounding for GUI agents via self\-evolutionary reinforcement learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2505\.12370Cited by:[§J\.2](https://arxiv.org/html/2609.02987#A10.SS2.p1.1),[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1),[§6\.3](https://arxiv.org/html/2609.02987#S6.SS3.p2.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.38\.External Links:2504\.13837Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p2.1),[§7](https://arxiv.org/html/2609.02987#S7.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025\)H\. Zeng, D\. Jiang, H\. Wang, P\. Nie, X\. Chen, and W\. ChenACECODER: acing coder RL via automated test\-case synthesis\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Vienna, Austria\.Note:arXiv:2502\.01718Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px8.p1.1)\.
- Zhanget al\.\(2020\)K\. Zhang, A\. Koppel, H\. Zhu, and T\. BaşarGlobal convergence of policy gradient methods to \(almost\) locally optimal policies\.SIAM Journal on Control and Optimization58\(6\),pp\. 3586–3612\.Cited by:[§C\.1](https://arxiv.org/html/2609.02987#A3.SS1.p4.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.External Links:2507\.18071,[Link](https://arxiv.org/abs/2507.18071)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02987#S1.p1.1)\.
- Zheng \(2026\)Y\. ZhengRL2ML: finite\-rollout surrogate objectives from reinforcement learning to maximum likelihood\.External Links:2605\.30154,[Link](https://arxiv.org/abs/2605.30154)Cited by:[Appendix B](https://arxiv.org/html/2609.02987#A2.SS0.SSS0.Px2.p1.1)\.

## Appendix Contents

## Appendix ANotation

[Table2](https://arxiv.org/html/2609.02987#A1.T2)collects every symbol used in the paper\. One symbol, one meaning is strictly enforced\.

Table 2:Notation\.
## Appendix BExtended Related Work

This section gives the full related\-work discussion summarized in[Section7](https://arxiv.org/html/2609.02987#S7), together with several literatures not covered there: inference\-time selection as its own topic, exploration and sample\-allocation methods, threshold decompositions of continuous rewards, and the literatures behind each evaluated domain\.

##### Reinforcement learning post\-training

Reinforcement learning now drives post\-training for reasoning and agentic models, and almost all of it maximizes expected reward\. The recipe descends from learning against human preferences\([Stiennon et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib90);[Ouyang et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib6)\)and, once verifiable rewards replaced learned reward models, produced the current generation of reasoning systems\([Guo et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib30);[OpenAI et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib29);[Team et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib31);[Lambert et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib9)\)\. Practical estimators have converged on critic\-free group\-relative updates: GRPO standardizes rewards within a group\([Shao et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib36)\), RLOO subtracts a leave\-one\-out baseline\([Ahmadian et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib37);[Kool et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib89)\), and later variants adjust clipping, normalization, and the level at which the ratio is formed\([Yu et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib38);[Zheng et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib11);[Liu et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib10)\)\. Every one of these estimators traces back to the score\-function policy gradient\([Williams, 1992](https://arxiv.org/html/2609.02987#bib.bib48);[Sutton et al\., 1999](https://arxiv.org/html/2609.02987#bib.bib7);[Schulman et al\., 2017](https://arxiv.org/html/2609.02987#bib.bib1)\)and differs from the others in variance rather than in target\. They share one population objective, the mean of the reward distribution, and in the weight view of[Section5](https://arxiv.org/html/2609.02987#S5)they share the flat gradient weightϕ′​\(p\)=1\\phi^\{\\prime\}\(p\)=1\. TailRL changes that target rather than the variance around it\.

##### Objectives beyond the mean

A recent line of work replaces the mean with a nonlinear functional of the policy’s success probability, and TailRL belongs to it\. MaxRL maximizes the log\-probability of success for binary rewards and expands that logarithm into a harmonically weighted sum of Pass@kkgradients\([Tajwar et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib39)\)\. PKPO derives unbiased estimators for Pass@kkand for its continuous generalization, the expected maximum ofkkrewards\([Walder and Karkhanis, 2025](https://arxiv.org/html/2609.02987#bib.bib59)\), while other work optimizes a chosen inference\-time metric directly\([Tang et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib65);[Chen et al\., 2025b](https://arxiv.org/html/2609.02987#bib.bib5)\)or reweighs rollouts by problem difficulty\([Yang et al\., 2025b](https://arxiv.org/html/2609.02987#bib.bib41)\)\.[Davis and Recht \(2025\)](https://arxiv.org/html/2609.02987#bib.bib19)unify much of this by showing that popular algorithms implicitly ascend a monotone transform of the success probability, and[Thrampoulidis et al\. \(2026\)](https://arxiv.org/html/2609.02987#bib.bib93)recast the resulting Pass@kkadvantage weights as surrogate reward maximization\. Related objectives target diversity\([Hamid et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib44);[Orney et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib43)\), risk sensitivity\([Jiang et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib46)\), and rare\-success amplification\([Osband, 2026](https://arxiv.org/html/2609.02987#bib.bib42);[Kaddour, 2026](https://arxiv.org/html/2609.02987#bib.bib45)\)\. Contemporaneous works extend two pieces of this picture: RL2ML generalizes the harmonic coefficient for binary rewards\([Zheng, 2026](https://arxiv.org/html/2609.02987#bib.bib91)\), and OrderGrad estimates gradients of L\-statistics over sorted rewards\([Parmas et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib92)\)\. Each of these methods commits to one threshold, one transform, or one order statistic\. TailRL instead integrates the log tail\-probability over every threshold, so its order\-TTtruncation is a harmonic combination of Best\-of\-kkobjectives and reduces to MaxRL exactly when the reward is binary\.

##### Inference\-time scaling and selection

Drawing many rollouts and keeping the best one has become a standard axis of deployment compute, which makes Pass@kkand Best\-of\-kkthe metrics that matter at inference\. Repeated sampling raises coverage predictably over several orders of magnitude\([Brown et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib63);[Schaeffer et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib35)\), and allocating compute at test\-time can beat allocating it to parameters\([Snell et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib64)\)\. Selecting among the samples requires either agreement\([Wang et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib4)\), a reward function\([Cobbe et al\., 2021](https://arxiv.org/html/2609.02987#bib.bib61);[Lightman et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib62)\), or execution\([Li et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib52)\), and the unbiased Pass@kkestimator we use comes from this literature\([Chen et al\., 2021](https://arxiv.org/html/2609.02987#bib.bib51)\)\. Selection also has limits, since optimizing hard against an imperfect reward function eventually degrades true quality\([Gao et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib100)\)\. This body of work measures or exploits the upper tail after training has finished\. TailRL makes that same upper tail the training objective\.

##### Distribution narrowing under expected reward training

Expected\-reward post\-training reliably sharpens the policy, which is the failure mode TailRL is designed to resist\.[Cui et al\. \(2025\)](https://arxiv.org/html/2609.02987#bib.bib80)characterize this as an entropy mechanism in which the policy\-gradient covariance term stays positive and entropy falls monotonically without intervention\. Measured at large sampling budgets, the effect is visible as a narrowing of what the policy can produce: base models can match or exceed post\-trained models on Pass@kkat largekk\([Yue et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib8)\), the reachable support changes little\([Wu et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib40);[Nguyen et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib14)\), and diversity drops across settings\([Kirk et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib81);[Dang et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib12)\)\. Regularization is one reported cause, since a KL penalty to a reference policy can itself favor mode collapse\([GX\-Chen et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib3)\)\. A parallel line intervenes on entropy directly, through token\-level selection\([Wang et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib23)\), entropy\-aware bonuses\([Cheng et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib22);[Hao et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib20)\), or smoothing\([Gai et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib27)\), and these interventions show that coverage can be partly recovered\. TailRL takes a different route and reweights reward thresholds by the inverse of how often the policy reaches them \(tail\-probability\), so rare high\-reward threshold keep influence without an added regularizer\.

##### Exploration and sample allocation

Exploration methods change which rollouts a learner sees, whereas TailRL changes how a fixed group of rollouts is weighted\. Classical approaches add bonuses from visitation counts, prediction error, or information gain\([Bellemare et al\., 2016](https://arxiv.org/html/2609.02987#bib.bib21);[Pathak et al\., 2017](https://arxiv.org/html/2609.02987#bib.bib17);[Burda et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib18)\)\. Their language\-model counterparts penalize repeated outcomes\([Song et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib15)\), reward novelty in representation space\([Tuyls et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib25);[Dai et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib24)\), widen the rollout budget\([Hu et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib13)\), or schedule problems by difficulty\([Chen et al\., 2025a](https://arxiv.org/html/2609.02987#bib.bib28);[Qu et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib26)\)\. Closest to us, Reinforce\-Ada allocates samples adaptively under a non\-linear objective and arrives at a difficulty\-prioritizing weighted estimator\([Xiong et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib16)\)\.

##### Distributional and risk\-sensitive reinforcement learning

Modeling a whole reward or return distribution is not new, so it is worth stating precisely what TailRL does differently\. Distributional reinforcement learning learns the return distribution as a critic, representing it categorically\([Bellemare et al\., 2017](https://arxiv.org/html/2609.02987#bib.bib84)\)or through quantiles\([Dabney et al\., 2018b](https://arxiv.org/html/2609.02987#bib.bib85);[Dabney et al\., 2018a](https://arxiv.org/html/2609.02987#bib.bib95);[Bellemare et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib96)\), and propagates it with a distributional Bellman operator\. Risk\-sensitive methods optimize a tail functional of that distribution, most often the conditional value at risk\([Rockafellar and Uryasev, 2000](https://arxiv.org/html/2609.02987#bib.bib86);[Tamar et al\., 2015](https://arxiv.org/html/2609.02987#bib.bib87);[Chow et al\., 2018](https://arxiv.org/html/2609.02987#bib.bib97);[Clavier et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib34)\)\. Two differences separate these from TailRL\. First, TailRL carries no critic and performs no bootstrapping: the reward function supplies the reward in one step, and the tail probabilities appear only inside a policy\-gradient weight computed from the group itself\. Second, risk measures fix a single risk level, and a fixed level can make a learner blind to rare successes\([Greenberg et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib98)\), whereas TailRL integrates the log tail\-probability uniformly across every threshold\.

##### Threshold decompositions of continuous rewards

Splitting a continuous reward into a family of binary threshold events is a classical device in supervised learning, and it is the direct ancestor of the decomposition in[Fig\.2](https://arxiv.org/html/2609.02987#S3.F2)\. Cumulative\-link models predictP⁡\(y\>j\)P\(y\>j\)at each threshold\([McCullagh, 1980](https://arxiv.org/html/2609.02987#bib.bib47)\), and reductions to extended binary classification train one classifier per threshold and reassemble the prediction\([Frank and Hall, 2001](https://arxiv.org/html/2609.02987#bib.bib82);[Li and Lin, 2006](https://arxiv.org/html/2609.02987#bib.bib83)\)\. Neural versions inherit the same structure, with multiple binary outputs\([Niu et al\., 2016](https://arxiv.org/html/2609.02987#bib.bib49)\)and rank\-consistency constraints across cuts\([Cao et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib50);[Shi et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib99)\)\.[Patel et al\. \(2026\)](https://arxiv.org/html/2609.02987#bib.bib94)carry the idea into policy optimization for discrete rewards\. TailRL is the policy\-gradient counterpart for continuous rewards: the thresholdτ\\tauplays the role of the ordinal cut, the indicator𝟙\{r\(x,z\)\>τ\}\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}plays the role of the binary label, and the decomposition runs over a continuum rather than a finite set of cuts\.

##### Reinforcement learning in the evaluated domains

Our four settings each connect to an established line of work\. Bounding\-box prediction is normally trained by direct supervision on coordinates, using anL​1L1term, a generalized intersection\-over\-union term, or the combination adopted by DETR\([Rezatofighi et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib67);[Carion et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib66)\), against which[Section6\.1](https://arxiv.org/html/2609.02987#S6.SS1)compares a purely reward\-driven policy\([Russakovsky et al\., 2015](https://arxiv.org/html/2609.02987#bib.bib32);[Deselaers et al\., 2010](https://arxiv.org/html/2609.02987#bib.bib60)\)\. GUI grounding has recently been posed as reinforcement learning with a dense point reward on top of vision\-language backbones\([Bai et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib68);[Yang et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib69);[Yuan et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib70);[Liu et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib79)\)and evaluated on high\-resolution professional interfaces\([Li et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib71)\)\. Code generation has long used execution feedback as a reward signal\([Le et al\., 2022](https://arxiv.org/html/2609.02987#bib.bib53);[Shojaee et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib54);[Liu et al\., 2023](https://arxiv.org/html/2609.02987#bib.bib55);[Dou et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib56);[Gehring et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib57);[Zeng et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib58)\), and optimizing runtime rather than correctness requires both a corpus of slow programs and a deterministic timing oracle\([Shypula et al\., 2024](https://arxiv.org/html/2609.02987#bib.bib72);[Binkert et al\., 2011](https://arxiv.org/html/2609.02987#bib.bib88)\)\. That last setting also admits a reward\-preserving shortcut, returning the input unchanged, which is an instance of the reward gaming characterized by[Skalse et al\. \(2022\)](https://arxiv.org/html/2609.02987#bib.bib103)\.[Section6\.4](https://arxiv.org/html/2609.02987#S6.SS4)uses it to separate an objective that settles for a dependable moderate outcome from one that keeps searching for a rare better one\.

## Appendix CSupporting Results for the TailRL Gradient Estimator

This section supplies the regularity conditions and proofs used in[Section4](https://arxiv.org/html/2609.02987#S4)\. We continue to work at a fixed inputxxand assumer⁡\(x,z\)∈\[0,1\]r\(x,z\)\\in\[0,1\]\.

### C\.1Regularity of the TailRL Gradient

The derivations of[Section4](https://arxiv.org/html/2609.02987#S4)rest on three operations\. The score\-function identity differentiates the tail\-probability under an expectation over the policy’s rollouts\. The second operation moves a gradient from outside the threshold integral to inside it\. The third swaps the order of the threshold integral and the rollout expectation\. Each operation exchanges two limits, and each exchange is valid once the quantity being moved is bounded by something with a finite integral\. We state one assumption for each operation, justify each in turn, and then record the lemma that licenses all three\. Throughout, fix an inputxx, let the rollouts take values in a countable set, and letΘ\\Thetabe an open set of parameters\.

###### Assumption 3\(Smooth policy\)\.

The support ofπθ\(⋅∣x\)\\pi\_\{\\theta\}\(\\cdot\\mid x\)is the same for everyθ∈Θ\\theta\\in\\Theta, each probabilityπθ​\(z∣x\)\\pi\_\{\\theta\}\(z\\mid x\)is differentiable inθ\\thetaonΘ\\Theta, and

∑zsupθ∈Θ‖∇θπθ​\(z∣x\)‖<∞\.\\sum\_\{z\}\\sup\_\{\\theta\\in\\Theta\}\\,\\bigl\\\|\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(z\\mid x\)\\bigr\\\|\\;<\\;\\infty\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-smooth\-policy\}\(35\)

[Assumption3](https://arxiv.org/html/2609.02987#Thmtheorem3)is the classical hypothesis behind likelihood\-ratio gradient estimators\([Williams, 1992](https://arxiv.org/html/2609.02987#bib.bib48);[Glynn, 1990](https://arxiv.org/html/2609.02987#bib.bib73)\): it is the interchange condition of[L’Ecuyer \(1995\)](https://arxiv.org/html/2609.02987#bib.bib74), and[Mohamed et al\. \(2020\)](https://arxiv.org/html/2609.02987#bib.bib75)survey it as the standing assumption of the score\-function estimator class\. It holds by inspection for the policies of this paper: a softmax policy has a fixed finite support, its rollout probabilities are differentiable inθ\\theta, and a finite sum of continuous gradient norms is bounded on a boundedΘ\\Theta\.

###### Assumption 4\(Finite objective\)\.

The tail\-likelihood is finite onΘ\\Theta:∫01\|log⁡pθ​\(x,τ\)\|​𝑑τ<∞\\int\_\{0\}^\{1\}\\bigl\|\\log p\_\{\\theta\}\(x,\\tau\)\\bigr\|\\,d\\tau<\\inftyfor everyθ∈Θ\\theta\\in\\Theta\.

[Assumption4](https://arxiv.org/html/2609.02987#Thmtheorem4)is the substantive assumption, and full support is what delivers it\. A softmax policy assigns strictly positive probability to every rollout in its finite support, and the auto\-regressive token policies of this paper, softmax at every step with bounded generation length, are of this form\. Letz⋆z^\{\\star\}be a rollout of maximal reward onxxand normalize that maximum to11\. Every thresholdτ∈\[0,1\)\\tau\\in\[0,1\)is then cleared at least byz⋆z^\{\\star\}, so the tail\-probability is squeezed between two positive constants:

0<πθ​\(z⋆∣x\)≤pθ​\(x,τ\)≤1,∫01\|log⁡pθ​\(x,τ\)\|​𝑑τ≤−log⁡πθ​\(z⋆∣x\)<∞\.0\\;<\\;\\pi\_\{\\theta\}\(z^\{\\star\}\\mid x\)\\;\\leq\\;p\_\{\\theta\}\(x,\\tau\)\\;\\leq\\;1,\\qquad\\int\_\{0\}^\{1\}\\bigl\|\\log p\_\{\\theta\}\(x,\\tau\)\\bigr\|\\,d\\tau\\;\\leq\\;\-\\log\\pi\_\{\\theta\}\(z^\{\\star\}\\mid x\)\\;<\\;\\infty\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-softmax\-check\}\(36\)The bound is uniform over any bounded parameter setΘ\\Theta, since−log⁡πθ​\(z⋆∣x\)\-\\log\\pi\_\{\\theta\}\(z^\{\\star\}\\mid x\)is continuous inθ\\theta\. If no rollout attains reward11, the same squeeze holds over thresholds below the largest attainable reward, and the threshold integral is read over that range\. One consequence is used repeatedly below: the finiteness of the integral forcespθ​\(x,τ\)\>0p\_\{\\theta\}\(x,\\tau\)\>0for almost everyτ\\tau, so the logarithm and every ratio withpθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)in its denominator are defined\.

###### Assumption 5\(Bounded weighted score\)\.

There is a constantC<∞C<\\inftysuch that, for almost everyτ\\tau,

supθ∈Θ𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)\>τ\}∥S\(x,z\)∥2\]pθ​\(x,τ\)≤C\.\\sup\_\{\\theta\\in\\Theta\}\\;\\frac\{\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,\\\|S\(x,z\)\\\|\_\{2\}\\right\]\}\{p\_\{\\theta\}\(x,\\tau\)\}\\;\\leq\\;C\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-dominated\-score\}\(37\)

[Assumption5](https://arxiv.org/html/2609.02987#Thmtheorem5)follows from the bounded\-reward assumptionsupθ∈Θsupz‖S⁡\(x,z\)‖≤C\\sup\_\{\\theta\\in\\Theta\}\\sup\_\{z\}\\\|S\(x,z\)\\\|\\leq Croutine in policy\-gradient convergence analyses\([Papini et al\., 2018](https://arxiv.org/html/2609.02987#bib.bib76);[Zhang et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib77);[Agarwal et al\., 2021](https://arxiv.org/html/2609.02987#bib.bib78)\), since𝔼z\[𝟙\{r\(x,z\)\>τ\}∥S\(x,z\)∥\]≤Cpθ\(x,τ\)\\mathbb\{E\}\_\{z\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,\\\|S\(x,z\)\\\|\\right\]\\leq C\\,p\_\{\\theta\}\(x,\\tau\)\. Scores are bounded for the softmax policies of this paper whenever the logits have bounded gradients\.

With the assumptions justified, the lemma states only its conclusion\.

###### Lemma 6\(Regularity for the TailRL gradient\)\.

Under[3](https://arxiv.org/html/2609.02987#Thmtheorem3),[4](https://arxiv.org/html/2609.02987#Thmtheorem4)and[5](https://arxiv.org/html/2609.02987#Thmtheorem5), at everyθ∈Θ\\theta\\in\\Theta, the population objective and every finite truncation are differentiable, and the score\-function identity and every exchange of a gradient, threshold integral, and rollout expectation in[Sections4](https://arxiv.org/html/2609.02987#S4)and[4\.2](https://arxiv.org/html/2609.02987#S4.SS2)are valid\.

Proof\.The proof verifies the three operations in order, at an arbitraryθ∈Θ\\theta\\in\\Theta\.

*The score\-function identity\.*Fix a thresholdτ\\tau\. The tail\-probability is the sum of the policy probabilities over the rollouts that clear it,

pθ\(x,τ\)=∑z𝟙\{r\(x,z\)\>τ\}πθ\(z∣x\)\.p\_\{\\theta\}\(x,\\tau\)=\\sum\_\{z\}\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,\\pi\_\{\\theta\}\(z\\mid x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-pass\-rate\-density\}\(38\)By[Assumption3](https://arxiv.org/html/2609.02987#Thmtheorem3), every term of the differentiated series is bounded by the summable envelope of[Eq\.35](https://arxiv.org/html/2609.02987#A3.E35), so the gradient of the sum is the sum of the gradients\.

∇θpθ\(x,τ\)=∑z𝟙\{r\(x,z\)\>τ\}∇θπθ\(z∣x\)\.\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)=\\sum\_\{z\}\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(z\\mid x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-pass\-rate\-density\-gradient\}\(39\)Substituting∇θπθ​\(z∣x\)=πθ​\(z∣x\)​S​\(x,z\)\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(z\\mid x\)=\\pi\_\{\\theta\}\(z\\mid x\)\\,S\(x,z\)in[Eq\.39](https://arxiv.org/html/2609.02987#A3.E39)turns the differentiated sum into the score\-function identity

∇θpθ\(x,τ\)=𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)\>τ\}S\(x,z\)\]\.\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,S\(x,z\)\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-score\-identity\}\(40\)
*Moving the gradient inside the threshold integral\.*The triangle inequality applied to[Eq\.40](https://arxiv.org/html/2609.02987#A3.E40), followed by[Eq\.37](https://arxiv.org/html/2609.02987#A3.E37), bounds the log\-gradient at every threshold:

‖∇θ​log​pθ​\(x,τ\)‖=‖∇θpθ​\(x,τ\)‖pθ​\(x,τ\)≤𝔼z\[𝟙\{r\(x,z\)\>τ\}∥S\(x,z\)∥\]pθ​\(x,τ\)≤C\.\\bigl\\\|\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(x,\\tau\)\\bigr\\\|=\\frac\{\\\|\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\\|\}\{p\_\{\\theta\}\(x,\\tau\)\}\\leq\\frac\{\\mathbb\{E\}\_\{z\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,\\\|S\(x,z\)\\\|\\right\]\}\{p\_\{\\theta\}\(x,\\tau\)\}\\leq C\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-log\-gradient\-bound\}\(41\)The objective is finite by[Assumption4](https://arxiv.org/html/2609.02987#Thmtheorem4), and the gradient of its integrand is bounded by the constantCC, so the gradient moves inside the threshold integral:

∇θJTailRL​\(θ,x\)=∫01∇θpθ​\(x,τ\)pθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-population\-differentiation\}\(42\)The truncations need nothing new\. The order\-NNthreshold multiplier is a finite geometric sum and is therefore bounded,

1−\(1−pθ​\(x,τ\)\)Npθ​\(x,τ\)=∑j=0N−1\(1−pθ​\(x,τ\)\)j≤N,\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\}\{p\_\{\\theta\}\(x,\\tau\)\}=\\sum\_\{j=0\}^\{N\-1\}\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{j\}\\leq N,\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-finite\-multiplier\-bound\}\(43\)so the gradient of the truncated integrand is bounded byN​CNCand the same argument differentiatesJTailRL\(N\)J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\.

*Swapping the threshold integral and the rollout expectation\.*Integrating[Eq\.37](https://arxiv.org/html/2609.02987#A3.E37)over the thresholds gives

∫01𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)\>τ\}pθ​\(x,τ\)∥S\(x,z\)∥\]dτ≤C<∞,\\int\_\{0\}^\{1\}\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,\\\|S\(x,z\)\\\|\\right\]d\\tau\\leq C<\\infty,\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-fubini\-bound\}\(44\)so the integrand is absolutely integrable and Fubini’s theorem permits taking the threshold integral and the rollout expectation in either order\. The truncated multiplier never exceeds1/pθ​\(x,τ\)1/p\_\{\\theta\}\(x,\\tau\), so the truncated weighted score\-function is bounded by the integrand of[Eq\.44](https://arxiv.org/html/2609.02987#A3.E44)and the same swap applies at every orderNN\. ∎

The countability of the rollout space is inessential: for a policy with densities over a continuous rollout\-space, the sum in[Eq\.38](https://arxiv.org/html/2609.02987#A3.E38)becomes an integral against a common reference measure and the proof is unchanged\.

### C\.2Derivation of the Rollout\-Weight Forms

We derive the population rollout\-weight form of the TailRL gradient and its order\-NNtruncation,[Eq\.25](https://arxiv.org/html/2609.02987#S4.E25)\. Because the tail\-likelihood averages per\-level log\-likelihoods, its gradient is an average of per\-level likelihood gradients,

∇θJTailRL​\(θ,x\)=∫01∇θpθ​\(x,τ\)pθ​\(x,τ\)​𝑑τ=𝔼τ∼U⁡\[0,1\]​\[∇θpθ​\(x,τ\)pθ​\(x,τ\)\]\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau=\\mathbb\{E\}\_\{\\tau\\sim\\mathrm\{U\}\[0,1\]\}\\left\[\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-gradient\-over\-thresholds\}\(45\)At a fixed threshold, the log\-derivative trick expresses the pass\-rate gradient as a score\-function expectation:

∇θpθ\(x,τ\)=𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,zj\)\>τ\}S\(x,z\)\]\.\\displaystyle\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}S\(x,z\)\\right\]\.\(46\)Substituting into[Eq\.45](https://arxiv.org/html/2609.02987#A3.E45)writes the population gradient as a double expectation,

∇θJTailRL\(θ;x\)=𝔼τ∼U⁡\[0,1\]\[𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)\>τ\}pθ​\(x,τ\)S\(x,z\)\]\],\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\mathbb\{E\}\_\{\\tau\\sim\\mathrm\{U\}\[0,1\]\}\\left\[\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\}\{p\_\{\\theta\}\(x,\\tau\)\}S\(x,z\)\\right\]\\right\],\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-double\-expectation\}\(47\)and the integrability condition of[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6)permits Fubini’s theorem to exchange the two\. The inner threshold integral then runs over exactly the thresholds cleared by the rollout, gives us[Eq\.48](https://arxiv.org/html/2609.02987#A3.E48)as restated below\.

∇θJTailRL​\(θ,x\)=𝔼z​\[\(∫0r⁡\(x,z\)1pθ​\(x,τ\)​𝑑τ\)​S​\(x,z\)\],\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\mathbb\{E\}\_\{z\}\\\!\\left\[\\left\(\\int\_\{0\}^\{r\(x,z\)\}\\frac\{1\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau\\right\)S\(x,z\)\\right\],\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-population\-gradient\}\(48\)The order\-NNmember follows the same route\. Differentiating[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)and summing the finite geometric series gives

∇θJTailRL\(N\)​\(θ,x\)=∫01\(1−\(1−pθ​\(x,τ\)\)Npθ​\(x,τ\)\)​∇θpθ​\(x,τ\)​𝑑τ,\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\left\(\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\right\)\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau,\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-order\-n\-gradient\}\(49\)and substituting[Eq\.46](https://arxiv.org/html/2609.02987#A3.E46)and exchanging the threshold integral with the rollout expectation restricts the integral to the thresholds below the rollout’s reward,

∇θJTailRL\(N\)​\(θ,x\)=𝔼z​\[\(∫0r⁡\(x,z\)1−\(1−pθ​\(x,τ\)\)Npθ​\(x,τ\)​𝑑τ\)​S​\(x,z\)\],\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)=\\mathbb\{E\}\_\{z\}\\\!\\left\[\\left\(\\int\_\{0\}^\{r\(x,z\)\}\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau\\right\)S\(x,z\)\\right\],\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-truncated\-weighted\-gradient\}\(50\)which is[Eq\.25](https://arxiv.org/html/2609.02987#S4.E25)\.

### C\.3Best\-of\-kkDecompositions and Probabilistic Interpretations

We prove the finite and population Best\-of\-kkdecompositions stated in[Section3](https://arxiv.org/html/2609.02987#S3)\. It also shows that threshold weighting is equivalent to rescaling the reward axis\. We continue to work at a fixed inputxxand assumer⁡\(x,z\)∈\[0,1\]r\(x,z\)\\in\[0,1\]\. The gradient statements use the regularity conditions of[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6)\.

#### C\.3\.1Layer\-Cake Form of Best\-of\-kk

Letz1,…,zk∼i\.i\.d\.πθ\(⋅∣x\)z\_\{1\},\\ldots,z\_\{k\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\pi\_\{\\theta\}\(\\cdot\\mid x\)and writeRi:=r⁡\(x,zi\)R\_\{i\}:=r\(x,z\_\{i\}\)andMk:=max1≤i≤k⁡RiM\_\{k\}:=\\max\_\{1\\leq i\\leq k\}R\_\{i\}\. SinceMk∈\[0,1\]M\_\{k\}\\in\[0,1\], the layer\-cake identity gives

𝔼⁡\[Mk\]=∫01Pr⁡\(Mk\>τ\)​𝑑τ\.\\mathbb\{E\}\[M\_\{k\}\]=\\int\_\{0\}^\{1\}\\Pr\(M\_\{k\}\>\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-layer\-cake\-start\}\(51\)The maximum fails to clearτ\\tauexactly when allkkrollouts fail it\. By independence,

Pr⁡\(Mk≤τ\)=\(1−pθ​\(x,τ\)\)k\.\\Pr\(M\_\{k\}\\leq\\tau\)=\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-max\-cdf\}\(52\)Therefore,

Best\-of\-​k​\(θ,x\)=∫01\[1−\(1−pθ​\(x,τ\)\)k\]​𝑑τ\.\\text\{Best\-of\-\}k\(\\theta;x\)=\\int\_\{0\}^\{1\}\\left\[1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\\right\]d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-bok\-c3\}\(53\)

#### C\.3\.2Finite Harmonic Decomposition

###### Proposition 7\(Harmonic Best\-of\-kkexpansion of the finite objective\)\.

For every truncation orderT≥1T\\geq 1,

JTailRL\(T\)​\(θ,x\)=∑k=1TBest\-of\-​k​\(θ,x\)−1k\.J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\sum\_\{k=1\}^\{T\}\\frac\{\\text\{Best\-of\-\}k\(\\theta;x\)\-1\}\{k\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-finite\-c3\}\(54\)

Proof\.Exchange the finite sum in[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)with the threshold integral:

JTailRL\(T\)​\(θ,x\)\\displaystyle J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=−∑k=1T1k∫01\(1−pθ\(x,τ\)\)kdτ\\displaystyle=\-\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\int\_\{0\}^\{1\}\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\\,d\\tau=∑k=1T1k​\(∫01\[1−\(1−pθ​\(x,τ\)\)k\]​𝑑τ−1\)\.\\displaystyle=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\left\(\\int\_\{0\}^\{1\}\\left\[1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\\right\]\\,d\\tau\-1\\right\)\.\(55\)The integral in parentheses isBest\-of\-​k​\(θ,x\)\\text\{Best\-of\-\}k\(\\theta;x\)by[Eq\.53](https://arxiv.org/html/2609.02987#A3.E53)\. ∎

###### Proposition 8\(Harmonic Best\-of\-kkexpansion of the finite gradient\)\.

Under[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6), for everyT≥1T\\geq 1,

∇θJTailRL\(T\)​\(θ,x\)=∑k=1T1k​∇θBest\-of\-​k​\(θ,x\)\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\nabla\_\{\\theta\}\\text\{Best\-of\-\}k\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-grad\-finite\}\(56\)Equivalently,

∇θJTailRL\(T\)​\(θ,x\)=∫011−\(1−pθ​\(x,τ\)\)Tpθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ,\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{T\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau,\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-grad\-c3\}\(57\)where the ratio is interpreted by continuity asTTwhenpθ​\(x,τ\)=0p\_\{\\theta\}\(x,\\tau\)=0\.

Proof\.Differentiating[Eq\.54](https://arxiv.org/html/2609.02987#A3.E54)term by term gives[Eq\.56](https://arxiv.org/html/2609.02987#A3.E56)\. Differentiating[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)under the integral gives

∇θJTailRL\(T\)​\(θ,x\)\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=∫01\[∑k=1T\(1−pθ​\(x,τ\)\)k−1\]​∇θpθ​\(x,τ\)​𝑑τ\\displaystyle=\\int\_\{0\}^\{1\}\\left\[\\sum\_\{k=1\}^\{T\}\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\-1\}\\right\]\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau=∫011−\(1−pθ​\(x,τ\)\)Tpθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ,\\displaystyle=\\int\_\{0\}^\{1\}\\frac\{1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{T\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau,\(58\)where the second equality uses the finite geometric\-series identity\. ∎

AtT=1T=1,[Eq\.54](https://arxiv.org/html/2609.02987#A3.E54)gives

JTailRL\(1\)​\(θ,x\)=Best\-of\-​1​\(θ,x\)−1=JRL​\(θ,x\)−1\.J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(1\)\}\(\\theta;x\)=\\text\{Best\-of\-\}1\(\\theta;x\)\-1=J\_\{\\mathrm\{RL\}\}\(\\theta;x\)\-1\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-first\-order\}\(59\)Also, since0≤\(1−pθ​\(x,τ\)\)k≤10\\leq\(1\-p\_\{\\theta\}\(x,\\tau\)\)^\{k\}\\leq 1,

−HT≤JTailRL\(T\)​\(θ,x\)≤0,HT:=∑k=1T1k\.\-H\_\{T\}\\leq J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)\\leq 0,\\qquad H\_\{T\}:=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-finite\-bounds\}\(60\)

#### C\.3\.3Population Harmonic Decomposition

Proof of[Theorem1](https://arxiv.org/html/2609.02987#Thmtheorem1)\.For each threshold, define

sT​\(τ\):=∑k=1T\(1−pθ​\(x,τ\)\)kk\.s\_\{T\}\(\\tau\):=\\sum\_\{k=1\}^\{T\}\\frac\{\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\}\{k\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-maclaurin\-partial\}\(61\)The sequencesT​\(τ\)s\_\{T\}\(\\tau\)is nondecreasing inTT\. The Maclaurin series gives the extended\-real pointwise limit

limT→∞sT​\(τ\)=−log⁡pθ​\(x,τ\)\.\\lim\_\{T\\to\\infty\}s\_\{T\}\(\\tau\)=\-\\log p\_\{\\theta\}\(x,\\tau\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-maclaurin\-limit\}\(62\)The monotone convergence theorem therefore gives

limT→∞JTailRL\(T\)​\(θ,x\)\\displaystyle\\lim\_\{T\\to\\infty\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=−∫01limT→∞sT\(τ\)dτ\\displaystyle=\-\\int\_\{0\}^\{1\}\\lim\_\{T\\to\\infty\}s\_\{T\}\(\\tau\)\\,d\\tau=∫01log⁡pθ​\(x,τ\)​𝑑τ=JTailRL​\(θ,x\)\.\\displaystyle=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau=J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\.\(63\)Combining this limit with[Eq\.54](https://arxiv.org/html/2609.02987#A3.E54)yields

JTailRL​\(θ,x\)=∑k=1∞Best\-of\-​k​\(θ,x\)−1kJ\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{\\text\{Best\-of\-\}k\(\\theta;x\)\-1\}\{k\}\(64\)whenever the population objective is finite\.

For the gradients, define

wT​\(p\):=1−\(1−p\)Tp=∑j=0T−1\(1−p\)j\.w\_\{T\}\(p\):=\\frac\{1\-\(1\-p\)^\{T\}\}\{p\}=\\sum\_\{j=0\}^\{T\-1\}\(1\-p\)^\{j\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-finite\-weight\}\(65\)For everyp\>0p\>0,wT​\(p\)↑1/pw\_\{T\}\(p\)\\uparrow 1/p, and0≤wT​\(p\)≤1/p0\\leq w\_\{T\}\(p\)\\leq 1/p\. By[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6),

‖∇θpθ​\(x,τ\)‖pθ​\(x,τ\)≤C\\frac\{\\left\\\|\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\right\\\|\}\{p\_\{\\theta\}\(x,\\tau\)\}\\leq C\(66\)for almost every threshold and some finiteCC\. Dominated convergence applied to[Eq\.57](https://arxiv.org/html/2609.02987#A3.E57)gives

limT→∞∇θJTailRL\(T\)​\(θ,x\)\\displaystyle\\lim\_\{T\\to\\infty\}\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=∫01∇θpθ​\(x,τ\)pθ​\(x,τ\)​𝑑τ\\displaystyle=\\int\_\{0\}^\{1\}\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau=∇θJTailRL​\(θ,x\)\.\\displaystyle=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\.\(67\)The finite gradient identity then identifies this limit with∑k=1∞k−1​∇θBest\-of\-​k​\(θ,x\)\\sum\_\{k=1\}^\{\\infty\}k^\{\-1\}\\nabla\_\{\\theta\}\\text\{Best\-of\-\}k\(\\theta;x\)\. ∎

#### C\.3\.4Threshold Weighting as Reward Reparameterization

The uniform threshold distribution in[Eq\.13](https://arxiv.org/html/2609.02987#S3.E13)is the simplest member of a weighted family\. Letw:\[0,1\]→\(0,∞\)w:\[0,1\]\\to\(0,\\infty\)be a fixed density satisfying∫01w⁡\(τ\)​𝑑τ=1\\int\_\{0\}^\{1\}w\(\\tau\)\\,d\\tau=1, and define

Jw​\(θ,x\):=∫01w⁡\(τ\)​log⁡pθ​\(x,τ\)​𝑑τ\.J\_\{w\}\(\\theta;x\):=\\int\_\{0\}^\{1\}w\(\\tau\)\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weighted\-objective\}\(68\)
###### Proposition 9\(Threshold weighting is reward\-axis rescaling\)\.

Define

Fw​\(t\):=∫0tw⁡\(s\)​𝑑s,r~​\(x,z\):=Fw​\(r⁡\(x,z\)\)\.F\_\{w\}\(t\):=\\int\_\{0\}^\{t\}w\(s\)\\,ds,\\qquad\\widetilde\{r\}\(x,z\):=F\_\{w\}\\\!\\left\(r\(x,z\)\\right\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-reward\-transform\}\(69\)ThenFwF\_\{w\}is a strictly increasing map from\[0,1\]\[0,1\]to\[0,1\]\[0,1\], and

Jw\(θ;x\)=∫01logPrz∼πθ\(⋅∣x\)\(r~\(x,z\)\>u\)du\.J\_\{w\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\(\\widetilde\{r\}\(x,z\)\>u\\right\)\\,du\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weighting\-equivalence\}\(70\)Thus, weighted TailRL onrris exactly uniform TailRL on the monotone rescalingr~=Fw​\(r\)\\widetilde\{r\}=F\_\{w\}\(r\)\.

Proof\.Sincewwis positive and integrates to one,FwF\_\{w\}is strictly increasing withFw​\(0\)=0F\_\{w\}\(0\)=0andFw​\(1\)=1F\_\{w\}\(1\)=1\. Foru∈\[0,1\]u\\in\[0,1\],

Pr⁡\(r~​\(x,z\)\>u\)\\displaystyle\\Pr\\\!\\left\(\\widetilde\{r\}\(x,z\)\>u\\right\)=Pr⁡\(Fw​\(r⁡\(x,z\)\)\>u\)\\displaystyle=\\Pr\\\!\\left\(F\_\{w\}\(r\(x,z\)\)\>u\\right\)=Pr⁡\(r⁡\(x,z\)\>Fw−1​\(u\)\)=pθ​\(x,Fw−1​\(u\)\)\.\\displaystyle=\\Pr\\\!\\left\(r\(x,z\)\>F\_\{w\}^\{\-1\}\(u\)\\right\)=p\_\{\\theta\}\\\!\\left\(x,F\_\{w\}^\{\-1\}\(u\)\\right\)\.\(71\)Therefore,

∫01log⁡Pr⁡\(r~​\(x,z\)\>u\)​𝑑u\\displaystyle\\int\_\{0\}^\{1\}\\log\\Pr\\\!\\left\(\\widetilde\{r\}\(x,z\)\>u\\right\)du=∫01log⁡pθ​\(x,Fw−1​\(u\)\)​𝑑u\\displaystyle=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\\\!\\left\(x,F\_\{w\}^\{\-1\}\(u\)\\right\)du=∫01w⁡\(τ\)​log⁡pθ​\(x,τ\)​𝑑τ,\\displaystyle=\\int\_\{0\}^\{1\}w\(\\tau\)\\log p\_\{\\theta\}\(x,\\tau\)d\\tau,\(72\)where the last equality uses the substitutionu=Fw​\(τ\)u=F\_\{w\}\(\\tau\)\. ∎

The proposition shows that choosing a threshold density is equivalent to choosing a monotone coordinate system for reward quality\. Uniform weighting corresponds to the original normalized reward axis and introduces no additional function or hyperparameter\. A nonuniform weighting may still be useful when a task provides a preferred reward scale, but selecting or learning that scale is outside the scope of this work\.

#### C\.3\.5Recovery of MaxRL on Binary Rewards

Proof of[Eq\.17](https://arxiv.org/html/2609.02987#S3.E17)\.Ifr⁡\(x,z\)∈\{0,1\}r\(x,z\)\\in\\\{0,1\\\}, then for everyτ∈\[0,1\)\\tau\\in\[0,1\),

\{r\(x,z\)\>τ\}=\{r\(x,z\)=1\}\.\\\{r\(x,z\)\>\\tau\\\}=\\\{r\(x,z\)=1\\\}\.\(73\)Thereforepθ​\(x,τ\)=qθ​\(x\)p\_\{\\theta\}\(x,\\tau\)=q\_\{\\theta\}\(x\)for every nontrivial threshold, and

JTailRL​\(θ,x\)=∫01log⁡qθ​\(x\)​𝑑τ=log⁡qθ​\(x\)\.J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log q\_\{\\theta\}\(x\)\\,d\\tau=\\log q\_\{\\theta\}\(x\)\.\(74\)The same substitution in the finite objective gives

JTailRL\(T\)\(θ;x\)=−∑k=1T\(1−qθ​\(x\)\)kk,J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(T\)\}\(\\theta;x\)=\-\\sum\_\{k=1\}^\{T\}\\frac\{\\left\(1\-q\_\{\\theta\}\(x\)\\right\)^\{k\}\}\{k\},\(75\)which is the order\-TTMaclaurin truncation of MaxRL\. Finally, the maximum ofkkbinary rewards equals one exactly when at least one rollout succeeds, soBest\-of\-​k​\(θ,x\)=Pass​@​k​\(θ,x\)\\text\{Best\-of\-\}k\(\\theta;x\)=\\mathrm\{Pass@\}k\(\\theta;x\)\. ∎

### C\.4Harmonic Best\-of\-kkExpansion of the Tail\-Likelihood

This section proves that the truncated objective is a partial sum of Best\-of\-kkobjectives with harmonic coefficients, and that the tail\-likelihood is the full series\. Since the maximum ofkkrewards clears a threshold unless allkkrollouts fail it, by layer cake identity,

Best\-of\-​k​\(θ,x\)=𝔼⁡\[max1≤i≤k⁡r⁡\(x,zi\)\]=∫01\[1−\(1−pθ​\(x,τ\)\)k\]​𝑑τ\.\\text\{Best\-of\-\}k\(\\theta;x\)=\\mathbb\{E\}\\Big\[\\max\_\{1\\leq i\\leq k\}r\(x,z\_\{i\}\)\\Big\]=\\int\_\{0\}^\{1\}\\Big\[1\-\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\\Big\]\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-bok\}\(76\)
###### Proposition 10\(Harmonic Best\-of\-kkexpansion of the objective\)\.

Fix an inputxxand a parameterθ\\theta\. For every truncation orderT≥1T\\geq 1,

JTailRL\(T\)​\(θ,x\)=∑k=1T1k​\(*Best\-of\-*​k​\(θ,x\)−1\),J^\{\(T\)\}\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\Big\(\\text\{\\emph\{Best\-of\-\}\}k\(\\theta;x\)\-1\\Big\),\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-finite\}\(77\)and ifJTailRL​\(θ,x\)\>−∞J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\>\-\\infty, the same identity holds for the full series,

JTailRL​\(θ,x\)=∑k=1∞1k​\(*Best\-of\-*​k​\(θ,x\)−1\)\.J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{\\infty\}\\frac\{1\}\{k\}\\Big\(\\text\{\\emph\{Best\-of\-\}\}k\(\\theta;x\)\-1\\Big\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\}\(78\)

Proof\.We prove the finite identity first and obtain the series as its limit\. Start from the definition of the truncated objective in[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)and exchange the finite sum with the threshold integral:

JTailRL\(T\)\(θ;x\)=−∫01∑k=1T\(1−pθ​\(x,τ\)\)kkdτ=−∑k=1T1k∫01\(1−pθ\(x,τ\)\)kdτ\.J^\{\(T\)\}\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\-\\int\_\{0\}^\{1\}\\sum\_\{k=1\}^\{T\}\\frac\{\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\}\{k\}\\,d\\tau=\-\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\int\_\{0\}^\{1\}\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-swap\}\(79\)Each integral on the right is one minus a Best\-of\-kkvalue by[Eq\.76](https://arxiv.org/html/2609.02987#A3.E76), and substituting it in gives[Eq\.77](https://arxiv.org/html/2609.02987#A3.E77)\.

For the series, we letT→∞T\\to\\inftyon both sides of[Eq\.77](https://arxiv.org/html/2609.02987#A3.E77)and identify the two limits\.

We first show the left side converges to the tail\-likelihood\. SinceJTailRL​\(θ,x\)\>−∞J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\>\-\\infty, the tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)is positive for almost every threshold\. Fix such aτ\\tau, so that0≤1−pθ​\(x,τ\)<10\\leq 1\-p\_\{\\theta\}\(x,\\tau\)<1\. The Maclaurin series of the logarithm, evaluated at1−pθ​\(x,τ\)1\-p\_\{\\theta\}\(x,\\tau\), gives the pointwise limit of the partial sums:

limT→∞∑k=1T\(1−pθ​\(x,τ\)\)kk=∑k=1∞\(1−pθ​\(x,τ\)\)kk=−log⁡pθ​\(x,τ\)\.\\lim\_\{T\\to\\infty\}\\sum\_\{k=1\}^\{T\}\\frac\{\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\}\{k\}=\\sum\_\{k=1\}^\{\\infty\}\\frac\{\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\}\{k\}=\-\\log p\_\{\\theta\}\(x,\\tau\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-maclaurin\}\(80\)The convergence is monotone: every added term\(1−pθ​\(x,τ\)\)T\+1/\(T\+1\)\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{T\+1\}/\(T\+1\)is nonnegative, so the partial sums only grow withTT\. The integrands in the definition[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)ofJTailRL\(T\)J^\{\(T\)\}\_\{\\mathrm\{TailRL\{\}\}\}are exactly these partial sums\. Because they are nonnegative and increasing inTT, the limit of their integrals is the integral of their limit, which is the monotone convergence theorem\. Therefore

limT→∞JTailRL\(T\)\(θ;x\)=−∫01limT→∞∑k=1T\(1−pθ​\(x,τ\)\)kkdτ=∫01logpθ\(x,τ\)dτ=JTailRL\(θ;x\)\.\\lim\_\{T\\to\\infty\}J^\{\(T\)\}\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\-\\int\_\{0\}^\{1\}\\lim\_\{T\\to\\infty\}\\sum\_\{k=1\}^\{T\}\\frac\{\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\}\}\{k\}\\,d\\tau=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau=J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-limit\}\(81\)
The right side needs no computation\. For everyTT, the right side of[Eq\.77](https://arxiv.org/html/2609.02987#A3.E77)is theTT\-th partial sum of the series in[Eq\.78](https://arxiv.org/html/2609.02987#A3.E78)\. Its limit exists because it equals the left side at everyTT, and by[Eq\.81](https://arxiv.org/html/2609.02987#A3.E81)that limit isJTailRL​\(θ,x\)J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\. This is[Eq\.78](https://arxiv.org/html/2609.02987#A3.E78)\. ∎

Every term of the series is nonpositive, since no Best\-of\-kkvalue exceeds11, so the partial sums decrease monotonically to the tail\-likelihood: each truncation is an upper bound onJTailRLJ\_\{\\mathrm\{TailRL\{\}\}\}, tightening asTTgrows\. At the other end,JTailRL\(1\)​\(θ,x\)=Best\-of\-​1​\(θ,x\)−1=JRL​\(θ,x\)−1J^\{\(1\)\}\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\text\{Best\-of\-\}1\(\\theta;x\)\-1=J\_\{\\mathrm\{RL\}\}\(\\theta;x\)\-1, so the first member of the family is expected reward reinforcement learning up to an additive constant and shares its gradient\.

###### Proposition 11\(Harmonic Best\-of\-kkexpansion of the gradient\)\.

Under[3](https://arxiv.org/html/2609.02987#Thmtheorem3),[4](https://arxiv.org/html/2609.02987#Thmtheorem4)and[5](https://arxiv.org/html/2609.02987#Thmtheorem5), for everyθ∈Θ\\theta\\in\\Thetaand everyT≥1T\\geq 1,

∇θJTailRL\(T\)​\(θ,x\)=∑k=1T1k​∇θ*Best\-of\-*​k​\(θ,x\)=∫011−\(1−pθ​\(x,τ\)\)Tpθ​\(x,τ\)​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}J^\{\(T\)\}\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\,\\nabla\_\{\\theta\}\\,\\text\{\\emph\{Best\-of\-\}\}k\(\\theta;x\)=\\int\_\{0\}^\{1\}\\frac\{1\-\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{T\}\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-grad\}\(82\)

Proof\.[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6)moves the gradient inside the threshold integral of[Eq\.76](https://arxiv.org/html/2609.02987#A3.E76), and the chain rule gives

∇θBest\-of\-​k​\(θ,x\)=k​∫01\(1−pθ​\(x,τ\)\)k−1​∇θpθ​\(x,τ\)​𝑑τ\.\\nabla\_\{\\theta\}\\,\\text\{Best\-of\-\}k\(\\theta;x\)=k\\int\_\{0\}^\{1\}\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\-1\}\\,\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-bok\-grad\}\(83\)Multiply[Eq\.83](https://arxiv.org/html/2609.02987#A3.E83)by1/k1/k, sum overk=1,…,Tk=1,\\ldots,T, and exchange the finite sum with the integral:

∑k=1T1k​∇θBest\-of\-​k​\(θ,x\)=∫01\[∑k=1T\(1−pθ​\(x,τ\)\)k−1\]​∇θpθ​\(x,τ\)​𝑑τ\.\\sum\_\{k=1\}^\{T\}\\frac\{1\}\{k\}\\,\\nabla\_\{\\theta\}\\,\\text\{Best\-of\-\}k\(\\theta;x\)=\\int\_\{0\}^\{1\}\\Bigg\[\\sum\_\{k=1\}^\{T\}\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)^\{k\-1\}\\Bigg\]\\,\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-harmonic\-grad\-sum\}\(84\)The bracket is a finite geometric sum,∑k=1T\(1−p\)k−1=\(1−\(1−p\)T\)/p\\sum\_\{k=1\}^\{T\}\(1\-p\)^\{k\-1\}=\\big\(1\-\(1\-p\)^\{T\}\\big\)/p, which gives the right side of[Eq\.82](https://arxiv.org/html/2609.02987#A3.E82)\. The left equality is[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6)again, differentiating[Eq\.77](https://arxiv.org/html/2609.02987#A3.E77)term by term\. Every coefficient1/k1/kis positive, so the truncated gradient is a fixed, positively weighted combination of the Best\-of\-kkgradients fork=1,…,Tk=1,\\ldots,T\. ∎

### C\.5Unbiasedness of the finite rollout Estimator

This section proves[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2), restated here in full\.

[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2)\(Unbiasedness of the TailRL estimator, restated\)\.The unbiased estimator of∇θJTailRL\(N\)​\(θ,x\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)can be expressed as,

gTailRL\(N\)​\(x\):=∑i=1Nω⁡\(ri\)​Si,ω⁡\(ri\):=∫0rid​τ∑j=1N𝟙\{rj\>τ\}g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\):=\\sum\_\{i=1\}^\{N\}\\omega\(r\_\{i\}\)\\,S\_\{i\},\\qquad\\omega\(r\_\{i\}\):=\\int\_\{0\}^\{r\_\{i\}\}\\frac\{d\\tau\}\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\_\{j\}\>\\tau\\\}\}\}\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weight\-restated\}\(85\)*Under[3](https://arxiv.org/html/2609.02987#Thmtheorem3),[4](https://arxiv.org/html/2609.02987#Thmtheorem4)and[5](https://arxiv.org/html/2609.02987#Thmtheorem5), the estimatorgTailRL\(N\)​\(x\):=∑i=1Nω⁡\(r⁡\(x,zi\)\)​S​\(x,zi\)g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\):=\\sum\_\{i=1\}^\{N\}\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)\\,S\(x,z\_\{i\}\)is unbiased for the order\-NNtruncated gradient:*

𝔼z1:N\[gTailRL\(N\)\(x\)\]=∇θJTailRL\(N\)\(θ;x\)\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\\right\]=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-unbiasedness\-restated\}\(86\)
Proof of[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2)\.

The weight[Eq\.85](https://arxiv.org/html/2609.02987#A3.E85)for rolloutiiover its integration range clears the threshold, so its denominator is at least one\. The first step extends the integral from\[0,r⁡\(x,zi\)\)\[0,r\(x,z\_\{i\}\)\)to\[0,1\)\[0,1\)with an integrand that vanishes beyondr⁡\(x,zi\)r\(x,z\_\{i\}\)\. We cap the denominator below at one; the cap is active only where the numerator already vanishes:

ω⁡\(r⁡\(x,zi\)\)=∫0rid​τ∑j=1N𝟙\{rj\>τ\}=∫01𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)​𝑑τ\.\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)=\\int\_\{0\}^\{r\_\{i\}\}\\frac\{d\\tau\}\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\_\{j\}\>\\tau\\\}\}\}=\\int\_\{0\}^\{1\}\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weight\-indicator\}\(87\)
Every ratio from here on carries this capped denominator, so no expression in the proof is ever indeterminate\. Multiplying byS⁡\(x,zi\)S\(x,z\_\{i\}\)with their weightsω⁡\(r⁡\(x,zi\)\)\\omega\(r\(x,z\_\{i\}\)\)and summing over the group, we make use of the score\-functions being independent of the thresholds, so we swap the order of integration and summation:

gTailRL\(N\)​\(x\)=∫01∑i=1N𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)​𝑑τ\.g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)=\\int\_\{0\}^\{1\}\\frac\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-threshold\-form\}\(88\)
*The chain of expectations\.*Take the expectation over the training group\.[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6)moves it inside the integral over thresholdsτ\\tau, and linearity then moves it inside the finite sum over rollouts:

𝔼z1:N\[gTailRL\(N\)\(x\)\]\\displaystyle\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\\right\]=∫01𝔼z1:N\[\(∑i=1N𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\)\]dτ\\displaystyle=\\int\_\{0\}^\{1\}\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\left\(\\sum\_\{i=1\}^\{N\}\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\right\)\\right\]d\\tau=∫01∑i=1N\(𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\]\)dτ\.\\displaystyle=\\int\_\{0\}^\{1\}\\sum\_\{i=1\}^\{N\}\\left\(\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\right\]\\right\)d\\tau\.\(89\)By regularity condition[Lemma6](https://arxiv.org/html/2609.02987#Thmtheorem6), this integral is bounded\. The next tool we use is the tower property of conditional expectation: for random variablesXXandYYon the same support,

𝔼X​\[X\]=𝔼Y​\[𝔼X​\[X∣Y\]\],\\mathbb\{E\}\_\{X\}\\\!\\left\[X\\right\]=\\mathbb\{E\}\_\{Y\}\\\!\\left\[\\,\\mathbb\{E\}\_\{X\}\\\!\\left\[X\\mid Y\\right\]\\right\],\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-tower\-property\}\(90\)where the inner expectation averagesXXwithYYheld at its realized value and the outer expectation averages the result overYY\. We apply it with the substitution

X:=𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\),Y:=\{𝟙\{r\(x,zi\)\>τ\}\}i=1N,X:=\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\},\\qquad Y:=\\\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\\}\_\{i=1\}^\{N\},\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-xy\-def\}\(91\)the clearance pattern of the group at thresholdτ\\tau; both are functions of the training groupz1:Nz\_\{1:N\}, so every expectation below averages overz1:Nz\_\{1:N\}\. The tower property[Eq\.90](https://arxiv.org/html/2609.02987#A3.E90)gives

𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\]=𝔼z1:N\[𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\|\{𝟙\{r\(x,zi\)\>τ\}\}i=1N\]\]\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\right\]=\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\,\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\left\.\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\,\\right\|\\,\\\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\\}\_\{i=1\}^\{N\}\\right\]\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-step\-tower\}\(92\)Before taking the inner expectation, note one pointwise identity\. An indicator equals its own square, so multiplying the fraction by a second copy of rolloutii’s indicator changes nothing:

𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)1\{r\(x,zi\)\>τ\}=𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\.\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\;\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}=\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-absorb\}\(93\)SoXXfactors into the fraction times the conditional score:

X=𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)1\{r\(x,zi\)\>τ\}S\(x,zi\),Y=\{𝟙\{r\(x,zj\)\>τ\}\}j=1N\.X=\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\;\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\),\\qquad Y=\\bigl\\\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\bigr\\\}\_\{j=1\}^\{N\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-x\-split\}\(94\)Conditioning on\{𝟙\{r\(x,zj\)\>τ\}\}j=1N\\bigl\\\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\bigr\\\}\_\{j=1\}^\{N\}determines allNNindicator functions, so the expression simplifies as:

𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\|Y\]=𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)𝔼z1:N\[1\{r\(x,zi\)\>τ\}S\(x,zi\)\|Y\]\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\left\.\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\,\\right\|\\,Y\\right\]=\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\;\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\,\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\\,\\middle\|\\,Y\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-step\-pullout\}\(95\)It remains to compute the conditional mean of the gated score\. The rollouts are independent, so the only part ofYYthat constrainsziz\_\{i\}is its own indicator, and we split on the two values it can take\. If rolloutiiclears,ziz\_\{i\}follows the policy restricted to the clearing set,πθ\(z∣x\)1\{r\(x,z\)\>τ\}/pθ\(x,τ\)\\pi\_\{\\theta\}\(z\\mid x\)\\,\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}/p\_\{\\theta\}\(x,\\tau\); the gate is one everywhere on this support, so the mean is the clearing\-rollout mean score, whose norm is at mostCCby[Assumption5](https://arxiv.org/html/2609.02987#Thmtheorem5)\. If rolloutiifails,ziz\_\{i\}is supported onr⁡\(x,z\)≤τr\(x,z\)\\leq\\tau, where the gate is zero, so the mean is exactly the zero vector\. Both branches are finite, and together they give:

𝔼z1:N\[1\{r\(x,zi\)\>τ\}S\(x,zi\)\|Y\]=𝟙\{r\(x,zi\)\>τ\}𝔼z∼πθ\(⋅∣x\)\[S\(x,z\)∣r\(x,z\)\>τ\]\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\,\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\\,\\middle\|\\,Y\\right\]=\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\;\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[S\(x,z\)\\mid r\(x,z\)\>\\tau\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-gated\-mean\}\(96\)Substituting[Eq\.96](https://arxiv.org/html/2609.02987#A3.E96)into[Eq\.95](https://arxiv.org/html/2609.02987#A3.E95)and applying[Eq\.93](https://arxiv.org/html/2609.02987#A3.E93)once more removes the extra indicator:

𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\|Y\]=𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)𝔼z∼πθ\(⋅∣x\)\[S\(x,z\)∣r\(x,z\)\>τ\]\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\left\.\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\,\\right\|\\,Y\\right\]=\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\;\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[S\(x,z\)\\mid r\(x,z\)\>\\tau\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-step\-branch\}\(97\)Finally the outer expectation of[Eq\.92](https://arxiv.org/html/2609.02987#A3.E92)wraps over[Eq\.97](https://arxiv.org/html/2609.02987#A3.E97)\. The clearing\-rollout mean score\-function is a fixed vector, so by linearity the outer expectation acts only on the fraction of indicators:

𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}S\(x,zi\)max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\]=𝔼z∼πθ\(⋅∣x\)\[S\(x,z\)∣r\(x,z\)\>τ\]×𝔼z1:N\[𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)\]\.\\begin\{split\}\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\\,S\(x,z\_\{i\}\)\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\right\]&=\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[S\(x,z\)\\mid r\(x,z\)\>\\tau\\right\]\\\\\[2\.0pt\] &\\quad\\times\\;\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}\\right\]\.\\end\{split\}\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-per\-term\}\(98\)Substituting[Eq\.98](https://arxiv.org/html/2609.02987#A3.E98)into[Eq\.89](https://arxiv.org/html/2609.02987#A3.E89), summing overii, and recombining theNNterms under one expectation by linearity leaves the sum of the fractions, which share one denominator and collapse to a single ratio of the same count:

∑i=1N𝟙\{r\(x,zi\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)=∑j=1N𝟙\{r\(x,zj\)\>τ\}max\(1,∑j=1N𝟙\{r\(x,zj\)\>τ\}\)=𝟙\{∑j=1N𝟙\{r\(x,zj\)\>τ\}≥1\}\.\\sum\_\{i=1\}^\{N\}\\frac\{\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{i\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}=\\frac\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\}\{\\max\\Bigl\(1,\\;\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\Bigr\)\}=\\mathbbm\{1\}\_\{\\\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\geq 1\\\}\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-share\-identity\}\(99\)The last equality is checked outcome by outcome: if the number of rollouts attaining a reward above the threshold is≥1\\geq 1, the cap is inactive and the ratio is=1=1; if no rollout clears, the ratio is0/1=00/1=0\. The expectation of this event indicator is its probability, and theNNrollouts fail the threshold independently, each with probability1−pθ​\(x,τ\)1\-p\_\{\\theta\}\(x,\\tau\):

𝔼z1:N\[𝟙\{∑j=1N𝟙\{r\(x,zj\)\>τ\}≥1\}\]=1−\(1−pθ\(x,τ\)\)N\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\\geq 1\\\}\}\\right\]=1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-survive\-prob\}\(100\)The remaining constant is evaluated by the definition of conditional expectation given an event,𝔼⁡\[S∣A\]=𝔼⁡\[S​1A\]/Pr⁡\(A\)\\mathbb\{E\}\[S\\mid A\]=\\mathbb\{E\}\[S\\,\\mathbbm\{1\}\_\{A\}\]/\\Pr\(A\), withA=\{r\(x,z\)\>τ\}A=\\\{r\(x,z\)\>\\tau\\\}andPrz∼πθ\(⋅∣x\)\(A\)=pθ\(x,τ\)\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\(A\)=p\_\{\\theta\}\(x,\\tau\); its numerator is the score\-function identity[Eq\.40](https://arxiv.org/html/2609.02987#A3.E40):

𝔼z∼πθ\(⋅∣x\)\[S\(x,z\)∣r\(x,z\)\>τ\]=𝔼z∼πθ\(⋅∣x\)\[𝟙\{r\(x,z\)\>τ\}S\(x,z\)\]pθ​\(x,τ\)=∇θpθ​\(x,τ\)pθ​\(x,τ\)\.\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[S\(x,z\)\\mid r\(x,z\)\>\\tau\\right\]=\\frac\{\\mathbb\{E\}\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\mathbbm\{1\}\_\{\\\{r\(x,z\)\>\\tau\\\}\}\\,S\(x,z\)\\right\]\}\{p\_\{\\theta\}\(x,\\tau\)\}=\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-conditional\-mean\}\(101\)
*Assembling the chain\.*Substituting[Eqs\.98](https://arxiv.org/html/2609.02987#A3.E98),[99](https://arxiv.org/html/2609.02987#A3.E99),[100](https://arxiv.org/html/2609.02987#A3.E100)and[101](https://arxiv.org/html/2609.02987#A3.E101)into[Eq\.89](https://arxiv.org/html/2609.02987#A3.E89)yields the truncated gradient of[Eq\.49](https://arxiv.org/html/2609.02987#A3.E49):

𝔼z1:N\[gTailRL\(N\)\(x\)\]=∫01\(1−\(1−pθ\(x,τ\)\)N\)∇θpθ​\(x,τ\)pθ​\(x,τ\)dτ=∇θJTailRL\(N\)\(θ;x\)\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\\right\]=\\int\_\{0\}^\{1\}\\Bigl\(1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\\Bigr\)\\frac\{\\nabla\_\{\\theta\}p\_\{\\theta\}\(x,\\tau\)\}\{p\_\{\\theta\}\(x,\\tau\)\}\\,d\\tau=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-finite\-unbiasedness\}\(102\)∎

The proof identifies the estimator as MaxRL run at every threshold on one shared group of rollouts: wherever any rollout clears a threshold the cap is inactive and the integrand of[Eq\.88](https://arxiv.org/html/2609.02987#A3.E88)is the average score\-function of the clearing rollouts, MaxRL’s success\-averaging rule for the threshold event, while at thresholds no rollout clears it is zero, MaxRL’s rule for a group with no successes; its expectation carries MaxRL’s order\-NNtruncated weight from[Eq\.34](https://arxiv.org/html/2609.02987#S5.E34), and the binary case, where a single threshold carries all the mass, recovers MaxRL exactly \([Corollary14](https://arxiv.org/html/2609.02987#Thmtheorem14)\)\.

##### Loss reduction

The estimator is a sum over the group\. A mean\-reduced policy\-gradient loss, which divides the group sum byNN, must therefore use the coefficientsN​ω​\(r⁡\(x,zi\)\)N\\,\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\), and after mean\-centeringN⁡\(ω⁡\(r⁡\(x,zi\)\)−ω¯\)N\\bigl\(\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)\-\\bar\{\\omega\}\\bigr\)\. A sum\-reduced loss usesω⁡\(r⁡\(x,zi\)\)\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\), and after mean\-centeringω⁡\(r⁡\(x,zi\)\)−ω¯\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)\-\\bar\{\\omega\}\.

### C\.6Closed\-Form Weights and Algorithm Correctness

The next proposition proves the recurrence in[Eq\.27](https://arxiv.org/html/2609.02987#S4.E27)\. Throughout,z1,…,zNz\_\{1\},\\ldots,z\_\{N\}is the training group of[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2), andr\(1\)≤⋯≤r\(N\)r\_\{\(1\)\}\\leq\\cdots\\leq r\_\{\(N\)\}denote the group rewardsr⁡\(x,z1\),…,r⁡\(x,zN\)r\(x,z\_\{1\}\),\\ldots,r\(x,z\_\{N\}\)sorted increasingly, withr\(0\):=0r\_\{\(0\)\}:=0\. The weight[Eq\.85](https://arxiv.org/html/2609.02987#A3.E85)depends on a rollout only through its reward, so rollouts with equal rewards receive equal weights andω⁡\(r\(i\)\)\\omega\\bigl\(r\_\{\(i\)\}\\bigr\)is well defined\.

###### Proposition 12\(Closed\-form empirical weights\)\.

For everyi=1,…,Ni=1,\\ldots,N,

ω⁡\(r\(i\)\)=∑k=1ir\(k\)−r\(k−1\)N−k\+1\.\\omega\\bigl\(r\_\{\(i\)\}\\bigr\)=\\sum\_\{k=1\}^\{i\}\\frac\{r\_\{\(k\)\}\-r\_\{\(k\-1\)\}\}\{N\-k\+1\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-ascending\-weights\}\(103\)Equivalently, all weights follow the recurrence

ω⁡\(r\(i\)\)=ω⁡\(r\(i−1\)\)\+r\(i\)−r\(i−1\)N−i\+1,ω⁡\(r\(0\)\)=0\.\\omega\\bigl\(r\_\{\(i\)\}\\bigr\)=\\omega\\bigl\(r\_\{\(i\-1\)\}\\bigr\)\+\\frac\{r\_\{\(i\)\}\-r\_\{\(i\-1\)\}\}\{N\-i\+1\},\\qquad\\omega\\bigl\(r\_\{\(0\)\}\\bigr\)=0\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weight\-recurrence\}\(104\)

Proof\.The empirical tail\-probability distribution is a piecewise constant function\. Fixk∈\{1,…,N\}k\\in\\\{1,\\ldots,N\\\}and a thresholdτ∈\[r\(k−1\),r\(k\)\)\\tau\\in\[r\_\{\(k\-1\)\},r\_\{\(k\)\}\)\. The rewardsr\(k\),…,r\(N\)r\_\{\(k\)\},\\ldots,r\_\{\(N\)\}are at leastr\(k\)r\_\{\(k\)\}and therefore exceedτ\\tau, while the rewardsr\(1\),…,r\(k−1\)r\_\{\(1\)\},\\ldots,r\_\{\(k\-1\)\}are at mostr\(k−1\)≤τr\_\{\(k\-1\)\}\\leq\\tauand fail, since clearance is strict:

∑j=1N𝟙\{r\(x,zj\)\>τ\}=N−k\+1for everyτ∈\[r\(k−1\),r\(k\)\)\.\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}=N\-k\+1\\qquad\\text\{for every \}\\tau\\in\[r\_\{\(k\-1\)\},r\_\{\(k\)\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-piecewise\-count\}\(105\)The intervals\[r\(k−1\),r\(k\)\)\[r\_\{\(k\-1\)\},r\_\{\(k\)\}\)fork=1,…,ik=1,\\ldots,ipartition the integration range\[0,r\(i\)\)\[0,r\_\{\(i\)\}\)of the weight, with tied rewards contributing empty intervals, so substituting[Eq\.105](https://arxiv.org/html/2609.02987#A3.E105)evaluates the integral interval by interval:

ω⁡\(r\(i\)\)=∫0r\(i\)d​τ∑j=1N𝟙\{r\(x,zj\)\>τ\}=∑k=1i∫r\(k−1\)r\(k\)d​τN−k\+1=∑k=1ir\(k\)−r\(k−1\)N−k\+1,\\omega\\bigl\(r\_\{\(i\)\}\\bigr\)=\\int\_\{0\}^\{r\_\{\(i\)\}\}\\frac\{d\\tau\}\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\(x,z\_\{j\}\)\>\\tau\\\}\}\}=\\sum\_\{k=1\}^\{i\}\\int\_\{r\_\{\(k\-1\)\}\}^\{r\_\{\(k\)\}\}\\frac\{d\\tau\}\{N\-k\+1\}=\\sum\_\{k=1\}^\{i\}\\frac\{r\_\{\(k\)\}\-r\_\{\(k\-1\)\}\}\{N\-k\+1\},\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weight\-partition\}\(106\)which is[Eq\.103](https://arxiv.org/html/2609.02987#A3.E103); every denominator on the range is at leastN−i\+1≥1N\-i\+1\\geq 1, so no ratio is ever indeterminate\. Subtracting[Eq\.103](https://arxiv.org/html/2609.02987#A3.E103)at ranksiiandi−1i\-1leaves the single termk=ik=i, which is the recurrence[Eq\.104](https://arxiv.org/html/2609.02987#A3.E104); the base case is the integral over the empty range\[0,r\(0\)\)\[0,r\_\{\(0\)\}\)\. ∎

##### Scale of the weights

Every weight is nonnegative, since every gapr\(k\)−r\(k−1\)r\_\{\(k\)\}\-r\_\{\(k\-1\)\}is nonnegative\. The total is the largest reward in the group: exchanging the order of the finite double sum counts each gap once per rank at or above it, and that count cancels its denominator, leaving a telescoping sum:

∑i=1Nω⁡\(r\(i\)\)=∑k=1N∑i=kNr\(k\)−r\(k−1\)N−k\+1=∑k=1N\(r\(k\)−r\(k−1\)\)=r\(N\)=max1≤j≤N⁡r⁡\(x,zj\)\.\\sum\_\{i=1\}^\{N\}\\omega\\bigl\(r\_\{\(i\)\}\\bigr\)=\\sum\_\{k=1\}^\{N\}\\;\\sum\_\{i=k\}^\{N\}\\frac\{r\_\{\(k\)\}\-r\_\{\(k\-1\)\}\}\{N\-k\+1\}=\\sum\_\{k=1\}^\{N\}\\bigl\(r\_\{\(k\)\}\-r\_\{\(k\-1\)\}\\bigr\)=r\_\{\(N\)\}=\\max\_\{1\\leq j\\leq N\}r\(x,z\_\{j\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-weight\-total\}\(107\)For rewards in\[0,1\]\[0,1\], the pre\-centering advantagesN​ω​\(r⁡\(x,zi\)\)N\\,\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)therefore sum toN​r\(N\)≤NN\\,r\_\{\(N\)\}\\leq Nand each lies in\[0,N​r\(N\)\]\[0,\\,N\\,r\_\{\(N\)\}\], so the update scale is controlled by the best reward observed in the group\.

### C\.7Effect of the Mean\-Centering Baseline

###### Proposition 13\(The mean baseline lowers the truncation order by one\)\.

ForN≥2N\\geq 2, let

ω¯:=1N​∑i=1Nω⁡\(r⁡\(x,zi\)\),g^TailRL\(N\)​\(x\):=∑i=1N\(ω⁡\(r⁡\(x,zi\)\)−ω¯\)​S​\(x,zi\)\.\\bar\{\\omega\}:=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\),\\qquad\\hat\{g\}\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\):=\\sum\_\{i=1\}^\{N\}\\Bigl\(\\omega\\bigl\(r\(x,z\_\{i\}\)\\bigr\)\-\\bar\{\\omega\}\\Bigr\)\\,S\(x,z\_\{i\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-centered\-estimator\}\(108\)Then

𝔼z1:N\[g^TailRL\(N\)\(x\)\]=∇θJTailRL\(N−1\)\(θ;x\)\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\hat\{g\}\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\\right\]=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\-1\)\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-centered\-target\}\(109\)

Proof\.By[Eq\.107](https://arxiv.org/html/2609.02987#A3.E107), the weights of a group sum to its largest reward, so the centered estimator is the uncentered one minus a max\-reward\-weighted sum of scores:

g^TailRL\(N\)​\(x\)=gTailRL\(N\)​\(x\)−1N​\(max1≤i≤N⁡r⁡\(x,zi\)\)​∑i=1NS⁡\(x,zi\)\.\\hat\{g\}\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)=g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\-\\frac\{1\}\{N\}\\Bigl\(\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\Bigr\)\\sum\_\{i=1\}^\{N\}S\(x,z\_\{i\}\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-centered\-decomposition\}\(110\)The subtracted term is the score\-function estimator of the Best\-of\-NNgradient: the group is one draw from the product policy, whose score\-function is∑i=1NS⁡\(x,zi\)\\sum\_\{i=1\}^\{N\}S\(x,z\_\{i\}\), so

∇θ𝔼z1:N\[max1≤i≤Nr\(x,zi\)\]=𝔼z1:N\[\(max1≤i≤Nr\(x,zi\)\)∑i=1NS\(x,zi\)\]\.\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\right\]=\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\Bigl\(\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\Bigr\)\\sum\_\{i=1\}^\{N\}S\(x,z\_\{i\}\)\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-best\-n\-gradient\}\(111\)Taking expectations in[Eq\.110](https://arxiv.org/html/2609.02987#A3.E110)and invoking[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2)for the first term and[Eq\.111](https://arxiv.org/html/2609.02987#A3.E111)for the second yields

𝔼z1:N\[g^TailRL\(N\)\(x\)\]=∇θJTailRL\(N\)\(θ;x\)−1N∇θ𝔼z1:N\[max1≤i≤Nr\(x,zi\)\]\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\hat\{g\}\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)\\right\]=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\-\\frac\{1\}\{N\}\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\right\]\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-centered\-expectation\}\(112\)It remains to identify the right side as the order\-\(N−1\)\(N\-1\)gradient\. The expected maximum is the Best\-of\-NNvalue, whose layer\-cake form is[Eq\.76](https://arxiv.org/html/2609.02987#A3.E76)atk=Nk=N:

𝔼z1:N\[max1≤i≤Nr\(x,zi\)\]=∫01\[1−\(1−pθ\(x,τ\)\)N\]dτ\.\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\right\]=\\int\_\{0\}^\{1\}\\Bigl\[1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}\\Bigr\]d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-best\-n\-layer\-cake\}\(113\)Meanwhile, the definitions of two consecutive truncations differ in one term of the inner sum:

JTailRL\(N\)\(θ;x\)−JTailRL\(N−1\)\(θ;x\)=−1N∫01\(1−pθ\(x,τ\)\)Ndτ\.J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\-J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\-1\)\}\(\\theta;x\)=\-\\frac\{1\}\{N\}\\int\_\{0\}^\{1\}\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{N\}d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-consecutive\-truncations\}\(114\)Combining[Eqs\.113](https://arxiv.org/html/2609.02987#A3.E113)and[114](https://arxiv.org/html/2609.02987#A3.E114)shows the two sides differ by a constant:

JTailRL\(N\)\(θ;x\)−JTailRL\(N−1\)\(θ;x\)=1N𝔼z1:N\[max1≤i≤Nr\(x,zi\)\]−1N\.J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\-J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\-1\)\}\(\\theta;x\)=\\frac\{1\}\{N\}\\,\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\right\]\-\\frac\{1\}\{N\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-truncation\-best\-n\}\(115\)The final term is constant inθ\\theta, so differentiating gives

∇θJTailRL\(N\)\(θ;x\)−1N∇θ𝔼z1:N\[max1≤i≤Nr\(x,zi\)\]=∇θJTailRL\(N−1\)\(θ;x\)\.\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\)\}\(\\theta;x\)\-\\frac\{1\}\{N\}\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{z\_\{1:N\}\}\\\!\\left\[\\max\_\{1\\leq i\\leq N\}r\(x,z\_\{i\}\)\\right\]=\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}^\{\(N\-1\)\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-centered\-gradient\-identity\}\(116\)Substituting[Eq\.116](https://arxiv.org/html/2609.02987#A3.E116)into[Eq\.112](https://arxiv.org/html/2609.02987#A3.E112)proves the result\. ∎

### C\.8Estimator\-Level Recovery of MaxRL on Binary Rewards

###### Corollary 14\(Binary\-reward recovery\)\.

Supposeri∈\{0,1\}r\_\{i\}\\in\\\{0,1\\\}and letM:=∑i=1N𝟙\{ri=1\}M:=\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\. IfM\>0M\>0, then every successful rollout receives weight1/M1/Mand every unsuccessful rollout receives weight zero\. Hence the uncentered TailRL estimator reduces to

gTailRL\(N\)\(x\)=1M∑i:ri=1Si,g\_\{\\mathrm\{TailRL\}\}^\{\(N\)\}\(x\)=\\frac\{1\}\{M\}\\sum\_\{i:r\_\{i\}=1\}S\_\{i\},\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-tailrl\-binary\-uncentered\}\(117\)which is the MaxRL estimator\. After mean\-centering, successful rollouts have weight1/M−1/N1/M\-1/Nand unsuccessful rollouts have weight−1/N\-1/N\. IfM=0M=0, all uncentered and centered weights are zero\.

Proof\.For binary rewards with∑i=1N𝟙\{ri=1\}\>0\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\>0,

ω⁡\(1\)=∫01d​τ∑i=1N𝟙\{ri=1\}=1∑i=1N𝟙\{ri=1\}\\omega\(1\)=\\int\_\{0\}^\{1\}\\frac\{d\\tau\}\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\}=\\frac\{1\}\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\}andω⁡\(0\)=0\\omega\(0\)=0\. The uncentered result follows immediately\. The mean weight is

ω¯=\(∑i=1N𝟙\{ri=1\}\)1∑i=1N𝟙\{ri=1\}⋅1N=1N,\\bar\{\\omega\}=\\left\(\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\\right\)\\frac\{1\}\{\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}\}\\cdot\\frac\{1\}\{N\}=\\frac\{1\}\{N\},which gives the centered weights\. When∑i=1N𝟙\{ri=1\}=0\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\\\{r\_\{i\}=1\\\}=0, every reward and hence every weight is zero\. ∎

## Appendix DTailRL on a General Reward Range

The main paper assumes rewards in\[0,1\]\[0,1\]only to simplify notation\. The same objective applies to any bounded reward range\.

Suppose the reward function returns rewards in an arbitrary bounded range,r⁡\(x,z\)∈\[rmin,rmax\]r\(x,z\)\\in\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\]withrmin<rmaxr\_\{\\mathrm\{min\}\}<r\_\{\\mathrm\{max\}\}\. For thresholdsτ∈\[rmin,rmax\)\\tau\\in\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\), define the tail\-probability as before:pθ\(x,τ\):=Prz∼πθ\(⋅∣x\)\[r\(x,z\)\>τ\]p\_\{\\theta\}\(x,\\tau\):=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\[r\(x,z\)\>\\tau\]\. The TailRL objective averages the log\-tail\-probability across this range:

JTailRL\[rmin,rmax\]​\(θ,x\):=1rmax−rmin​∫rminrmaxlog⁡pθ​\(x,τ\)​𝑑τ\.J\_\{\\mathrm\{TailRL\}\}^\{\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\]\}\(\\theta;x\):=\\frac\{1\}\{r\_\{\\mathrm\{max\}\}\-r\_\{\\mathrm\{min\}\}\}\\int\_\{r\_\{\\mathrm\{min\}\}\}^\{r\_\{\\mathrm\{max\}\}\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-general\-range\}\(118\)The factor1/\(rmax−rmin\)1/\(r\_\{\\mathrm\{max\}\}\-r\_\{\\mathrm\{min\}\}\)makes this the expected log\-tail\-probability under a threshold drawn uniformly from\[rmin,rmax\]\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\]\. It also prevents the scale of the objective from depending on the units of the reward\.

To recover the unit\-interval objective, substituteτ=rmin\+\(rmax−rmin\)​u\\tau=r\_\{\\mathrm\{min\}\}\+\(r\_\{\\mathrm\{max\}\}\-r\_\{\\mathrm\{min\}\}\)\\,uwithu∈\[0,1\)u\\in\[0,1\)and define the rescaled rewardr~:=\(r−rmin\)/\(rmax−rmin\)∈\[0,1\]\\tilde\{r\}:=\(r\-r\_\{\\mathrm\{min\}\}\)/\(r\_\{\\mathrm\{max\}\}\-r\_\{\\mathrm\{min\}\}\)\\in\[0,1\]\. Then

JTailRL\[rmin,rmax\]\(θ;x\)=∫01logPrz∼πθ\(⋅∣x\)\[r~\(x,z\)\>u\]du=JTailRL\(θ;x\)for the rewardr~\.J\_\{\\mathrm\{TailRL\}\}^\{\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\]\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\bigl\[\\tilde\{r\}\(x,z\)\>u\\bigr\]\\,du=J\_\{\\mathrm\{TailRL\}\}\(\\theta;\\,x\)\\quad\\text\{for the reward \}\\tilde\{r\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:tailrl\-affine\-invariance\}\(119\)Thus, applying TailRL to rewards in\[rmin,rmax\]\[r\_\{\\mathrm\{min\}\},r\_\{\\mathrm\{max\}\}\]is equivalent to applying the unit\-interval objective tor~\\tilde\{r\}\. Shifting or rescaling the reward only relabels its thresholds, so every result in the paper carries over directly\.

The finite rollout estimator behaves in the same way\. For sorted rewards, consecutive weights satisfy

ω\(i\)−ω\(i−1\)=r\(i\)−r\(i−1\)N−i\+1,\\omega\_\{\(i\)\}\-\\omega\_\{\(i\-1\)\}=\\frac\{r\_\{\(i\)\}\-r\_\{\(i\-1\)\}\}\{N\-i\+1\},\\addcontentsline\{lla\}\{section\}\{\\numberline q:general\-range\-gaps\}\(120\)so an affine rescaling of the rewards multiplies every weight byrmax−rminr\_\{\\mathrm\{max\}\}\-r\_\{\\mathrm\{min\}\}\. The normalization in[Eq\.118](https://arxiv.org/html/2609.02987#A4.E118)cancels this factor\. Without the normalization, the update direction remains unchanged and only its magnitude is rescaled\. The estimator can therefore operate on raw rewards from any bounded range without changing the algorithm\.

## Appendix EConnection to Ordinal Cross\-Entropy

Threshold decompositions turn an ordinal or continuous target into a family of binary events, one event per threshold, and fit each event with a binary classifier\([McCullagh, 1980](https://arxiv.org/html/2609.02987#bib.bib47);[Frank and Hall, 2001](https://arxiv.org/html/2609.02987#bib.bib82);[Li and Lin, 2006](https://arxiv.org/html/2609.02987#bib.bib83);[Niu et al\., 2016](https://arxiv.org/html/2609.02987#bib.bib49);[Cao et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib50)\)\. This appendix makes the relation to TailRL exact\. The population TailRL objective is the ordinal cross\-entropy objective evaluated at the maximal targetr=1r=1, and the finite rollout TailRL procedure is the finite sample estimation of that objective\.

Fix an inputxxand recall the tail\-probabilitypθ\(x,τ\)=Prz∼πθ\(⋅∣x\)\(r\(x,z\)\>τ\)p\_\{\\theta\}\(x,\\tau\)=\\Pr\_\{z\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\(r\(x,z\)\>\\tau\\right\)\. For a target levelt∈\[0,1\]t\\in\[0,1\], define the threshold\-decomposed cross\-entropy between the point targetttand the model’s family of threshold events,

CE\(t;θ,x\):=−∫01\[1\{t\>τ\}logpθ\(x,τ\)\+1\{t≤τ\}log\(1−pθ\(x,τ\)\)\]dτ\.\\mathrm\{CE\}\(t;\\theta,x\):=\-\\int\_\{0\}^\{1\}\\Big\[\\,\\mathbbm\{1\}\_\{\\\{t\>\\tau\\\}\}\\,\\log p\_\{\\theta\}\(x,\\tau\)\\;\+\\;\\mathbbm\{1\}\_\{\\\{t\\leq\\tau\\\}\}\\,\\log\\big\(1\-p\_\{\\theta\}\(x,\\tau\)\\big\)\\,\\Big\]\\,d\\tau\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:ordinal\-ce\-def\}\(121\)For each fixedτ\\tauthe bracket is the binary cross\-entropy of the event\{r\>τ\}\\\{r\>\\tau\\\}against the label𝟙\{t\>τ\}\\mathbbm\{1\}\_\{\\\{t\>\\tau\\\}\}, and the integral weights all thresholds by the same uniform measure that defines the tail\-likelihood\.

Setting the target to the maximal rewardt=1t=1makes𝟙\{t\>τ\}=1\\mathbbm\{1\}\_\{\\\{t\>\\tau\\\}\}=1for everyτ∈\[0,1\)\\tau\\in\[0,1\), so the second term in[Eq\.121](https://arxiv.org/html/2609.02987#A5.E121)vanishes on a set of full measure and

CE\(1;θ,x\)=−∫01logpθ\(x,τ\)dτ=−JTailRL\(θ;x\)\.\\mathrm\{CE\}\(1;\\theta,x\)\\;=\\;\-\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)\\,d\\tau\\;=\\;\-\\,J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:ordinal\-ce\-collapse\}\(122\)Maximizing the tail\-likelihood is therefore exactly minimizing the ordinal cross\-entropy against the ideal targetr=1r=1\. When the reward is binary the tail\-probability is constant inτ\\tau, the integral collapses to a single binary cross\-entropy against the label11, and[Eq\.122](https://arxiv.org/html/2609.02987#A5.E122)reduces to the MaxRL objectivelog⁡qθ​\(x\)\\log q\_\{\\theta\}\(x\), consistent with[Section3\.2](https://arxiv.org/html/2609.02987#S3.SS2)\.

The correspondence extends to the finite rollout procedure\. The order\-NNobjectiveJTailRL\(N\)J^\{\(N\)\}\_\{\\mathrm\{TailRL\{\}\}\}of[Eq\.23](https://arxiv.org/html/2609.02987#S4.E23)truncates the same integral at the resolution a group ofNNrollouts can support, and the estimator of[Theorem2](https://arxiv.org/html/2609.02987#Thmtheorem2)is unbiased for its gradient using only theNNsampled rewards\. Training with TailRL onNNrollouts is in this sense the finite sample estimation of the ordinal cross\-entropy objective at targetr=1r=1: the rollouts play the role of the samples from which the threshold events are estimated, and the truncation order grows with the sample size, recovering[Eq\.122](https://arxiv.org/html/2609.02987#A5.E122)asN→∞N\\to\\infty\.

## Appendix FAdvantage Functions of GRPO, RLOO, and TailRL

All three methods share the same critic\-free template: drawNNrollouts, map their rewards to advantages, and form the update∑i=1NAi​S​\(x,zi\)\\sum\_\{i=1\}^\{N\}A\_\{i\}\\,S\(x,z\_\{i\}\)\. They differ only in that map, which we record here\. For convenience of notation, we writeri:=r⁡\(x,zi\)r\_\{i\}:=r\(x,z\_\{i\}\)for the reward of rolloutziz\_\{i\}\.

##### RLOO

RLOO subtracts the leave\-one\-out mean of the other rollouts,

Ai\(RLOO\)=ri−1N−1​∑j≠irj\.A\_\{i\}^\{\(\\mathrm\{RLOO\}\)\}=r\_\{i\}\-\\frac\{1\}\{N\-1\}\\sum\_\{j\\neq i\}r\_\{j\}\.\(123\)The baseline is independent of rolloutii, so the update is an unbiased estimator of the expected reward gradient\. The advantage is affine inrir\_\{i\}with unit slope: a rollout is promoted by its raw margin over the rest of the group\.

##### GRPO

GRPO standardizes within the group,

Ai\(GRPO\)=ri−r¯σ⁡\(r\)\+ϵ,r¯=1N​∑j=1Nrj,σ​\(r\)2=1N​∑j=1N\(rj−r¯\)2\.A\_\{i\}^\{\(\\mathrm\{GRPO\}\)\}=\\frac\{r\_\{i\}\-\\bar\{r\}\}\{\\sigma\(r\)\+\\epsilon\},\\qquad\\bar\{r\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}r\_\{j\},\\qquad\\sigma\(r\)^\{2\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\(r\_\{j\}\-\\bar\{r\}\)^\{2\}\.\(124\)Dividing by the group standard deviation makes the update invariant to affine rescaling of the reward\. The same divisor applies to every rollout in the group, so it changes the size of an input’s update rather than the relative weight of rewards within it\.

##### TailRL

TailRL assigns weights proportional to inverse tail\-probability’s integral and centers it,

Ai\(TailRL\)=ω⁡\(ri\)−ω¯,ω⁡\(ri\)=∫0rid​τ∑j=1N𝟙\{rj\>τ\},ω¯=1N​∑j=1Nω⁡\(rj\)\.A\_\{i\}^\{\(\\mathrm\{TailRL\{\}\}\)\}=\\omega\(r\_\{i\}\)\-\\bar\{\\omega\},\\qquad\\omega\(r\_\{i\}\)=\\int\_\{0\}^\{r\_\{i\}\}\\frac\{d\\tau\}\{\\sum\_\{j=1\}^\{N\}\\mathbbm\{1\}\_\{\\\{r\_\{j\}\>\\tau\\\}\}\},\\qquad\\bar\{\\omega\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\omega\(r\_\{j\}\)\.\(125\)The integrand is the reciprocal of the number of rollouts attaining a reward above a given reward threshold, so a level cleared by one rollout out ofNNcontributesNNtimes the weight per unit of reward of one cleared by all\. The map is computed in closed form by the recurrence in[Eq\.27](https://arxiv.org/html/2609.02987#S4.E27)and reduces to the MaxRL advantages when the reward is binary \([SectionC\.8](https://arxiv.org/html/2609.02987#A3.SS8)\)\.

## Appendix GPass@kkand Best\-of\-kkEmpirical Calculation

Pass@kkis used for binary rewards and measures the probability that at least one rollout succeeds\. Best\-of\-kkis used for continuous rewards and measures the expected reward of the highest\-scoring rollout\. Both metrics evaluate a policy when we samplekkrollouts and keep the best one\. The two metrics are identical for binary rewards, and both improve or remain unchanged askkincreases\. When we haveKKevaluation rollouts to estimate Pass@kkor Best\-of\-kkwithK\>kK\>k, we use allKKrollouts to compute their unbiased estimators below\.

### G\.1Pass@kk

IfMMof theKKsampled rollouts succeed, the unbiased estimator of[Chen et al\. \(2021\)](https://arxiv.org/html/2609.02987#bib.bib51)is

Pass​@​k^=1−\(K−Mk\)\(Kk\)\.\\widehat\{\\mathrm\{Pass@\}k\}=1\-\\frac\{\\binom\{K\-M\}\{k\}\}\{\\binom\{K\}\{k\}\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-passk\-estimator\}\(126\)The ratio is the probability that a uniformly random size\-kksubset of theKKrollouts avoids allMMsuccesses\.

### G\.2Best\-of\-kk

Best\-of\-kkis defined as in[Eq\.76](https://arxiv.org/html/2609.02987#A3.E76)

Best\-of\-​k​\(θ,x\)=𝔼⁡\[max1≤i≤k⁡r⁡\(x,zi\)\]=∫01\[1−\(1−pθ​\(x,τ\)\)k\]​𝑑τ,\\text\{Best\-of\-\}k\(\\theta;x\)=\\mathbb\{E\}\\\!\\left\[\\max\_\{1\\leq i\\leq k\}r\(x,z\_\{i\}\)\\right\]=\\int\_\{0\}^\{1\}\\left\[1\-\\left\(1\-p\_\{\\theta\}\(x,\\tau\)\\right\)^\{k\}\\right\]d\\tau,\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-bok\-restated\}\(127\)the second form by the layer\-cake identity applied to the maximum, which reduces to Pass@kkwhen the reward is binary\. Sorting theKKsampled rewards increasingly asr\(1\)≤⋯≤r\(K\)r\_\{\(1\)\}\\leq\\cdots\\leq r\_\{\(K\)\}, the unbiased estimator weights each order statistic by the probability that it is the maximum of a uniformly random size\-kksubset:

Best\-of\-​k^=∑i=kK\(i−1k−1\)\(Kk\)​r\(i\)\.\\widehat\{\\text\{Best\-of\-\}k\}=\\sum\_\{i=k\}^\{K\}\\frac\{\\binom\{i\-1\}\{k\-1\}\}\{\\binom\{K\}\{k\}\}\\,r\_\{\(i\)\}\.\\addcontentsline\{lla\}\{section\}\{\\numberline q:app\-bok\-estimator\}\(128\)

## Appendix HImageNet Object Localization

We study single\-object localization on ImageNet\-scale data\([Russakovsky et al\., 2015](https://arxiv.org/html/2609.02987#bib.bib32)\)\. Given an image, the policy predicts a bounding box and receives its intersection\-over\-union \(IoU\) with the ground\-truth box as a reward in\[0,1\]\[0,1\]\.

The policy uses a pretrained ResNet\-50 backbone\([He et al\., 2016](https://arxiv.org/html/2609.02987#bib.bib33)\)with four categorical heads, one for each box coordinate\. Each head contains5050uniformly spaced bins, and a rollout samples one bin from each head to form a box\. All methods train for3030epochs with Adam, a learning rate of5×10−45\\times 10^\{\-4\}with warmup, and a batch size of128128\. They differ only in how they compute the advantages\. We train TailRL withN∈\{16,64,256,1024\}N\\in\\\{16,64,256,1024\\\}rollouts and separately optimize the exact population\-level objective described in[SectionH\.2](https://arxiv.org/html/2609.02987#A8.SS2)\.

For comparison, we also train supervised models directly on the ground\-truth coordinates using MSE, L1, GIoU\([Rezatofighi et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib67)\), and L1\+GIoU losses \([SectionH\.1](https://arxiv.org/html/2609.02987#A8.SS1)\)\.

We evaluate on held\-out images using CorLoc@δ\\delta, mean IoU, and Best\-of\-10241024IoU\. CorLoc@δ\\deltais the fraction of images whose greedy prediction exceeds IoUδ\\delta\. Best\-of\-10241024IoU is the highest IoU among10241024sampled boxes for each image\. We report results over33seeds\. The large\-NNgradient measurements in[Fig\.6](https://arxiv.org/html/2609.02987#S6.F6)use one seed, as noted in the figure\.[Table3](https://arxiv.org/html/2609.02987#A8.T3)summarizes the full configuration\.

Table 3:Training hyperparameters for ImageNet object localization\.### H\.1Supervised Anchor Losses

Letb=\(x1,y1,x2,y2\)b=\(x\_\{1\},y\_\{1\},x\_\{2\},y\_\{2\}\)denote the predicted box andb⋆b^\{\\star\}the ground\-truth box\. Both use normalized corner coordinates\. The L1 loss measures the absolute error across the four coordinates:

ℒL1​\(b,b⋆\)=‖b−b⋆‖1=∑j=14‖bj−bj⋆‖1\.\\mathcal\{L\}\_\{\\mathrm\{L1\}\}\(b,b^\{\\star\}\)=\\left\\\|b\-b^\{\\star\}\\right\\\|\_\{1\}=\\sum\_\{j=1\}^\{4\}\\left\\\|b\_\{j\}\-b^\{\\star\}\_\{j\}\\right\\\|\_\{1\}\.\(129\)The GIoU loss\([Rezatofighi et al\., 2019](https://arxiv.org/html/2609.02987#bib.bib67)\)extends IoU with a penalty based on the smallest box enclosing both boxes\. We use\|b\|\|b\|to denote the area enclosed by the bounding boxbb\. LetI=\|b∩b⋆\|\\mathrm\{I\}=\|b\\cap b^\{\\star\}\|be the intersection area,U=\|b∪b⋆\|\\mathrm\{U\}=\|b\\cup b^\{\\star\}\|the union area, andccthe smallest axis\-aligned box enclosing bothbbandb⋆b^\{\\star\}and has an area\|c\|\|c\|:

ℒGIoU​\(b,b⋆\)=1−IU⏟IoU\+\|c\|−U\|c\|\.\\mathcal\{L\}\_\{\\mathrm\{GIoU\}\}\(b,b^\{\\star\}\)=1\-\\underbrace\{\\frac\{\\mathrm\{I\}\}\{\\mathrm\{U\}\}\}\_\{\\mathrm\{IoU\}\}\+\\frac\{\|c\|\-\\mathrm\{U\}\}\{\|c\|\}\.\(130\)The final term measures the empty space inside the enclosing box\. Unlike IoU, it remains informative when the predicted and ground\-truth boxes do not overlap\. The combined loss follows DETR\([Carion et al\., 2020](https://arxiv.org/html/2609.02987#bib.bib66)\):

ℒL1\+GIoU​\(b,b⋆\)=λL1​ℒL1​\(b,b⋆\)\+λGIoU​ℒGIoU​\(b,b⋆\)\.\\mathcal\{L\}\_\{\\mathrm\{L1\+GIoU\}\}\(b,b^\{\\star\}\)=\\lambda\_\{\\mathrm\{L1\}\}\\mathcal\{L\}\_\{\\mathrm\{L1\}\}\(b,b^\{\\star\}\)\+\\lambda\_\{\\mathrm\{GIoU\}\}\\mathcal\{L\}\_\{\\mathrm\{GIoU\}\}\(b,b^\{\\star\}\)\.\(131\)We use the DETR weightsλL1=5\\lambda\_\{\\mathrm\{L1\}\}=5andλGIoU=2\\lambda\_\{\\mathrm\{GIoU\}\}=2\. All three supervised anchors make one deterministic prediction and use no rollouts\. Their greedy, mean, and Best\-of\-kkoutputs are therefore identical\.

### H\.2Computing the Population\-Level Objective

Computing the population objectiveJTailRL​\(θ,x\)=∫01log⁡pθ​\(x,τ\)​𝑑τJ\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)=\\int\_\{0\}^\{1\}\\log p\_\{\\theta\}\(x,\\tau\)d\\tauusually requires the unknown tail\-probabilitypθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)\. In this setting, we can compute it exactly because the policy has a finite output space\. The policy predicts each of the four box coordinates with an independent categorical head overK=50K=50uniformly spaced bins\. A rollout samples one bin from each head, so its probability factorizes as

πθ​\(b∣x\)=∏j=14πθ\(j\)​\(bj∣x\),b∈ℬ,\|ℬ\|=K4=6,250,000\.\\pi\_\{\\theta\}\(b\\mid x\)=\\prod\_\{j=1\}^\{4\}\\pi\_\{\\theta\}^\{\(j\)\}\(b\_\{j\}\\mid x\),\\qquad b\\in\\mathcal\{B\},\\quad\|\\mathcal\{B\}\|=K^\{4\}=6\{,\}250\{,\}000\.\(132\)Each boxbbhas a deterministic rewardr⁡\(x,b\)r\(x,b\)given by its IoU with the ground\-truth box\. We can therefore compute\(πθ​\(b∣x\),r⁡\(x,b\)\)\\left\(\\pi\_\{\\theta\}\(b\\mid x\),r\(x,b\)\\right\)for every box inℬ\\mathcal\{B\}\. The tail\-probability at a threshold is the total probability of all boxes whose rewards exceed that threshold:

pθ\(x,τ\)=∑b∈ℬπθ\(b∣x\)𝟙\{r\(x,b\)\>τ\}p\_\{\\theta\}\(x,\\tau\)=\\sum\_\{b\\in\\mathcal\{B\}\}\\pi\_\{\\theta\}\(b\\mid x\)\\mathbbm\{1\}\_\{\\\{r\(x,b\)\>\\tau\\\}\}\(133\)The finite setℬ\\mathcal\{B\}produces a finite set of reward values\. The functionτ↦pθ​\(x,τ\)\\tau\\mapsto p\_\{\\theta\}\(x,\\tau\)is constant between consecutive reward values, so the threshold integral becomes a finite sum\. This calculation introduces no discretization error beyond the original coordinate bins\. In practice, we use the independence of the four heads to avoid explicitly enumerating allK4K^\{4\}boxes\.[Alg\.1](https://arxiv.org/html/2609.02987#alg1)forms the joint probabilities from the four marginal distributions and accumulates them over the sorted reward values\.

Algorithm 1Population\-level TailRL objective and gradient for one image1:Head distributions

πθ\(j\)\(⋅∣x\)\\pi\_\{\\theta\}^\{\(j\)\}\(\\cdot\\mid x\)over

KKbins,

j=1,…,4j=1,\\ldots,4; ground\-truth box

b⋆b^\{\\star\}
2:

ℬ←\{1,…,K\}4\\mathcal\{B\}\\leftarrow\\\{1,\\ldots,K\\\}^\{4\}// every box the policy can emit

3:for

b=\(b1,…,b4\)∈ℬb=\(b\_\{1\},\\ldots,b\_\{4\}\)\\in\\mathcal\{B\}do

4:

πθ​\(b∣x\)←∏j=14πθ\(j\)​\(bj∣x\)\\pi\_\{\\theta\}\(b\\mid x\)\\leftarrow\\prod\_\{j=1\}^\{4\}\\pi\_\{\\theta\}^\{\(j\)\}\(b\_\{j\}\\mid x\)// the heads are independent

5:

r⁡\(x,b\)←IoU⁡\(b,b⋆\)r\(x,b\)\\leftarrow\\mathrm\{IoU\}\(b,b^\{\\star\}\)
6:endfor

7:

u1<⋯<um←u\_\{1\}<\\cdots<u\_\{m\}\\leftarrowthe distinct values of

\{r⁡\(x,b\):b∈ℬ\}\\\{\\,r\(x,b\):b\\in\\mathcal\{B\}\\,\\\}, sorted increasingly

8:for

i=1,…,mi=1,\\ldots,mdo

9:

qi←∑b:r⁡\(x,b\)=uiπθ\(b∣x\)q\_\{i\}\\leftarrow\\sum\_\{b\\,:\\,r\(x,b\)=u\_\{i\}\}\\pi\_\{\\theta\}\(b\\mid x\)// reward distribution

10:endfor

11:

pθ​\(x,um\)←0p\_\{\\theta\}\(x,u\_\{m\}\)\\leftarrow 0// no box exceeds the largest attainable reward

12:for

i=m−1,…,1i=m\-1,\\ldots,1do

13:

pθ​\(x,ui\)←pθ​\(x,ui\+1\)\+qi\+1p\_\{\\theta\}\(x,u\_\{i\}\)\\leftarrow p\_\{\\theta\}\(x,u\_\{i\+1\}\)\+q\_\{i\+1\}//pθ​\(x,τ\)=pθ​\(x,ui\)p\_\{\\theta\}\(x,\\tau\)=p\_\{\\theta\}\(x,u\_\{i\}\)forτ∈\[ui,ui\+1\)\\tau\\in\[u\_\{i\},u\_\{i\+1\}\)

14:endfor

15:

JTailRL​\(θ,x\)←∑i=1m−1\(ui\+1−ui\)​log⁡pθ​\(x,ui\)J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\\leftarrow\\sum\_\{i=1\}^\{m\-1\}\(u\_\{i\+1\}\-u\_\{i\}\)\\log p\_\{\\theta\}\(x,u\_\{i\}\)
16:

∇θJTailRL​\(θ,x\)←∑i=1m−1\(ui\+1−ui\)​∇θpθ​\(x,ui\)pθ​\(x,ui\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)\\leftarrow\\sum\_\{i=1\}^\{m\-1\}\(u\_\{i\+1\}\-u\_\{i\}\)\\dfrac\{\\nabla\_\{\\theta\}\\,p\_\{\\theta\}\(x,u\_\{i\}\)\}\{p\_\{\\theta\}\(x,u\_\{i\}\)\}
17:

JTailRL​\(θ,x\)J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\)and

∇θJTailRL​\(θ,x\)\\nabla\_\{\\theta\}J\_\{\\mathrm\{TailRL\{\}\}\}\(\\theta;x\), exact, with no sampling error

Substituting the exactpθ​\(x,τ\)p\_\{\\theta\}\(x,\\tau\)into[Eq\.45](https://arxiv.org/html/2609.02987#A3.E45)gives the gradient used by the population variant\. This is an exact population gradient, not a large\-NNapproximation\. We obtain it by automatically differentiating through the probability accumulation because each tail\-probability value is a differentiable function of the prediction head probabilities\.

### H\.3Additional Results

##### Training dynamics

[Figure13](https://arxiv.org/html/2609.02987#A8.F13)shows training from three perspectives\. The left panel reports final held\-out CorLoc@0\.50\.5across training rollout budgetsNN\. TailRL improves steadily asNNincreases and approaches the exact population objective\. GRPO improves only slightly, while RLOO does not improve with additional rollouts\. At every budget, both baselines perform worse than TailRL at its smallest reported budget\.

The center panel shows the mean gradient norm by epoch atN=1024N=1024\. GRPO produces the largest gradients, while RLOO produces the smallest\. However, GRPO also achieves the lowest final accuracy, showing that larger gradients do not explain the performance differences in[Fig\.7](https://arxiv.org/html/2609.02987#S6.F7)\.The right panel shows each method’s training loss over RL steps using a logarithmic step axis\. Because the methods optimize different objectives, their loss values are not directly comparable\. Nevertheless, all three losses flatten near zero late in training\.

Figure 13:\(ImageNet object localization\)Left: final validation CorLoc@0\.50\.5against the training rollout budgetNNfor all three methods, with the population\-level objective as a dashed reference\. Center: mean gradient norm by epoch atN=1024N=1024, log scale\. Right: each method’s training loss against RL steps, logarithmic step axis; each method optimizes its own surrogate objective\. Means over the seeds of[Fig\.7](https://arxiv.org/html/2609.02987#S6.F7)\.Figure 14:\(ImageNet object localization\)Mean IoU by difficulty band for the three methods atN=1024N=1024\. Bands partition the validation set by the area of the ground\-truth box as a fraction of the image: easy is0\.300\.30to0\.700\.70\(19,62819\{,\}628images\), medium is0\.100\.10to0\.300\.30or0\.700\.70to0\.950\.95\(19,97719\{,\}977\), and hard is below0\.100\.10or above0\.950\.95\(10,39510\{,\}395\)\. Curves are means with standard\-error bands over33seeds\.[Figure14](https://arxiv.org/html/2609.02987#A8.F14)stratifies validation performance by difficulty band\. The bands are defined by the area of the ground\-truth box as a fraction of the image, since a box that fills a moderate part of the frame is far easier to localize than a very small or a nearly full\-frame one\. An image is easy when that area lies between0\.300\.30and0\.700\.70, medium when it lies between0\.100\.10and0\.300\.30or between0\.700\.70and0\.950\.95, and hard when it falls below0\.100\.10or above0\.950\.95; the three bands partition the50,00050\{,\}000validation images into19,62819\{,\}628,19,97719\{,\}977, and10,39510\{,\}395\. The gap between TailRL and the stronger expected reward baseline widens as the band gets harder: roughly0\.090\.09IoU on the easy band,0\.160\.16on medium, and0\.180\.18on hard, where both baselines sit near0\.300\.30and TailRL reaches0\.490\.49\. Hard inputs are those on which high\-reward rollouts are rare, and rarity is where the inverse\-probability weighting of the tail\-likelihood concentrates its effort, so the ordering of the gaps matches the mechanism the objective is built around\.SR: pkpo td

### H\.4The Effect of Binarizing the Reward

A continuous reward contains more information than binary reward\. Binarizing it with a threshold removes this information by treating all outputs above or below the threshold as equally goor or bad\. We test the cost of this lost signal by binarizing IoU at\{0\.5,0\.75\}\\\{0\.5,0\.75\\\}and training MaxRL on each binary reward\. We compare both variants with TailRL trained on the original continuous reward across four rollout budgets, with all other settings held fixed\.

Figure 15:\(ImageNet object localization\)Effect of binarizing the IoU reward on that task of ImageNet object localization\. We compare TailRL trained on continuous IoU with MaxRL trained on rewards binarized atIoU\>0\.5\\mathrm\{IoU\}\>0\.5andIoU\>0\.75\\mathrm\{IoU\}\>0\.75\. All methods perform similarly at the easiest evaluation threshold, but the binarized methods degrade at CorLoc above their binarization threshold\.Binarization produces policies that perform well near their chosen threshold but poorly at higher quality levels \([Fig\.15](https://arxiv.org/html/2609.02987#A8.F15)\)\. At CorLoc@0\.50\.5, all methods reach approximately0\.780\.78\. At CorLoc@0\.750\.75, MaxRL trained with the0\.50\.5threshold reaches only0\.300\.30, compared with approximately0\.630\.63for TailRL and MaxRL trained with the0\.750\.75threshold\. At CorLoc@0\.90\.9, the gap widens further: TailRL reaches approximately0\.390\.39, compared with0\.240\.24for the0\.750\.75threshold and0\.020\.02for the0\.50\.5threshold\.

Once a rollout has an IoU above its binarization threshold, the binary reward no longer distinguishes a barely acceptable box from a nearly perfect one\. This lost ordering produces both qualitative specialization to the chosen threshold and large quantitative losses above it\. Increasing the rollout budget cannot recover information removed from the reward, so the binarized methods flatten while TailRL continues to improve at stricter thresholds\. TailRL avoids choosing a specific threshold and learns from the full continuous reward, allowing one policy to perform well across all evaluated quality levels\.

## Appendix IText\-Maze Navigation

We study navigation in17×1717\\times 17gridworld mazes represented as text\. Given a maze, the policy generates a sequence of movement tokens\. The continuous reward measures whether the sequence is well formed, how close it ends to the goal, and how its length compares with the shortest path\. A rollout that reaches the goal along a shortest path receives a reward of11\.

The policy is a33M\-parameter decoder\-only transformer trained from scratch\. We first pretrain it through supervised learning on1\.31\.3M mazes with up to1616annotated paths per maze\. To test how each RL method behaves under different initial policy qualities, we retain seven pretraining checkpoints\. Their held\-out shortest\-path rates range from0\.012%0\.012\\%, or roughly one success in ten thousand attempts, to0\.83%0\.83\\%\.

Starting from each checkpoint, we train every method for5,0005\{,\}000steps usingN=16N=16rollouts for each of256256mazes per step\. Training is fully on\-policy, uses no KL regularization, and is repeated over three seeds\. We report the shortest\-path rate, defined as the probability that a single sampled rollout reaches the goal using an optimal\-length path\. We estimate this rate from6464rollouts on each of256256held\-out mazes\.

### I\.1Dataset and Pretraining

Each maze is a17×1717\\times 17grid with the start and goal at opposite corners\. We generate a perfect maze using Prim’s algorithm and then remove a uniformly sampled55to30%30\\%of its interior walls\. Removing these walls creates alternative routes and varies the number of valid paths across mazes\.

For pretraining, we pair each maze with up to1616goal\-reaching paths selected by[Alg\.2](https://arxiv.org/html/2609.02987#alg2)\. We first find the shortest\-path lengthL⋆L^\{\\star\}using breadth\-first search\. A budgeted depth\-first search then finds simple paths shorter thanub=60\\mathrm\{ub\}=60\. To prevent common path lengths from dominating the corpus, reservoir sampling retains at most four paths of each length\.

We select paths that span a range of solution qualities\. Each candidate pathPPreceives a reward, where a shortest path receives11and longer paths receive smaller values\. We divide rewardr∈\(0,1\]r\\in\(0,1\]into1616equal intervals and select at most one path from each nonempty interval\. When an interval contains multiple candidates, we select the path that overlaps least with those already chosen\. This produces paths that vary in both length and spatial route\. We train the policy on the resulting maze\-path pairs using next\-token prediction\.

Algorithm 2Selecting pretraining paths for one maze1:Maze

mm; length limit

ub=60\\mathrm\{ub\}=60; target number of paths

npaths=16n\_\{\\mathrm\{paths\}\}=16
2:

𝒞←\\mathcal\{C\}\\leftarrowall simple start\-to\-goal paths found by breadth\-first search with path length

<ub<\\mathrm\{ub\}
3:Divide

𝒞\\mathcal\{C\}into

npathsn\_\{\\mathrm\{paths\}\}equal buckets according to path length

4:

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset
5:foreach nonempty bucket

𝒞t\\mathcal\{C\}\_\{t\}do

6:Select one path

Pt∈𝒞tP\_\{t\}\\in\\mathcal\{C\}\_\{t\}
7:

𝒮←𝒮∪\{Pt\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\{\\\{P\_\{t\}\\\}\}
8:endfor

9:A set

𝒮\\mathcal\{S\}of paths with diverse lengths

### I\.2Post\-Training Configuration

The reinforcement\-learning stage starts from the checkpoints above and uses the configuration of[Table4](https://arxiv.org/html/2609.02987#A9.T4), which also lists the evaluation sampling\.

Table 4:Training hyperparameters for Text\-Maze navigation\.
### I\.3Reward Function

A rollout is parsed into a sequence of moves and replayed in the maze\. WriteL⋆L^\{\\star\}for the length of a shortest path from start to goal,LLfor the length of the rollout’s path when it reaches the goal, andddfor the breadth\-first distance from the rollout’s final cell to the goal\. The reward is the sum of a progress term and a solution term,

r⁡\(x,z\)=12​min⁡\(1,L⋆−dL⋆\)⏟progress\+12​min⁡\(1,L⋆L\)​𝟙\{goal reached\}⏟solution,r\(x,z\)=\\underbrace\{\\tfrac\{1\}\{2\}\\min\\\!\\left\(1,\\tfrac\{L^\{\\star\}\-d\}\{L^\{\\star\}\}\\right\)\}\_\{\\text\{progress\}\}\\;\+\\;\\underbrace\{\\tfrac\{1\}\{2\}\\min\\\!\\left\(1,\\tfrac\{L^\{\\star\}\}\{L\}\\right\)\\mathbbm\{1\}\_\{\\\{\\text\{goal reached\}\\\}\}\}\_\{\\text\{solution\}\},\(134\)with both terms clipped below at zero, and withr⁡\(x,z\)=0r\(x,z\)=0whenever the rollout is malformed or walks into a wall\. The progress term pays partial credit for ending closer to the goal than the start, so a rollout that never arrives is still ranked by how far it got\. The solution term pays full credit only for a shortest path and decays asL⋆/LL^\{\\star\}/Lwhen the policy reaches the goal by a longer route\. The reward equals11exactly when the rollout reaches the goal along a shortest path, the event called shortest\-path success in[Section6\.2](https://arxiv.org/html/2609.02987#S6.SS2)\.

### I\.4Task Representation and Prompt Template

A maze is serialized as a flat token sequence rather than as natural language, so the policy never sees English and cannot rely on pretrained language priors\. The grid is written cell by cell with one token per cell, rows separated by a newline token, and the prompt ends at the marker that opens the path; the model then generates the path itself as a sequence of coordinate tokens\. Each row of the17×1717\\times 17grid contributes seventeen cell tokens drawn fromWALL,PATH,START, andGOAL, followed byNEWLINE; a rollout is the continuation afterPATH\_START, a sequence of movement tokens closed byDONE\. It is scored for well\-formedness, for how close it ends to the goal, and for its length against the shortest path\.

17×1717\\times 17Maze Example Model Input and Output FormatInput:<bos\> GRID\_START WALL WALL WALL…\\ldotsWALL START PATH PATH…\\ldotsNEWLINE…\\ldotsGOAL WALL NEWLINE…\\ldotsGRID\_END PATH\_STARTOutput:RIGHT RIGHT DOWN DOWN…\\ldotsRIGHT DONE <eos\>

##### Reward examples

[Figure8](https://arxiv.org/html/2609.02987#S6.F8)shows four rollouts on one maze with the reward each receives\. The two left paths never reach the goal and are scored by how far they get, which is what makes the reward continuous rather than binary\. The third path reaches the goal but wanders, so it scores below a shortest path; only the rightmost path, which reaches the goal along a shortest path, receives reward11\.

### I\.5Additional Results

Figure 16:\(Text\-Maze\)Inference\-time scaling on Text\-Maze for four initial policies, labeled by their shortest\-path success before reinforcement learning\. The rows report Pass@kk, the probability that any ofkkrollouts reaches the goal along a shortest path, and Best\-of\-kkreward is the expected maximum reward attained by the policy amongkkrollouts\.##### Effect of the inference rollout budget

We evaluate how the learned rollout distributions respond to additional inference rollouts\.[Figure16](https://arxiv.org/html/2609.02987#A9.F16)reports Pass@kkand Best\-of\-kkreward askkincreases\. At smallkk, policies trained with the three methods can appear similar\. Askkincreases, both metrics improve more rapidly for TailRL than for GRPO or RLOO\. Despite GRPO and RLOO improving their Pass@11after RL post\-training on policy initializations with good coverage, TailRL still outperforms them at test\-time scaling\. Evaluation atk=1k=1therefore understates the differences among the learned rollout distributions\. The increasing separation is consistent with the Best\-of\-kkdecomposition in[Section3](https://arxiv.org/html/2609.02987#S3): TailRL assigns greater probability to high\-reward rollouts, making them more likely to be discovered askkgrows\.

##### Effect of the training rollout budget

To study the effect of the training rollout budget, we varyNNwhile holding fixed an initial policy with a shortest\-path success rate of approximately0\.02%0\.02\\%\([Fig\.17](https://arxiv.org/html/2609.02987#A9.F17)\)\. For TailRL, both Pass@kkand Best\-of\-kkreward increase sharply betweenN=4N=4andN=16N=16\. AtN=4N=4, TailRL remains near the same floor as the expected reward baselines\. AtN=16N=16, rare high\-reward rollouts appear in the training group frequently enough for TailRL to learn, while further increases inNNyield smaller gains\. GRPO and RLOO do not obtain a comparable benefit from increasingNN\. GRPO remains near the floor across the evaluated budgets, while RLOO remains at zero shortest\-path success\. Increasingkkcannot compensate for a policy that failed to learn during training\.

Figure 17:\(Text\-Maze\)Training rollout budget sweep from an initial shortest\-path success rate of0\.024%0\.024\\%, with one column perNN\. Rows show Pass@kkand Best\-of\-kkreward against the inference budget\. TailRL converts additional training rollouts into learning betweenN=4N=4andN=16N=16, while the expected reward baselines gain little\.
##### Training dynamics

[Figure18](https://arxiv.org/html/2609.02987#A9.F18)reports policy entropy and mean generated path length during training from four of the seven pretraining checkpoints atN=16N=16\. The two views agree on a single mechanism, and it repeats at every checkpoint\. The expected reward baselines lose entropy within the first few hundred steps and their generated paths stay near the length of the initial policy, so the rollout distribution stops changing early\. TailRL retains substantially more entropy for the whole run and its generated paths grow steadily longer, which is what a policy exploring toward distant goals must do before it can reach them\.

Figure 18:\(Text\-Maze\)Training dynamics atN=16N=16from four pretraining checkpoints, one per column\. Top row: policy entropy by training step, log scale\. Bottom row: mean generated path length by training step\. All curves are exponential\-moving\-average smoothed\.

## Appendix JGUI Grounding

### J\.1Model and Data

We fine\-tune the 3B and 7B versions of Qwen2\.5\-VL\([Bai et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib68)\)on GTA1\([Yang et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib69)\), which contains70,52870\{,\}528screenshot\-instruction pairs\. We process each image at its native resolution using a shared range of3,1363\{,\}136to12,845,05612\{,\}845\{,\}056pixels and a maximum of16,38416\{,\}384image tokens\. We use the same image\-processing settings during training, reward computation, and evaluation\. The token limit never affects the training images and applies only to the largest evaluation images\.

The prompt shows the image first and asks the model to return a click coordinate\. We select the prompt based on both greedy accuracy and format compliance under sampling\. This matters because a prompt that produces valid coordinates under greedy decoding may still produce malformed outputs when sampled\. Both model sizes use the same prompt\.

### J\.2Reward

We use the dense point reward from SE\-GUI\([Yuan et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib70)\)\. Lety^\\hat\{y\}be the predicted click andyythe center of the target box, both in per\-axis image\-normalized coordinates\. We defined=∥y^−y∥d=\\lVert\\hat\{y\}\-y\\rVertand letdmaxd\_\{\\max\}be the largest distance fromyyto an image corner\. The total reward is

r⁡\(x,z\)=𝟙\{y^∈box\}⏟inside\+\(1−\(d/dmax\)2\)𝟙\{d≤1\}⏟proximity\+12​1\{y^​parses\}⏟format∈\[0,2\.5\]\.r\(x,z\)=\\underbrace\{\\mathbbm\{1\}\_\{\\\{\\hat\{y\}\\in\\mathrm\{box\}\\\}\}\}\_\{\\text\{inside\}\}\+\\underbrace\{\\left\(1\-\(d/d\_\{\\max\}\)^\{2\}\\right\)\\mathbbm\{1\}\_\{\\\{d\\leq 1\\\}\}\}\_\{\\text\{proximity\}\}\+\\underbrace\{\\tfrac\{1\}\{2\}\\,\\mathbbm\{1\}\_\{\\\{\\hat\{y\}\\text\{ parses\}\\\}\}\}\_\{\\text\{format\}\}\\;\\in\\;\[0,2\.5\]\.\(135\)The first term rewards clicks inside the target box\. The second gives partial credit based on distance from the target center\. Its scale is set bydmaxd\_\{\\max\}, so it does not require a tunable distance parameter\. The final term adds0\.50\.5when the predicted coordinate can be parsed\.

The SE\-GUI paper defines the proximity term as\(1−d/dmax\)2\(1\-d/d\_\{\\max\}\)^\{2\}, while its released code uses1−\(d/dmax\)21\-\(d/d\_\{\\max\}\)^\{2\}together with the indicator𝟙\{d≤1\}\\mathbbm\{1\}\_\{\\\{d\\leq 1\\\}\}\. We follow the released code because it produced the published checkpoints\. We verified our implementation against the reference code on500500randomized examples\. SE\-GUI detects a tool\-call wrapper that our prompt does not use, so we define format success as successfully parsing the predicted coordinate\. We apply this rule identically to all methods\.

### J\.3Training Protocol

Training is fully on\-policy, with one optimizer update per batch\. Each batch contains88prompts and88rollouts per prompt, sampled at temperature11\. We use no KL regularization\. The learning rate starts at10−610^\{\-6\}and decays linearly to zero over three passes through the dataset, totaling26,44826\{,\}448steps\. All methods use the same fixed training horizon\.

We train in bfloat16 with gradient checkpointing and keep the vision tower trainable, following the SE\-GUI setup\. The maximum gradient norm is100100, which clips approximately3%3\\%of updates and prevents only large gradient spikes\. A threshold of11would clip nearly every update and obscure differences in update scale across methods\. The 3B and 7B models use the same configuration\. We run one seed per method\.[Table5](https://arxiv.org/html/2609.02987#A10.T5)summarizes the full configuration\.

Table 5:Training hyperparameters for GUI grounding\.
### J\.4Prompt Template

Both scales use the same prompt, selected by measurement at 3B and then frozen\. It is image\-first and asks for a bare pixel coordinate, with no tool\-call wrapper and no chain of thought, so that a rollout parses under sampling as reliably as under greedy decoding:

System:You are an expert UI element locator\. Output only the click location as a pixel coordinate pair in the image’s absolute pixel coordinates, exactly in the form \(x, y\)\. For elements with area, return the center point\. Output nothing else\. User:<image\> Grounding instruction is: \{instruction\}\. Where should you click to do this? Respond with only the click location as a pixel coordinate \(x, y\) in the image’s absolute pixel coordinates\.

### J\.5On\-Policy Implementation

Training is strictly on\-policy: one optimizer update per batch of rollouts, no importance\-sampling correction, no clipping, and no KL penalty to a reference policy\. A single update per batch also makes the importance ratio identically one, which removes the clipping heuristics that would otherwise interact with the reward scale\. Recent on\-policy work on GUI grounding adopts the same setting\([Liu et al\., 2026](https://arxiv.org/html/2609.02987#bib.bib79)\)\.

### J\.6Evaluation Protocol

We evaluate zero\-shot performance on all1,5811\{,\}581items in ScreenSpot\-Pro\([Li et al\., 2025](https://arxiv.org/html/2609.02987#bib.bib71)\), a benchmark of professional high\-resolution software interfaces\. We use two evaluation protocols\. First, we run one greedy prediction per item at the end of training using temperature00and report greedy accuracy\. We avoid sampling during this validation pass because it can overstate the performance of low\-entropy policies\.

Our primary evaluation uses the saved checkpoints\. We draw4,0964\{,\}096samples per item using temperature0\.60\.6, nucleus sampling with nucleus probability=0\.95=0\.95, and no top\-kkfiltering\. These settings are fixed across all methods and follow the Pass@kkevaluation protocol\([Chen et al\., 2021](https://arxiv.org/html/2609.02987#bib.bib51)\)\. The lower sampling temperature concentrates the rollout distribution and reduces the observed differences between methods by roughly half compared with temperature11\. We report the unbiased Pass@kkestimate and its continuous extension, Best\-of\-kk, using the reward on\[0,2\.5\]\[0,2\.5\]\. We compute95%95\\%confidence intervals with a1,0001\{,\}000\-sample percentile bootstrap over evaluation items\. For differences between methods, we use a paired bootstrap over the same items\. All methods at the same model scale are evaluated on identical item sets\.

Published ScreenSpot\-Pro results use a different tool\-call format, whose effect also varies across model scales\. Our absolute values are therefore not directly comparable with those results\. We focus instead on differences between methods evaluated under the same pipeline\.

### J\.7Additional Results

##### Category\-wise inference scaling

[Figures19](https://arxiv.org/html/2609.02987#A10.F19)and[20](https://arxiv.org/html/2609.02987#A10.F20)break the final\-checkpoint inference\-scaling ladders down by ScreenSpot\-Pro task category, at both model scales\. Atk=1k=1the three post\-trained methods are close in every category, within a few points either way\. The separation appears as the budget grows: atk=128k=128, TailRL leads the better baseline in eleven of the twelve scale\-category cells, by up to12\.012\.0points on Dev at 3B and11\.511\.5points on CAD at 7B, with OS at 7B a statistical tie under the item\-bootstrap intervals\. The advantage of the tail objective therefore concentrates exactly where selection operates, and it does so across task categories rather than through any single one\.

Figure 19:\(GUI\-grounding\)Category\-wise inference scaling on ScreenSpot\-Pro dataset, Qwen2\.5\-VL\-3B at the final checkpoint\. Pass@kkagainst the inference rollout budget within each task category,512512samples per item\.Figure 20:\(GUI\-grounding\)Category\-wise inference scaling on ScreenSpot\-Pro, Qwen2\.5\-VL\-7B at the final checkpoint\. Pass@kkagainst the inference rollout budget within each task category,512512samples per item\.
##### Pass@kkand Best\-of\-kkthrough training

[Figures21](https://arxiv.org/html/2609.02987#A10.F21)and[22](https://arxiv.org/html/2609.02987#A10.F22)evaluate every checkpoint rather than only the final one, at four inference rollout budgets\. Two facts hold throughout training rather than only at its end\. Atk=1k=1the three post\-trained methods stay close together, while at every larger budget TailRL separates from both baselines within the first epoch and holds that separation\. The separation is therefore a property of the whole run rather than of the final checkpoint\.

Figure 21:\(GUI\-grounding\)ScreenSpot\-Pro Pass@kkagainst training epoch at four inference rollout budgets\. Top row: Qwen2\.5\-VL\-3B\. Bottom row: Qwen2\.5\-VL\-7B\. Every checkpoint is evaluated with512512samples per item\.Figure 22:\(GUI\-grounding\)Best\-of\-kkreward against training steps\. Top row: Qwen2\.5\-VL\-3B\. Bottom row: Qwen2\.5\-VL\-7B\. Every checkpoint is evaluated with512512samples per item\.
##### Reward components

[Figure23](https://arxiv.org/html/2609.02987#A10.F23)separates the training reward into its parts\. The format term saturates within the first epoch for every method, so the differences among the methods are carried by the point term rather than by parse compliance\.

Figure 23:\(GUI\-grounding\)Training reward by epoch, decomposed\. Left: total reward on\[0,2\.5\]\[0,2\.5\]\. Center: the point term\. Right: the format term\. Top row: Qwen2\.5\-VL\-3B\. Bottom row: Qwen2\.5\-VL\-7B\. All curves are exponential\-moving\-average smoothed\.
##### Training dynamics

[Figure24](https://arxiv.org/html/2609.02987#A10.F24)reports gradient norm and policy entropy at both scales\. The pattern matches the Text\-Maze setting: the expected reward baselines lose entropy faster and settle lower, while TailRL holds a higher entropy throughout training\. This is the training\-time counterpart of the evaluation result, since a policy that keeps more probability mass away from its own mode is the one whose Pass@kkcontinues to rise with the inference rollout budget\.

Figure 24:\(GUI\-grounding\)We compare the training dynamics of different policy gradient algorithms\. Top row: Qwen2\.5\-VL\-3B gradient norm and policy entropy by training epoch\. Bottom row: the same quantities for Qwen2\.5\-VL\-7B\. All curves are exponential\-moving\-average smoothed\.

## Appendix KCode runtime optimization

We use the PIE corpus of competitive\-programming solutions\. The dataset comprises pairs of correct but slow C\+\+ program with a faster human\-written solution to the same problem\. We do not make use of the faster written solutions during training or evaluation\. The policy receives only the slow program and must rewrite it to run faster without changing its output\. It returns the rewritten program as a single fenced C\+\+ block\.

### K\.1Reward

We evaluate each rollout for correctness before measuring its speed\. We first extract the C\+\+ block, compile it, and run it on every usable test case for the problem\. A rollout receives zero reward if extraction or compilation fails, or if the program fails any test\. Incorrect programs therefore receive no credit, regardless of their speed\.

A correct rollout receives its speedup over the original program:

r\(x,z\)=csrcc⁡\(x,z\)𝟙\{every test passes\}r\(x,z\)=\\frac\{c\_\{\\mathrm\{src\}\}\}\{c\(x,z\)\}\\mathbbm\{1\}\_\{\\\{\\text\{every test passes\\lx@text@rbrace\}\}\(136\)wherec⁡\(x,z\)c\(x,z\)is the cost of the rewritten program andcsrcc\_\{\\mathrm\{src\}\}is the cost of the original program\.

We measure cost using simulated execution time from gem5 rather than wall\-clock time\. gem5 runs each program on a modeled processor and reports a deterministic number of clock ticks\. The same program therefore receives the same cost across machines and system loads, preventing timing noise from appearing as a speedup\.

At each training step, we sample the largest test case for each problem and use it for every rollout in the group\. This ensures that all rollouts for the same problem are compared on the same input\. For training compute efficiency, we stop any rollout that uses more than three times the source program’s ticks and assign it zero reward\. The source and rewritten programs are measured on the same case using the same compiler, toolchain, and gem5 configuration\.

This reward has two important properties\. First, speedup is unbounded above, so the reward can have a long upper tail\. Second, copying the input program always passes the correctness tests and receives a reward of exactly11\. Copying is therefore a safe but suboptimal shortcut, as discussed in[Section6\.4](https://arxiv.org/html/2609.02987#S6.SS4)\.

### K\.2Dataset Construction

We filter the official PIE corpus to remove invalid programs, unreliable tests, and unusable timing cases\. We compile every program and run it on its problem’s test suite\. We exclude programs that fail to compile or fail more thanmax⁡\(5,5%\)\\max\(5,5\\%\)of the test cases\.

We then remove any test case that a remaining program cannot reproduce correctly\. This step removes inconsistent or unreliable tests\. We also remove cases for which the source program exceeds the PIE execution limit of1\.4×10101\.4\\times 10^\{10\}gem5 ticks\.

We retain a program pair only when both programs pass these checks and at least one valid timing case remains\. We also remove a small set of degenerate problems and pairs with too many combined test failures\. Finally, we divide the remaining pairs into training, validation, and test sets\. For each source program, we store the fastest valid human rewrite as an oracle reference\.

### K\.3Model and Training

We train Qwen3\-1\.7B\. Training is fully on\-policy and uses no KL regularization\. Each batch contains6464programs and1616rollouts per program\. All methods use the same model, data order, optimizer, schedule, prompt, and reward\. They differ only in their advantage estimator\.[Table6](https://arxiv.org/html/2609.02987#A11.T6)provides the full training configuration\.

Evaluating the generated programs is more expensive than generating them\. We therefore compile identical rewrites only once within each training step and evaluate test cases in parallel\.

Table 6:Training hyperparameters for Code runtime optimization\.
### K\.4Prompt Template

Every arm uses the same minimal prompt, a single user turn with no system message and no reasoning scaffold; the slow program is inlined verbatim:

User:You are given a working C\+\+ program\. Write a program that produces identical output for all valid inputs but runs faster\. \#\#\# Slow Version: \`\`\`cpp \{slow program\} \`\`\` Give your final program under a line that reads exactly \`\#\#\# Optimized Version:\`, as a single \`\`\`cpp code block\.

### K\.5Reported Quantities

The quantities in[Fig\.12](https://arxiv.org/html/2609.02987#S6.F12)are measured on the training rollouts themselves, which is the right instrument for a claim about what a policy learns to emit\. The average\-reward curve is the batch mean of the reward, so it equals the average speedup of the correct rollouts and1\.01\.0exactly when every correct rollout merely reproduces its input\. The correctness curve is the fraction of the1616rollouts per program that compile and pass every test\. The entropy curve is the policy entropy reported by the trainer\.[Figure11](https://arxiv.org/html/2609.02987#S6.F11)is measured separately, before any reinforcement learning, from40964096rollouts on each of the878878held\-out test problems under the same reward\. Curves are means over three runs per objective\. We report training\-time behavior in the main text because the effect under study is the collapse of the rollout distribution onto a single degenerate answer, which is visible directly in what the policy emits; the held\-out evaluation of Best\-of\-10241024reward on all878878test problems is reported in[Fig\.12](https://arxiv.org/html/2609.02987#S6.F12)and, problem by problem Best\-of\-kkreward in[Fig\.26](https://arxiv.org/html/2609.02987#A11.F26)\.

### K\.6Additional Results

##### Training entropy and per\-problem held\-out evaluation

[Figure26](https://arxiv.org/html/2609.02987#A11.F26)reports the policy entropy referenced in the main\-text Results paragraph, and[Fig\.26](https://arxiv.org/html/2609.02987#A11.F26)resolves the held\-out evaluation of[Fig\.12](https://arxiv.org/html/2609.02987#S6.F12)problem by problem\.

Figure 25:\(Code runtime optimization\)Policy entropy during training, EMA over three runs per objective\. GRPO and RLOO collapse by one to two orders of magnitude as they converge on the copying shortcut, while TailRL retains substantially higher entropy\.Figure 26:\(Code runtime optimization\)Each dot is one of the878878test\-set problems of PIE dataset at its Best\-of\-kkspeedup,kkfrom11\(light\) to10241024\(dark\), TailRL in red and the pretrained model in gray\. Selection compounds TailRL’s advantage problem by problem\.

##### Best\-of\-kkdensities

[Figure27](https://arxiv.org/html/2609.02987#A11.F27)shows the same held\-out evaluation as one density per inference budget\. TailRL’s mass is already centered near5×5\\timesatk=1k=1and sharpens rightward askkgrows, while the GRPO and RLOO needle stays pinned at the copy line at every budget\. The pretrained model’s hump climbs from below1×1\\timestoward2×2\\times, but never reaches the region where TailRL’s mass lives\.

Figure 27:\(Code runtime optimization\)Kernel densities of the per\-problem Best\-of\-kkspeedup on the PIE dataset held\-out problems, one panel per inference budget, log axes\. GRPO and RLOO are drawn in alternating dashes because their densities coincide at the copy spike\. All post\-trained policies are at step300300\(1 epoch\), one training epoch, and the pretrained model is evaluated before any post\-training\.

### K\.7Sample Generations

We show one held\-out problem, p02318, computing edit distance with a dynamic program: the input prompt exactly as the models receive it, followed by the complete best rollout from each policy, reasoning trace and code, selected by verified speedup among the6464rollouts dumped per problem\. All post\-trained policies are at step300300, one training epoch\. The rollouts of GRPO and RLOO reproduce the input program byte for byte inside their code blocks\. TailRL’s best is a genuine rewrite reaching a verified27\.0×27\.0\\times, and the pretrained model’s best reaches5\.05×5\.05\\times\. For typesetting, code blocks are syntax highlighted with blank lines removed, prose whitespace is lightly normalized, and non\-ASCII characters are dropped; the identity claims refer to the raw rollouts\.

Input prompt, held\-out problem p02318[⬇](data:text/plain;base64,PHxpbV9zdGFydHw+dXNlcgpZb3UgYXJlIGEgQysrIG9wdGltaXphdGlvbiBleHBlcnQuIEJlbG93IGlzIGEgd29ya2luZyBDKysgcHJvZ3JhbS4gUmV3cml0ZSBpdCB0byBydW4gZmFzdGVyIHdoaWxlIHByb2R1Y2luZyBpZGVudGljYWwgb3V0cHV0IGZvciBhbGwgdmFsaWQgaW5wdXRzLiBLZWVwIHRoZSBzYW1lIGlucHV0L291dHB1dCBmb3JtYXQgYW5kIG9ubHkgY2hhbmdlIHRoZSBhbGdvcml0aG0vaW1wbGVtZW50YXRpb24uCgojIyMgU2xvdyBWZXJzaW9uOg==)<\|im\_start\|\>userYouareaC\+\+optimizationexpert\.BelowisaworkingC\+\+program\.Rewriteittorunfasterwhileproducingidenticaloutputforallvalidinputs\.Keepthesameinput/outputformatandonlychangethealgorithm/implementation\.\#\#\#SlowVersion:[⬇](data:text/plain;base64,I2luY2x1ZGUgPGJpdHMvc3RkYysrLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CnR5cGVkZWYgbG9uZyBsb25nIGludCBsbDsKY29uc3QgaW50IElORiA9IDEwMDAwMDAwMDA7CiNkZWZpbmUgUkVQKGkscyxuKSBmb3IoaW50IGk9KGludCkocyk7aTwoaW50KShuKTtpKyspCiNkZWZpbmUgcmVwKGksbikgUkVQKGksIDAsIG4pCmludCBkcFsxMDAxXVsxMDAxXTsKaW50IG1haW4oKXsKICAgIGNpbi50aWUoMCk7CiAgICBpb3M6OnN5bmNfd2l0aF9zdGRpbyhmYWxzZSk7CiAgICBzdHJpbmcgYSwgYjsKICAgIGNpbiA+PiBhID4+IGI7CiAgICBpbnQgbiA9IGEuc2l6ZSgpLCBtID0gYi5zaXplKCk7CiAgICBtZW1zZXQoZHAsIDAsIHNpemVvZihkcCkpOwogICAgcmVwKGksIG0gKyAxKSBkcFswXVtpXSA9IGk7CiAgICByZXAoaSwgbiArIDEpIGRwW2ldWzBdID0gaTsKICAgIFJFUChpLCAxLCBuICsgMSkgUkVQKGosIDEsIG0gKyAxKXsKICAgICAgICBkcFtpXVtqXSA9IGRwW2kgLSAxXVtqIC0gMV07CiAgICAgICAgaWYoYVtpIC0gMV0gIT0gYltqLSAxXSkgZHBbaV1bal0gKz0gMTsKICAgICAgICBpbnQgdGVtcCA9IG1pbihkcFtpIC0gMV1bal0gKyAxLCBkcFtpXVtqIC0gMV0gKyAxKTsKICAgICAgICBkcFtpXVtqXSA9IG1pbih0ZW1wLCBkcFtpXVtqXSk7CiAgICB9CiAgICBjb3V0IDw8IGRwW25dW21dIDw8IGVuZGw7CiAgICByZXR1cm4gMDsKfQ==)usingnamespacestd;typedeflonglongintll;constintINF=1000000000;intdp\[1001\]\[1001\];intmain\(\)\{cin\.tie\(0\);ios::sync\_with\_stdio\(false\);stringa,b;cin\>\>a\>\>b;intn=a\.size\(\),m=b\.size\(\);memset\(dp,0,sizeof\(dp\)\);rep\(i,m\+1\)dp\[0\]\[i\]=i;rep\(i,n\+1\)dp\[i\]\[0\]=i;REP\(i,1,n\+1\)REP\(j,1,m\+1\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)dp\[i\]\[j\]\+=1;inttemp=min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\);dp\[i\]\[j\]=min\(temp,dp\[i\]\[j\]\);\}cout<<dp\[n\]\[m\]<<endl;return0;\}[⬇](data:text/plain;base64,IyMjIE9wdGltaXplZCBWZXJzaW9uOgo8fGltX2VuZHw+Cjx8aW1fc3RhcnR8PmFzc2lzdGFudAo8dGhpbms+Cgo8L3RoaW5rPg==)\#\#\#OptimizedVersion:<\|im\_end\|\><\|im\_start\|\>assistant<think\></think\>

TailRL, step300300, best of6464rollouts, verified speedup27\.0×27\.0\\times[⬇](data:text/plain;base64,U3VyZSEgTGV0J3MgdGFrZSB5b3VyICoqc2xvdyB2ZXJzaW9uKiogb2YgdGhlIEMrKyBjb2RlIGFuZCAqKnJld3JpdGUgaXQgdG8gcnVuIGZhc3RlcioqIHdoaWxlICoqcHJvZHVjaW5nIGlkZW50aWNhbCBvdXRwdXQqKiBmb3IgYWxsIHZhbGlkIGlucHV0cywgKiprZWVwaW5nIHRoZSBzYW1lIGlucHV0L291dHB1dCBmb3JtYXQqKiBhbmQgKipvbmx5IGNoYW5naW5nIHRoZSBhbGdvcml0aG0vaW1wbGVtZW50YXRpb24qKi4KCi0tLQoKIyMjICAqKlNsb3cgVmVyc2lvbioq)Sure\!Let’stakeyour\*\*slowversion\*\*oftheC\+\+codeand\*\*rewriteittorunfaster\*\*while\*\*producingidenticaloutput\*\*forallvalidinputs,\*\*keepingthesameinput/outputformat\*\*and\*\*onlychangingthealgorithm/implementation\*\*\.\-\-\-\#\#\#\*\*SlowVersion\*\*[⬇](data:text/plain;base64,I2luY2x1ZGUgPGJpdHMvc3RkYysrLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CnR5cGVkZWYgbG9uZyBsb25nIGludCBsbDsKY29uc3QgaW50IElORiA9IDEwMDAwMDAwMDA7CiNkZWZpbmUgUkVQKGkscyxuKSBmb3IoaW50IGk9KGludCkocyk7aTwoaW50KShuKTtpKyspCiNkZWZpbmUgcmVwKGksbikgUkVQKGksIDAsIG4pCmludCBkcFsxMDAxXVsxMDAxXTsKaW50IG1haW4oKXsKICAgIGNpbi50aWUoMCk7CiAgICBpb3M6OnN5bmNfd2l0aF9zdGRpbyhmYWxzZSk7CiAgICBzdHJpbmcgYSwgYjsKICAgIGNpbiA+PiBhID4+IGI7CiAgICBpbnQgbiA9IGEuc2l6ZSgpLCBtID0gYi5zaXplKCk7CiAgICBtZW1zZXQoZHAsIDAsIHNpemVvZihkcCkpOwogICAgcmVwKGksIG0gKyAxKSBkcFswXVtpXSA9IGk7CiAgICByZXAoaSwgbiArIDEpIGRwW2ldWzBdID0gaTsKICAgIFJFUChpLCAxLCBuICsgMSkgUkVQKGosIDEsIG0gKyAxKXsKICAgICAgICBkcFtpXVtqXSA9IGRwW2kgLSAxXVtqIC0gMV07CiAgICAgICAgaWYoYVtpIC0gMV0gIT0gYltqLSAxXSkgZHBbaV1bal0gKz0gMTsKICAgICAgICBpbnQgdGVtcCA9IG1pbihkcFtpIC0gMV1bal0gKyAxLCBkcFtpXVtqIC0gMV0gKyAxKTsKICAgICAgICBkcFtpXVtqXSA9IG1pbih0ZW1wLCBkcFtpXVtqXSk7CiAgICB9CiAgICBjb3V0IDw8IGRwW25dW21dIDw8IGVuZGw7CiAgICByZXR1cm4gMDsKfQ==)usingnamespacestd;typedeflonglongintll;constintINF=1000000000;intdp\[1001\]\[1001\];intmain\(\)\{cin\.tie\(0\);ios::sync\_with\_stdio\(false\);stringa,b;cin\>\>a\>\>b;intn=a\.size\(\),m=b\.size\(\);memset\(dp,0,sizeof\(dp\)\);rep\(i,m\+1\)dp\[0\]\[i\]=i;rep\(i,n\+1\)dp\[i\]\[0\]=i;REP\(i,1,n\+1\)REP\(j,1,m\+1\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)dp\[i\]\[j\]\+=1;inttemp=min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\);dp\[i\]\[j\]=min\(temp,dp\[i\]\[j\]\);\}cout<<dp\[n\]\[m\]<<endl;return0;\}[⬇](data:text/plain;base64,LS0tCgojIyMgICoqT3B0aW1pemVkIFZlcnNpb24qKiAoUmV3cml0dGVuIGZvciBzcGVlZCBhbmQgY29ycmVjdG5lc3Mp)\-\-\-\#\#\#\*\*OptimizedVersion\*\*\(Rewrittenforspeedandcorrectness\)[⬇](data:text/plain;base64,I2luY2x1ZGUgPGNzdGRpbz4KI2luY2x1ZGUgPGFsZ29yaXRobT4KI2luY2x1ZGUgPGNzdHJpbmc+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CmNvbnN0IGludCBNQVggPSAxMDAxOwppbnQgZHBbTUFYXVtNQVhdOwppbnQgbWFpbigpIHsKICAgIGNoYXIgYVsxMDAxXSwgYlsxMDAxXTsKICAgIHNjYW5mKCIgJXMlcyIsIGEsIGIpOwogICAgaW50IG4gPSBzdHJsZW4oYSksIG0gPSBzdHJsZW4oYik7CiAgICAvLyBJbml0aWFsaXplIERQIHRhYmxlCiAgICBmb3IgKGludCBpID0gMDsgaSA8PSBtOyBpKyspIGRwWzBdW2ldID0gaTsKICAgIGZvciAoaW50IGkgPSAwOyBpIDw9IG47IGkrKykgZHBbaV1bMF0gPSBpOwogICAgLy8gRmlsbCBEUCB0YWJsZQogICAgZm9yIChpbnQgaSA9IDE7IGkgPD0gbjsgaSsrKSB7CiAgICAgICAgZm9yIChpbnQgaiA9IDE7IGogPD0gbTsgaisrKSB7CiAgICAgICAgICAgIGRwW2ldW2pdID0gZHBbaS0xXVtqLTFdOwogICAgICAgICAgICBpZiAoYVtpLTFdICE9IGJbai0xXSkgZHBbaV1bal0rKzsKICAgICAgICAgICAgaW50IHRlbXAgPSBzdGQ6Om1pbihkcFtpLTFdW2pdICsgMSwgZHBbaV1bai0xXSArIDEpOwogICAgICAgICAgICBkcFtpXVtqXSA9IHN0ZDo6bWluKGRwW2ldW2pdLCB0ZW1wKTsKICAgICAgICB9CiAgICB9CiAgICAvLyBPdXRwdXQgcmVzdWx0CiAgICBwcmludGYoIiVkXG4iLCBkcFtuXVttXSk7CiAgICByZXR1cm4gMDsKfQ==)usingnamespacestd;constintMAX=1001;intdp\[MAX\]\[MAX\];intmain\(\)\{chara\[1001\],b\[1001\];scanf\("%s%s",a,b\);intn=strlen\(a\),m=strlen\(b\);for\(inti=0;i<=m;i\+\+\)dp\[0\]\[i\]=i;for\(inti=0;i<=n;i\+\+\)dp\[i\]\[0\]=i;for\(inti=1;i<=n;i\+\+\)\{for\(intj=1;j<=m;j\+\+\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)dp\[i\]\[j\]\+\+;inttemp=std::min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\);dp\[i\]\[j\]=std::min\(dp\[i\]\[j\],temp\);\}\}printf\("%d\\n",dp\[n\]\[m\]\);return0;\}[⬇](data:text/plain;base64,LS0tCgojIyMgICoqS2V5IENoYW5nZXMqKgoKLSAqKlJlbW92ZWQgYCNpbmNsdWRlIDxiaXRzL3N0ZGMrKy5oPmAqKjogVGhpcyBpcyBub3Qgc3RhbmRhcmQgQysrIGFuZCBjYW4gY2F1c2UgaXNzdWVzLiBXZSB1c2VkIGBjc3RkaW9gIGFuZCBgY3N0cmluZ2AgZm9yIGlucHV0L291dHB1dCBhbmQgc3RyaW5nIGhhbmRsaW5nLgotICoqUmVtb3ZlZCBgc3RkOjptaW5gIGFuZCBgc3RkOjptYXhgKio6IFJlcGxhY2VkIHdpdGggaW5saW5lIGBzdGQ6Om1pbmAgKHdoaWNoIGlzIHN0aWxsIHZhbGlkIGluIEMrKzExKykuCi0gKipSZW1vdmVkIGBSRVBgIGFuZCBgcmVwYCBtYWNyb3MqKjogUmVwbGFjZWQgd2l0aCBkaXJlY3QgbG9vcHMuCi0gKipVc2VkIGBzdHJsZW5gIGluc3RlYWQgb2YgYHNpemUoKWAqKjogTW9yZSBzdGFuZGFyZCBhbmQgc2FmZS4KLSAqKlJlbW92ZWQgYG1lbXNldGAgYW5kIGBtZW1zZXQoZHAsIDAsIC4uLilgKio6IFJlcGxhY2VkIHdpdGggZGlyZWN0IGluaXRpYWxpemF0aW9uLgotICoqVXNlZCBgcHJpbnRmYCBpbnN0ZWFkIG9mIGBjb3V0YCoqOiBGb3IgZmFzdGVyIEkvTy4KLSAqKlVzZWQgYHN0ZDo6bWluYCBhbmQgYHN0ZDo6bWF4YCoqIHdoZXJlIGFwcHJvcHJpYXRlIChidXQgaW4gdGhpcyBjYXNlLCB3ZSB1c2VkIGBzdGQ6Om1pbmAgZGlyZWN0bHkpLgotICoqS2VwdCB0aGUgc2FtZSBpbnB1dCBmb3JtYXQqKiAoc3BhY2Utc2VwYXJhdGVkIHN0cmluZ3MpLgotICoqS2VwdCB0aGUgc2FtZSBvdXRwdXQgZm9ybWF0KiogKHNpbmdsZSBsaW5lLCBpbnRlZ2VyLCB0cmFpbGluZyBuZXdsaW5lKS4KCi0tLQoKIyMjICAqKlJlc3VsdCoqCi0gKipGYXN0ZXIqKiB0aGFuIHRoZSBvcmlnaW5hbCBkdWUgdG86CiAgLSBEaXJlY3QgQy1zdHlsZSBJL08KICAtIE5vIG1hY3JvLWJhc2VkIGxvb3BzCiAgLSBObyB1bm5lY2Vzc2FyeSBpbmNsdWRlcwogIC0gRGlyZWN0IGFycmF5IGFjY2VzcwotICoqSWRlbnRpY2FsIG91dHB1dCoqIGZvciBhbGwgdmFsaWQgaW5wdXRzCi0gKipObyBjaGFuZ2UgdG8gaW5wdXQvb3V0cHV0IGZvcm1hdCoqCgpMZXQgbWUga25vdyBpZiB5b3UnZCBsaWtlIGZ1cnRoZXIgb3B0aW1pemF0aW9ucyAoZS5nLiwgdXNpbmcgYGlubGluZWAgb3IgYF9fYXR0cmlidXRlX19gIGZvciBjb21waWxlciBoaW50cyku)\-\-\-\#\#\#\*\*KeyChanges\*\*\-\*\*Removed‘\#include<bits/stdc\+\+\.h\>‘\*\*:ThisisnotstandardC\+\+andcancauseissues\.Weused‘cstdio‘and‘cstring‘forinput/outputandstringhandling\.\-\*\*Removed‘std::min‘and‘std::max‘\*\*:Replacedwithinline‘std::min‘\(whichisstillvalidinC\+\+11\+\)\.\-\*\*Removed‘REP‘and‘rep‘macros\*\*:Replacedwithdirectloops\.\-\*\*Used‘strlen‘insteadof‘size\(\)‘\*\*:Morestandardandsafe\.\-\*\*Removed‘memset‘and‘memset\(dp,0,\.\.\.\)‘\*\*:Replacedwithdirectinitialization\.\-\*\*Used‘printf‘insteadof‘cout‘\*\*:ForfasterI/O\.\-\*\*Used‘std::min‘and‘std::max‘\*\*whereappropriate\(butinthiscase,weused‘std::min‘directly\)\.\-\*\*Keptthesameinputformat\*\*\(space\-separatedstrings\)\.\-\*\*Keptthesameoutputformat\*\*\(singleline,integer,trailingnewline\)\.\-\-\-\#\#\#\*\*Result\*\*\-\*\*Faster\*\*thantheoriginaldueto:\-DirectC\-styleI/O\-Nomacro\-basedloops\-Nounnecessaryincludes\-Directarrayaccess\-\*\*Identicaloutput\*\*forallvalidinputs\-\*\*Nochangetoinput/outputformat\*\*Letmeknowifyou’dlikefurtheroptimizations\(e\.g\.,using‘inline‘or‘\_\_attribute\_\_‘forcompilerhints\)\.

GRPO, step300300, best of6464rollouts, speedup1\.00×1\.00\\times[⬇](data:text/plain;base64,I2luY2x1ZGUgPGJpdHMvc3RkYysrLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CnR5cGVkZWYgbG9uZyBsb25nIGludCBsbDsKY29uc3QgaW50IElORiA9IDEwMDAwMDAwMDA7CiNkZWZpbmUgUkVQKGkscyxuKSBmb3IoaW50IGk9KGludCkocyk7aTwoaW50KShuKTtpKyspCiNkZWZpbmUgcmVwKGksbikgUkVQKGksIDAsIG4pCmludCBkcFsxMDAxXVsxMDAxXTsKaW50IG1haW4oKXsKICAgIGNpbi50aWUoMCk7CiAgICBpb3M6OnN5bmNfd2l0aF9zdGRpbyhmYWxzZSk7CiAgICBzdHJpbmcgYSwgYjsKICAgIGNpbiA+PiBhID4+IGI7CiAgICBpbnQgbiA9IGEuc2l6ZSgpLCBtID0gYi5zaXplKCk7CiAgICBtZW1zZXQoZHAsIDAsIHNpemVvZihkcCkpOwogICAgcmVwKGksIG0gKyAxKSBkcFswXVtpXSA9IGk7CiAgICByZXAoaSwgbiArIDEpIGRwW2ldWzBdID0gaTsKICAgIFJFUChpLCAxLCBuICsgMSkgUkVQKGosIDEsIG0gKyAxKXsKICAgICAgICBkcFtpXVtqXSA9IGRwW2kgLSAxXVtqIC0gMV07CiAgICAgICAgaWYoYVtpIC0gMV0gIT0gYltqLSAxXSkgZHBbaV1bal0gKz0gMTsKICAgICAgICBpbnQgdGVtcCA9IG1pbihkcFtpIC0gMV1bal0gKyAxLCBkcFtpXVtqIC0gMV0gKyAxKTsKICAgICAgICBkcFtpXVtqXSA9IG1pbih0ZW1wLCBkcFtpXVtqXSk7CiAgICB9CiAgICBjb3V0IDw8IGRwW25dW21dIDw8IGVuZGw7CiAgICByZXR1cm4gMDsKfQ==)usingnamespacestd;typedeflonglongintll;constintINF=1000000000;intdp\[1001\]\[1001\];intmain\(\)\{cin\.tie\(0\);ios::sync\_with\_stdio\(false\);stringa,b;cin\>\>a\>\>b;intn=a\.size\(\),m=b\.size\(\);memset\(dp,0,sizeof\(dp\)\);rep\(i,m\+1\)dp\[0\]\[i\]=i;rep\(i,n\+1\)dp\[i\]\[0\]=i;REP\(i,1,n\+1\)REP\(j,1,m\+1\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)dp\[i\]\[j\]\+=1;inttemp=min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\);dp\[i\]\[j\]=min\(temp,dp\[i\]\[j\]\);\}cout<<dp\[n\]\[m\]<<endl;return0;\}[⬇](data:text/plain;base64,VGhpcyBpcyB0aGUgb3B0aW1pemVkIHZlcnNpb24gb2YgdGhlIGdpdmVuIEMrKyBwcm9ncmFtLiBJdCBtYWludGFpbnMgdGhlIHNhbWUgaW5wdXQvb3V0cHV0IGZvcm1hdCBhbmQgYWxnb3JpdGhtL2ltcGxlbWVudGF0aW9uLCBidXQgaXQgaXMgb3B0aW1pemVkIGZvciBwZXJmb3JtYW5jZS4gVGhlIGNvZGUgaGFzIGJlZW4gcmV3cml0dGVuIHRvIHJ1biBmYXN0ZXIgd2hpbGUgcHJvZHVjaW5nIGlkZW50aWNhbCBvdXRwdXQgZm9yIGFsbCB2YWxpZCBpbnB1dHMu)ThisistheoptimizedversionofthegivenC\+\+program\.Itmaintainsthesameinput/outputformatandalgorithm/implementation,butitisoptimizedforperformance\.Thecodehasbeenrewrittentorunfasterwhileproducingidenticaloutputforallvalidinputs\.

RLOO, step300300, best of6464rollouts, speedup1\.00×1\.00\\times[⬇](data:text/plain;base64,I2luY2x1ZGUgPGJpdHMvc3RkYysrLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CnR5cGVkZWYgbG9uZyBsb25nIGludCBsbDsKY29uc3QgaW50IElORiA9IDEwMDAwMDAwMDA7CiNkZWZpbmUgUkVQKGkscyxuKSBmb3IoaW50IGk9KGludCkocyk7aTwoaW50KShuKTtpKyspCiNkZWZpbmUgcmVwKGksbikgUkVQKGksIDAsIG4pCmludCBkcFsxMDAxXVsxMDAxXTsKaW50IG1haW4oKXsKICAgIGNpbi50aWUoMCk7CiAgICBpb3M6OnN5bmNfd2l0aF9zdGRpbyhmYWxzZSk7CiAgICBzdHJpbmcgYSwgYjsKICAgIGNpbiA+PiBhID4+IGI7CiAgICBpbnQgbiA9IGEuc2l6ZSgpLCBtID0gYi5zaXplKCk7CiAgICBtZW1zZXQoZHAsIDAsIHNpemVvZihkcCkpOwogICAgcmVwKGksIG0gKyAxKSBkcFswXVtpXSA9IGk7CiAgICByZXAoaSwgbiArIDEpIGRwW2ldWzBdID0gaTsKICAgIFJFUChpLCAxLCBuICsgMSkgUkVQKGosIDEsIG0gKyAxKXsKICAgICAgICBkcFtpXVtqXSA9IGRwW2kgLSAxXVtqIC0gMV07CiAgICAgICAgaWYoYVtpIC0gMV0gIT0gYltqLSAxXSkgZHBbaV1bal0gKz0gMTsKICAgICAgICBpbnQgdGVtcCA9IG1pbihkcFtpIC0gMV1bal0gKyAxLCBkcFtpXVtqIC0gMV0gKyAxKTsKICAgICAgICBkcFtpXVtqXSA9IG1pbih0ZW1wLCBkcFtpXVtqXSk7CiAgICB9CiAgICBjb3V0IDw8IGRwW25dW21dIDw8IGVuZGw7CiAgICByZXR1cm4gMDsKfQ==)usingnamespacestd;typedeflonglongintll;constintINF=1000000000;intdp\[1001\]\[1001\];intmain\(\)\{cin\.tie\(0\);ios::sync\_with\_stdio\(false\);stringa,b;cin\>\>a\>\>b;intn=a\.size\(\),m=b\.size\(\);memset\(dp,0,sizeof\(dp\)\);rep\(i,m\+1\)dp\[0\]\[i\]=i;rep\(i,n\+1\)dp\[i\]\[0\]=i;REP\(i,1,n\+1\)REP\(j,1,m\+1\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)dp\[i\]\[j\]\+=1;inttemp=min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\);dp\[i\]\[j\]=min\(temp,dp\[i\]\[j\]\);\}cout<<dp\[n\]\[m\]<<endl;return0;\}

Base model, before post\-training, best of6464rollouts, verified speedup5\.05×5\.05\\times[⬇](data:text/plain;base64,Q2VydGFpbmx5ISBMZXQncyBhbmFseXplIHRoZSAqKnNsb3cgdmVyc2lvbioqIG9mIHRoZSBjb2RlIGFuZCBpZGVudGlmeSB0aGUgKipwZXJmb3JtYW5jZSBib3R0bGVuZWNrcyoqIGFuZCAqKm9wdGltaXphdGlvbnMqKiB3ZSBjYW4gbWFrZSB0byBzcGVlZCBpdCB1cCB3aGlsZSBrZWVwaW5nIHRoZSBzYW1lIGxvZ2ljIGFuZCBvdXRwdXQuCgotLS0KCiMjIyAgKipTbG93IFZlcnNpb24gQW5hbHlzaXMqKgoKVGhlIGNvZGUgdXNlcyBhIDJEIERQIGFycmF5IGBkcFtpXVtqXWAgdG8gcmVwcmVzZW50IHRoZSBtaW5pbXVtIG51bWJlciBvZiBlZGl0cyAoaW5zZXJ0aW9ucywgZGVsZXRpb25zLCBzdWJzdGl0dXRpb25zKSByZXF1aXJlZCB0byBjb252ZXJ0IHRoZSBmaXJzdCBgaWAgY2hhcmFjdGVycyBvZiBgYWAgdG8gdGhlIGZpcnN0IGBqYCBjaGFyYWN0ZXJzIG9mIGBiYC4KCiMjIyMgS2V5IElzc3VlczoKCjEuICoqRG91YmxlIGBSRVBgIGxvb3BzKio6CiAgIC0gVGhlIGNvZGUgaGFzIHR3byBgUkVQYCBsb29wczogb25lIGZvciBgaWAgZnJvbSAwIHRvIGBuYCwgYW5kIG9uZSBmb3IgYGpgIGZyb20gMCB0byBgbWAuCiAgIC0gVGhpcyBpcyBub3QgdGhlIG1vc3QgZWZmaWNpZW50IHdheSB0byBpbml0aWFsaXplIHRoZSBEUCB0YWJsZS4KCjIuICoqVW5uZWNlc3NhcnkgYG1lbXNldGAqKjoKICAgLSBUaGUgYG1lbXNldChkcCwgMCwgc2l6ZW9mKGRwKSlgIGlzIGNhbGxlZCwgYnV0IHdlIGNhbiBpbml0aWFsaXplIHRoZSBEUCB0YWJsZSB1c2luZyBhIG1vcmUgZWZmaWNpZW50IG1ldGhvZC4KCjMuICoqUmVkdW5kYW50IGBSRVBgIGxvb3AqKjoKICAgLSBUaGUgYFJFUChpLCAxLCBuICsgMSlgIGFuZCBgUkVQKGosIDEsIG0gKyAxKWAgYXJlIHJlZHVuZGFudCBhbmQgY2FuIGJlIHNpbXBsaWZpZWQuCgo0LiAqKkluZWZmaWNpZW50IGBtaW5gIG9wZXJhdGlvbioqOgogICAtIFRoZSBgbWluYCBvcGVyYXRpb24gaXMgdXNlZCBpbiB0aGUgRFAgdXBkYXRlLCBidXQgaXQncyBub3QgdGhlIG1vc3QgZWZmaWNpZW50IHdheSB0byBjb21wdXRlIHRoZSBtaW5pbXVtLgoKLS0tCgojIyMgICoqT3B0aW1pemVkIFZlcnNpb24qKgoKSGVyZSdzIHRoZSAqKm9wdGltaXplZCB2ZXJzaW9uKiogb2YgdGhlIGNvZGUsIHdoaWNoIGlzIGZhc3RlciBhbmQgbW9yZSBlZmZpY2llbnQ6)Certainly\!Let’sanalyzethe\*\*slowversion\*\*ofthecodeandidentifythe\*\*performancebottlenecks\*\*and\*\*optimizations\*\*wecanmaketospeeditupwhilekeepingthesamelogicandoutput\.\-\-\-\#\#\#\*\*SlowVersionAnalysis\*\*Thecodeusesa2DDParray‘dp\[i\]\[j\]‘torepresenttheminimumnumberofedits\(insertions,deletions,substitutions\)requiredtoconvertthefirst‘i‘charactersof‘a‘tothefirst‘j‘charactersof‘b‘\.\#\#\#\#KeyIssues:1\.\*\*Double‘REP‘loops\*\*:\-Thecodehastwo‘REP‘loops:onefor‘i‘from0to‘n‘,andonefor‘j‘from0to‘m‘\.\-ThisisnotthemostefficientwaytoinitializetheDPtable\.2\.\*\*Unnecessary‘memset‘\*\*:\-The‘memset\(dp,0,sizeof\(dp\)\)‘iscalled,butwecaninitializetheDPtableusingamoreefficientmethod\.3\.\*\*Redundant‘REP‘loop\*\*:\-The‘REP\(i,1,n\+1\)‘and‘REP\(j,1,m\+1\)‘areredundantandcanbesimplified\.4\.\*\*Inefficient‘min‘operation\*\*:\-The‘min‘operationisusedintheDPupdate,butit’snotthemostefficientwaytocomputetheminimum\.\-\-\-\#\#\#\*\*OptimizedVersion\*\*Here’sthe\*\*optimizedversion\*\*ofthecode,whichisfasterandmoreefficient:[⬇](data:text/plain;base64,I2luY2x1ZGUgPGJpdHMvc3RkYysrLmg+CnVzaW5nIG5hbWVzcGFjZSBzdGQ7CnR5cGVkZWYgbG9uZyBsb25nIGludCBsbDsKY29uc3QgaW50IElORiA9IDEwMDAwMDAwMDA7CiNkZWZpbmUgUkVQKGksIHMsIG4pIGZvcihpbnQgaSA9IChpbnQpKHMpOyBpIDwgKGludCkobik7IGkrKykKI2RlZmluZSByZXAoaSwgbikgUkVQKGksIDAsIG4pCmludCBkcFsxMDAxXVsxMDAxXTsKaW50IG1haW4oKSB7CiAgICBjaW4udGllKDApOwogICAgaW9zOjpzeW5jX3dpdGhfc3RkaW8oZmFsc2UpOwogICAgc3RyaW5nIGEsIGI7CiAgICBjaW4gPj4gYSA+PiBiOwogICAgaW50IG4gPSBhLnNpemUoKSwgbSA9IGIuc2l6ZSgpOwogICAgLy8gSW5pdGlhbGl6ZSBEUCB0YWJsZQogICAgcmVwKGksIG0gKyAxKSBkcFswXVtpXSA9IGk7CiAgICByZXAoaSwgbiArIDEpIGRwW2ldWzBdID0gaTsKICAgIC8vIEZpbGwgRFAgdGFibGUKICAgIGZvciAoaW50IGkgPSAxOyBpIDw9IG47IGkrKykgewogICAgICAgIGZvciAoaW50IGogPSAxOyBqIDw9IG07IGorKykgewogICAgICAgICAgICAvLyBDb3B5IHByZXZpb3VzIHZhbHVlCiAgICAgICAgICAgIGRwW2ldW2pdID0gZHBbaSAtIDFdW2ogLSAxXTsKICAgICAgICAgICAgLy8gQ2hlY2sgaWYgY2hhcmFjdGVycyBkaWZmZXIKICAgICAgICAgICAgaWYgKGFbaSAtIDFdICE9IGJbaiAtIDFdKSB7CiAgICAgICAgICAgICAgICBkcFtpXVtqXSArPSAxOwogICAgICAgICAgICB9CiAgICAgICAgICAgIC8vIFRha2UgdGhlIG1pbmltdW0gb2YgdGhlIHRocmVlIG9wdGlvbnMKICAgICAgICAgICAgZHBbaV1bal0gPSBtaW4oZHBbaV1bal0sIG1pbihkcFtpIC0gMV1bal0gKyAxLCBkcFtpXVtqIC0gMV0gKyAxKSk7CiAgICAgICAgfQogICAgfQogICAgY291dCA8PCBkcFtuXVttXSA8PCBlbmRsOwogICAgcmV0dXJuIDA7Cn0=)usingnamespacestd;typedeflonglongintll;constintINF=1000000000;intdp\[1001\]\[1001\];intmain\(\)\{cin\.tie\(0\);ios::sync\_with\_stdio\(false\);stringa,b;cin\>\>a\>\>b;intn=a\.size\(\),m=b\.size\(\);rep\(i,m\+1\)dp\[0\]\[i\]=i;rep\(i,n\+1\)dp\[i\]\[0\]=i;for\(inti=1;i<=n;i\+\+\)\{for\(intj=1;j<=m;j\+\+\)\{dp\[i\]\[j\]=dp\[i\-1\]\[j\-1\];if\(a\[i\-1\]\!=b\[j\-1\]\)\{dp\[i\]\[j\]\+=1;\}dp\[i\]\[j\]=min\(dp\[i\]\[j\],min\(dp\[i\-1\]\[j\]\+1,dp\[i\]\[j\-1\]\+1\)\);\}\}cout<<dp\[n\]\[m\]<<endl;return0;\}[⬇](data:text/plain;base64,LS0tCgojIyMgICoqUGVyZm9ybWFuY2UgSW1wcm92ZW1lbnRzKioKCnwgQ2hhbmdlIHwgRGVzY3JpcHRpb24gfCBJbXBhY3QgfAp8LS0tLS0tLS18LS0tLS0tLS0tLS0tLXwtLS0tLS0tLXwKfCBgUkVQYCBtYWNybyB8IFJlcGxhY2VzIHJlZHVuZGFudCBgUkVQYCBsb29wcyB8IFJlZHVjZXMgY29kZSBkdXBsaWNhdGlvbiB8CnwgYG1lbXNldGAgIGByZXBgIHwgTW9yZSBlZmZpY2llbnQgaW5pdGlhbGl6YXRpb24gb2YgRFAgdGFibGUgfCBGYXN0ZXIgaW5pdGlhbGl6YXRpb24gfAp8IGBtaW5gIG9wZXJhdGlvbiB8IFVzZXMgYG1pbmAgZGlyZWN0bHkgaW4gdGhlIERQIHVwZGF0ZSB8IE1vcmUgZWZmaWNpZW50IHRoYW4gbXVsdGlwbGUgYG1pbmAgY2FsbHMgfAp8IEF2b2lkcyB1bm5lY2Vzc2FyeSBgUkVQYCBsb29wcyB8IFJlZHVjZXMgb3ZlcmhlYWQgfCBJbXByb3ZlcyBzcGVlZCB8CgotLS0KCiMjIyAgKipLZXkgVGFrZWF3YXlzKioKCi0gVGhlIG9wdGltaXplZCB2ZXJzaW9uIGlzICoqZmFzdGVyKiogZHVlIHRvIHJlZHVjZWQgb3ZlcmhlYWQgYW5kIGJldHRlciBjb2RlIHN0cnVjdHVyZS4KLSBJdCBtYWludGFpbnMgKippZGVudGljYWwgb3V0cHV0KiogZm9yIGFsbCB2YWxpZCBpbnB1dHMuCi0gSXQgaXMgKipjbGVhbmVyIGFuZCBtb3JlIHJlYWRhYmxlKiogdGhhbiB0aGUgb3JpZ2luYWwuCgotLS0KCiMjIyAgKipGaW5hbCBOb3RlKioKClRoaXMgb3B0aW1pemVkIHZlcnNpb24gaXMgc3VpdGFibGUgZm9yICoqbGFyZ2UgaW5wdXRzKiogKHVwIHRvIDEwMDAgY2hhcmFjdGVycyBlYWNoKSBhbmQgaXMgZWZmaWNpZW50IGVub3VnaCBmb3IgdHlwaWNhbCB1c2UgY2FzZXMuIEl0IGlzIGFsc28gZWFzeSB0byB1bmRlcnN0YW5kIGFuZCBtYWludGFpbi4=)\-\-\-\#\#\#\*\*PerformanceImprovements\*\*\|Change\|Description\|Impact\|\|\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\|\|‘REP‘macro\|Replacesredundant‘REP‘loops\|Reducescodeduplication\|\|‘memset‘‘rep‘\|MoreefficientinitializationofDPtable\|Fasterinitialization\|\|‘min‘operation\|Uses‘min‘directlyintheDPupdate\|Moreefficientthanmultiple‘min‘calls\|\|Avoidsunnecessary‘REP‘loops\|Reducesoverhead\|Improvesspeed\|\-\-\-\#\#\#\*\*KeyTakeaways\*\*\-Theoptimizedversionis\*\*faster\*\*duetoreducedoverheadandbettercodestructure\.\-Itmaintains\*\*identicaloutput\*\*forallvalidinputs\.\-Itis\*\*cleanerandmorereadable\*\*thantheoriginal\.\-\-\-\#\#\#\*\*FinalNote\*\*Thisoptimizedversionissuitablefor\*\*largeinputs\*\*\(upto1000characterseach\)andisefficientenoughfortypicalusecases\.Itisalsoeasytounderstandandmaintain\.