Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

# Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery
Source: [https://arxiv.org/html/2609.19164](https://arxiv.org/html/2609.19164)
Runhao LiuYongkang ZhangYuhao LiaoZijian ZengHuiming Yang

###### Abstract

In verifier\-style RLVR, group\-relative optimization often treats advantage scale as an implementation detail\. This paper separates two low\-variance cases: sub\-resolution jitter that should not become a preference signal, and credible but small cardinal gaps that should be learned without distorting KL calibration\. We propose an advantage\-scale three\-way calibration interface: the same within\-group scale denominator simultaneously determines the reward\-branch strength, prompt\-level batch weight, and the effective KL calibration induced when the reward branch is re\-expressed on the original cardinal scale\. This interface explains why RLOO / Dr\.GRPO can let credible small gaps become KL dominated, whereas GRPO’s standard\-deviation denominator can amplify tiny gaps without bound\. Based on this interface, we further introduce the Reward\-Resolution Protocol and MaxNorm\-AC, respectively filtering sub\-resolution gaps and providing bounded cardinal recovery on credible nonzero gaps\. Across dense / MoE architectures and math / code reasoning, MaxNorm\-AC improves over the prespecified percentile\-scale reference \(p=90p=90, hereafter thep​90p90reference\) while truncating the low\-variance inverse\-scale tail\.

1Alibaba Group

2Tsinghua University

Figure 1:Two low\-variance regimes expose three existing failures at one reward\-scale interface\. For sub\-resolution jitterqnq\_\{n\}, GRPO restoresx=O⁡\(σq\)x=O\(\\sigma\_\{q\}\)toO⁡\(1\)O\(1\), but1/σq1/\\sigma\_\{q\}has no uniform bound and reweights prompts in a fixed batch\. For a credible small gapqcq\_\{c\}, theO⁡\(ϵc\)O\(\\epsilon\_\{c\}\)reward push of RLOO / Dr\.GRPO may be relatively dominated by fixed KL\. The scale path, spring, and balance encode unbounded amplification, KL pressure, and batch reweighting; our method is not shown\.## Introduction

In verifier\-style RLVR training for reasoning, group\-relative optimization constructs advantages from multiple responses to the same prompt and avoids an explicit value function\. We distinguish two low\-variance cases\. A sub\-resolution gap is numerical jitter below the minimum credible reward resolution and should not be used as a preference signal\. A credible low\-variance gap can arise from partial credit, process quality\([Wang et al\. 2026b](https://arxiv.org/html/2609.19164#bib.bib54);[Yue et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib67)\), confidence, or error severity; such gaps should be learned, but their scale must be calibrated\. The Reward\-Resolution Protocol first merges sub\-resolution gaps before any group\-relative update and skips both reward and KL updates for zero\-gap groups\.

After this protocol, we focus on credible within\-group reward gaps that are small but nonzero\. In this regime, the prompt\-level scale is a three\-way interface: it sets the reward\-branch strength, the KL calibration seen by that prompt, and the prompt’s weight inside the batch\. Standard\-deviation normalization is therefore not ordinary preprocessing\. Without scale recovery, RLOO / Dr\.GRPO can become dominated by KL as the reward signal weakens; with standard\-deviation recovery, GRPO restores the signal but introduces unbounded amplification, effective\-KL imbalance, and prompt\-level reweighting as the within\-group standard deviation approaches zero\. Figure[1](https://arxiv.org/html/2609.19164#S0.F1)maps these three existing failures to the drive shaft, fixed\-KL spring, and batch balance of one scale machine; our bounded recovery is shown separately in Figure[2](https://arxiv.org/html/2609.19164#Sx8.F2)\.

This paper identifies low\-variance group\-relative updates as a three\-way scale\-calibration interface and gives a minimal bounded cardinal operator that only replaces the within\-group scale denominator\. This minimal intervention leaves KL regularization, the policy ratio, clipping, token aggregation, and length normalization unchanged, making the mechanism easier to attribute, the evidence more auditable, and practical transfer lower\-cost\. MaxNorm\-AC replaces the standard\-deviation denominator with the maximum absolute raw advantage under a reward\-resolution floor and freezes the result as the loss weight for the current optimization step, thereby jointly bounding reward amplification, the effective\-KL lower bound, and the prompt\-level weight tail; the exact bounds are summarized in the theory section below\. The main instance, MaxNorm\-RLOO, is only one way to attach this bounded\-scale operator\. If reward magnitudes are untrustworthy and only the within\-group order is reliable, the applicability assumption of our method no longer holds; order\-only methods should be treated as external boundaries rather than objects whose superiority is established by this paper\. Exactly identical within\-group rewards are likewise a signal\-construction boundary rather than a scale\-calibration case\.

The contributions are:

- •To our knowledge, we first formulate the within\-group denominator in RLVR as a unified scale\-calibration problem with three failure modes:O⁡\(1\)O\(1\)amplification of sub\-resolution jitter,O⁡\(ϵ\)O\(\\epsilon\)reward\-branch decay for credible small nonzero gaps, and prompt\-level batch reweighting induced by1/sq1/s\_\{q\}; the denominator should therefore jointly preserve credible cardinal gaps, suppress sub\-resolution jitter, and control KL calibration and prompt\-level reweighting;
- •We propose the Reward\-Resolution Protocol, which separates sub\-resolution noise from credible cardinal gaps before all group\-relative updates through bounded binning, a dead zone, and zero\-gap skipping;
- •We propose KL\-calibrated bounded scale as a diagnostic criterion and use MaxNorm\-AC as the minimal bounded intervention that verifies it; the operator only replaces the within\-group scale denominator and keeps KL regularization, the policy ratio, clipping, token aggregation, and length normalization unchanged;
- •We provide real benchmarks covering dense / MoE architectures and math / code RLVR, compare against the prespecifiedp​90p90reference, MAD, Huber, GRPO\+βq\\beta\_\{q\}\-comp, and REINFORCE\+\+, and validate the mechanism through controlled sub\-resolution diagnostics, inverse scales, the reward/KL gradient ratio, direction cosine, and KL P95\.

## Background and Related Work

This paper studies value\-free group\-relative policy optimization\. RLOO uses a leave\-one\-out REINFORCE baseline\([Ahmadian et al\. 2024](https://arxiv.org/html/2609.19164#bib.bib2)\); GRPO normalizes centered rewards by the within\-group standard deviation\([Shao et al\. 2024](https://arxiv.org/html/2609.19164#bib.bib49)\); Dr\.GRPO removes that standard\-deviation normalization and fixes length normalization\([Liu et al\. 2025c](https://arxiv.org/html/2609.19164#bib.bib31)\); REINFORCE\+\+ uses global advantage normalization\([Hu et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib17)\); and DAPO scales GRPO\-style training through system\-level recipes such as dynamic sampling and token\-level losses\([Yu et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib64)\)\. These lines make the advantage denominator a central algorithmic choice rather than bookkeeping\.

Related work studies difficulty bias, group weighting, KL placement, homogeneous rewards, reward corruption, zero\-variance signal construction, trajectory\-level correction, hard\-example selection, and multi\-objective / negative\-example pipelines\([Fontana et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib10);[Yao et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib63);[Liu et al\. 2025a](https://arxiv.org/html/2609.19164#bib.bib28);[He et al\. 2026b](https://arxiv.org/html/2609.19164#bib.bib15);[Mansouri et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib35);[Le et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib25);[Pang, Luo, and Jin 2026](https://arxiv.org/html/2609.19164#bib.bib43);[Pikus, Tiwari, and Ye 2025](https://arxiv.org/html/2609.19164#bib.bib44);[Li et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib26);[Liu and Xiao 2026](https://arxiv.org/html/2609.19164#bib.bib27)\)\. Recent GRPO design\-space studies, includingλ\\lambda\-GRPO, Hybrid GRPO, MEML\-GRPO, Stepwise / Spectral\-style policy optimization, Sharpness\-Guided GRPO, and on\-/off\-policy GRPO analyses, reshape token preferences, rollout construction, supervision, or update sharpness rather than giving a fixed bounded denominator for credible low\-variance cardinal gaps\([Wang et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib56);[Sane 2025](https://arxiv.org/html/2609.19164#bib.bib48);[Jia et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib20);[Chen et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib6);[Le, Van, and Le 2026](https://arxiv.org/html/2609.19164#bib.bib24);[Mroueh et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib40)\)\. Compute\-supervision analyses of RLVR further motivate treating verifier quality, reward reliability, and rollout structure as part of the learning problem\([Mitsuhashi et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib37)\)\. Ranking/listwise objectives, robust scales, and bounded\-ratio updates are important boundaries or baselines\([Choi et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib7);[Xiao, Zhang, and Cao 2025](https://arxiv.org/html/2609.19164#bib.bib59);[Zeng et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib68);[Ao et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib3)\)\. Robust\-scale denominators such as the prespecifiedp​90p90reference, MAD, Huber, and std\-floor can mitigate outlier scales or inverse\-scale tails, but they do not simultaneously satisfy our three\-way scale\-calibration constraints\. Our narrower target is*bounded cardinal advantage calibration*: preserve credible reward gaps while explicitly controlling normalized advantages, effective\-KL lower bounds, and prompt\-weight tails\. Detailed positioning is in the supplementary material\.

## Problem Setting: The Low\-Variance Reward Dilemma

In group\-based reinforcement learning for LLMs, a common practice is to sample multiple responses for the same prompt and construct relative advantages from the rewards of responses in the same group\. This paper does not study all within\-group learning algorithms, but instead focuses on a more specific question:in verifier\-style RLVR where reward gaps are credible and low\-variance but nonzero, should group\-relative advantages be divided by the standard deviation, and is the standard deviation an appropriate denominator?

#### Unified notation\.

Letrq,ir\_\{q,i\}be the reward of responseiifor promptqq,r¯q\\bar\{r\}\_\{q\}andσq\\sigma\_\{q\}be the group mean and standard deviation, andxq,i=rq,i−r¯qx\_\{q,i\}=r\_\{q,i\}\-\\bar\{r\}\_\{q\}be the centered reward\. To avoid overloading notation, we useuq,iu\_\{q,i\}for the raw numerator,cq,i=uq,i/sqc\_\{q,i\}=u\_\{q,i\}/s\_\{q\}for the scale\-calibrated advantage, andwq,i=sg⁡\(cq,i\)w\_\{q,i\}=\\operatorname\{sg\}\(c\_\{q,i\}\)for the frozen loss weight:

uq,i\\displaystyle u\_\{q,i\}raw numerator,\\displaystyle\\text\{raw numerator\},cq,i\\displaystyle c\_\{q,i\}=uq,i/sq,\\displaystyle=u\_\{q,i\}/s\_\{q\},wq,i\\displaystyle w\_\{q,i\}=sg⁡\(cq,i\)\.\\displaystyle=\\operatorname\{sg\}\(c\_\{q,i\}\)\.Onlywq,iw\_\{q,i\}enters the policy\-gradient loss\. GRPO usesAq,istd=xq,i/σqA^\{\\mathrm\{std\}\}\_\{q,i\}=x\_\{q,i\}/\\sigma\_\{q\}, whereσq=\(G−1​∑j\(rq,j−r¯q\)2\)1/2\\sigma\_\{q\}=\(G^\{\-1\}\\sum\_\{j\}\(r\_\{q,j\}\-\\bar\{r\}\_\{q\}\)^\{2\}\)^\{1/2\}\. RLOO usesaq,iRLOO=rq,i−\(G−1\)−1​∑j≠irq,ja^\{\\mathrm\{RLOO\}\}\_\{q,i\}=r\_\{q,i\}\-\(G\-1\)^\{\-1\}\\sum\_\{j\\neq i\}r\_\{q,j\}\. Thus the main question is the scale choice: RLOO / Dr\.GRPO preserve raw gaps and can suffer signal depletion, whereas GRPO restores magnitude but introduces a1/σq1/\\sigma\_\{q\}prompt\-dependent weight\. MaxNorm\-AC preserves credible cardinal gaps while bounding this weight; MaxNorm\-RLOO applies the operator to the RLOO numerator\.

## Without Standard\-Deviation Division: RLOO / Dr\.GRPO and Signal Depletion

RLOO is naturally obtained from REINFORCE with a leave\-one\-out baseline, not as a GRPO variant\. With the group mean including the current sample,aq,iRLOO=GG−1​\(rq,i−r¯q\)a^\{\\mathrm\{RLOO\}\}\_\{q,i\}=\\frac\{G\}\{G\-1\}\(r\_\{q,i\}\-\\bar\{r\}\_\{q\}\), so the RLOO numerator and the centered GRPO numerator have the same signs, rankings, and directions; the derivation is in the supplementary material\. The core distinction is therefore whether the centered signal is divided byσq\\sigma\_\{q\}\. Removing that denominator avoids random prompt reweighting, but under low\-variance rewards it can also leave the reward branch too weak\.

## Consequences of Signal Depletion: KL Domination and Policy Regression

If the credible reward gap under the same prompt is small, the raw RLOO / Dr\.GRPO advantage also becomes small\. Forr=\[0\.51,0\.50,0\.49,0\.50\]r=\[0\.51,0\.50,0\.49,0\.50\], the RLOO advantage is approximately\[0\.0133,0,−0\.0133,0\]\[0\.0133,0,\-0\.0133,0\]\. A KL\-regularized update can be abstracted as

gtotal=gR−β​gKL\.g\_\{\\mathrm\{total\}\}=g\_\{R\}\-\\beta g\_\{\\mathrm\{KL\}\}\.\(1\)
When low variance shrinks‖gR‖\\\|g\_\{R\}\\\|belowβ​‖gKL‖\\beta\\\|g\_\{\\mathrm\{KL\}\}\\\|, the total direction becomes closer to−gKL\-g\_\{\\mathrm\{KL\}\}, and the update primarily reduces the KL between the current policy and the reference policy rather than moving in the direction that improves task reward\. This phenomenon is not an ordinary plateau: if the policy shift away from the reference is precisely the direction that yields reasoning gains, a KL\-dominated update can pull the model back toward the reference model, appearing as a decline in accuracy or reward together with a decrease in KL\.

Thus, avoiding scale recovery prevents GRPO’s denominator explosion but exposes the opposite failure mode: credible low\-magnitude gaps may not provide a sufficiently strong reward signal\. Gradient\-norm bounds and KL\-decrease conditions are in the supplementary material\.

## Equivalent Interpretation of KL Domination and Performance Regression

LetgRg\_\{R\}be the unattenuated reward gradient andgKLg\_\{\\mathrm\{KL\}\}the KL gradient\. If a credible low\-variance signal shrinks the reward branch toα​gR\\alpha g\_\{R\},0<α<10<\\alpha<1, then the total update direction is

gα=α​gR−β​gKL=α⁡\(gR−βα​gKL\)\.g\_\{\\alpha\}=\\alpha g\_\{R\}\-\\beta g\_\{\\mathrm\{KL\}\}=\\alpha\\left\(g\_\{R\}\-\\frac\{\\beta\}\{\\alpha\}g\_\{\\mathrm\{KL\}\}\\right\)\.\(2\)
Thus, in the sense that an overall positive scaling does not change the direction, low\-variance signal attenuation is equivalent to increasing the KL coefficient to

βeff=βα\.\\beta\_\{\\mathrm\{eff\}\}=\\frac\{\\beta\}\{\\alpha\}\.\(3\)
Ifα​‖gR‖<β​‖gKL‖\\alpha\\\|g\_\{R\}\\\|<\\beta\\\|g\_\{\\mathrm\{KL\}\}\\\|, the update is closer to−gKL\-g\_\{\\mathrm\{KL\}\}and can reduce KL to the reference while also reducing task performance\. We use this only as a local diagnostic; rigorous small\-step proofs are in the supplementary material\.

## Standard\-Deviation Normalization: Signal Recovery and Scale Rewriting

Standard\-deviation normalization can recover low\-variance signals, but it recovers a recalibrated surrogate rather than the original cardinal reward objective itself\. For a single promptqq, letgR,q0g\_\{R,q\}^\{0\}denote the cardinal reward gradient without standard\-deviation division,gKL,qg\_\{\\mathrm\{KL\},q\}denote the KL gradient, andsqs\_\{q\}denote the within\-group scale denominator\. A GRPO\-style update can be written as

gqGRPO=1sq​gR,q0−β​gKL,q=1sq​\(gR,q0−β​sq​gKL,q\)\.g\_\{q\}^\{\\mathrm\{GRPO\}\}=\\frac\{1\}\{s\_\{q\}\}g\_\{R,q\}^\{0\}\-\\beta g\_\{\\mathrm\{KL\},q\}=\\frac\{1\}\{s\_\{q\}\}\\left\(g\_\{R,q\}^\{0\}\-\\beta s\_\{q\}g\_\{\\mathrm\{KL\},q\}\\right\)\.\(4\)
Becausesq\>0s\_\{q\}\>0, if the current batch and denominator are frozen and the pre\-optimizer first\-order direction is viewed on the unnormalized cardinal reward scale, this direction can be diagnostically re\-expressed with the local effective KL coefficient

βeff,qGRPO=β​sq\.\\beta\_\{\\mathrm\{eff\},q\}^\{\\mathrm\{GRPO\}\}=\\beta s\_\{q\}\.\(5\)
Whensq=σq→0s\_\{q\}=\\sigma\_\{q\}\\to 0, GRPO restores anO⁡\(sq\)O\(s\_\{q\}\)reward difference to anO⁡\(1\)O\(1\)advantage, while the prompt’s local effective KL diagnostic becomesβ​σq\\beta\\sigma\_\{q\}and its reward\-branch batch weight becomes1/σq1/\\sigma\_\{q\}\. Thus training may enter a local calibration imbalance: low variance can weaken the reward branch and increase the risk of KL\-dominated regression, whereas an overly small standard\-deviation denominator can amplify reward noise and prompt\-level weight fluctuations, reducing overall training stability\. This is a diagnosable local mechanism, not a guarantee about the full PPO / AdamW optimization dynamics\. We therefore filter sub\-resolution gaps first, then apply bounded scale recovery only to nonzero cardinal gaps that pass the protocol\. General derivations, batch reweighting, and AdamW caveats are in the supplementary material\.

## Reward\-Resolution Protocol and MaxNorm\-AC

Figure 2:MaxNorm\-AC inserts the scale cardsq=max⁡\{τres,maxj⁡\|uq,j\|\}s\_\{q\}=\\max\\\{\\tau\_\{\\mathrm\{res\}\},\\max\_\{j\}\|u\_\{q,j\}\|\\\}only into the denominator of the trusted raw numeratoruq,iu\_\{q,i\}, yielding ratio\- and rank\-preservingcq,i=uq,i/sqc\_\{q,i\}=u\_\{q,i\}/s\_\{q\}with\|cq,i\|≤1\|c\_\{q,i\}\|\\leq 1\. The frozen weightwq,i=sg⁡\(cq,i\)w\_\{q,i\}=\\operatorname\{sg\}\(c\_\{q,i\}\)enters only the reward branch: gradients flow throughρθ\\rho\_\{\\theta\}and stop at SG, while base KL joins independently without division bysqs\_\{q\}\. The probability ratio, clipping, token aggregation, length normalization, and optimizer remain unchanged; the card replaces GRPO’sσq\\sigma\_\{q\}or the equivalent denominator11of unscaled methods\.### Reward\-Resolution Protocol

Arbitrarily small numerical differences in continuous verifier rewards do not necessarily correspond to credible preferences\. To prevent group\-relative methods from treating sub\-resolution noise as preference signal, this paper defines a unified Reward\-Resolution Protocol in the reward\-function design / reward pipeline:

- •Bounded rewards: clip rewards to a fixed range such as\[0,1\]\[0,1\]and cap the contribution of each reward component; single anomalous or extreme reward values should be handled at the reward\-function design stage through bounding, clipping, saturating maps, and per\-component contribution caps, rather than detected by MaxNorm\-AC;
- •Minimum credible resolution: set the minimum credible reward resolution toδres\>0\\delta\_\{\\mathrm\{res\}\}\>0\. Differences below this resolution are treated as belonging to the same bin and do not induce a relative preference;
- •Zero\-gap skipping: if all responses to the same prompt fall into the same bin, the group contains no credible cardinal reward gap, and both reward and KL updates are skipped during training to avoid KL\-only reference regression\.

This protocol is not an additional optimizer, but a predefined reliability gate that specifies which cardinal reward gaps are credible enough to enter subsequent bounded scale calibration\. All methods share bounded rewards, component caps, data, and optimizer settings; the resolution gate, train\-time merging, and zero\-gap skipping are reported as method\-level switches in Table[2](https://arxiv.org/html/2609.19164#Sx11.T2)\. The standard GRPO row disables the resolution gate, whereas GRPO\+Protocol adds only that gate to isolate its effect\. In our\[0,1\]\[0,1\]verifier setting,δres=10−2\\delta\_\{\\mathrm\{res\}\}=10^\{\-2\}is fixed from the reward\-design resolution before training rather than chosen by validation search; more concretely, if reward componentkkhas minimum effective stepΔk\\Delta\_\{k\}and aggregation weightwk\>0w\_\{k\}\>0, then one auditable component\-level change contributes at leastδpipe=mink:wk\>0wkΔk\\delta\_\{\\mathrm\{pipe\}\}=\\min\_\{k:w\_\{k\}\>0\}w\_\{k\}\\Delta\_\{k\}after aggregation, and we setδres\\delta\_\{\\mathrm\{res\}\}to the same order as this pipeline resolution as a conservative gate rather than a MaxNorm\-AC hyperparameter tuned on validation accuracy\. Differences below this scale are treated as unresolved after component quantization, clipping, process\-score aggregation, and repeated verifier noise\. Theτres\\tau\_\{\\mathrm\{res\}\}used in MaxNorm\-AC denotes the explicit safety floor induced by this reward resolution in the advantage denominator and satisfiesτres=Θ⁡\(δres\)\\tau\_\{\\mathrm\{res\}\}=\\Theta\(\\delta\_\{\\mathrm\{res\}\}\); it is not an independently searched optimization hyperparameter, and zero\-gap groups are accounted for separately through the reported exact zero\-gap skip rate\. If a reward pipeline lacks explicit reward\-component resolutions,δres\\delta\_\{\\mathrm\{res\}\}should be pre\-calibrated from an upper quantile of reward jitter estimated by repeated verification, expert re\-labeling, or historical agreement, with neighboring sensitivity reported rather than validation\-accuracy search\.

### MaxNorm\-AC

For a low\-variance group that still contains credible nonzero gaps after passing the Reward\-Resolution Protocol, our method chooses neither no normalization nor standard\-deviation division, but instead uses the maximum absolute raw numerator with a reward\-resolution floor as the denominator\. We call this module MaxNorm\-AC\. Its goal is to recover credible low\-variance signals while avoiding unbounded signal amplification caused by the standard\-deviation denominator, and to bound the arbitrarily large relative weight that a low\-variance group can receive in batch aggregation\.

Letuq,iu\_\{q,i\}denote the raw numerator produced by any within\-group method\. Taking RLOO as an example, the raw numerator is:

aq,iRLOO=rq,i−1G−1​∑j≠irq,j\.a\_\{q,i\}^\{\\mathrm\{RLOO\}\}=r\_\{q,i\}\-\\frac\{1\}\{G\-1\}\\sum\_\{j\\neq i\}r\_\{q,j\}\.\(6\)
The centered numerator of GRPO or Dr\.GRPO can be written as:

xq,i=rq,i−r¯q,uq,i=xq,i\.x\_\{q,i\}=r\_\{q,i\}\-\\bar\{r\}\_\{q\},\\qquad u\_\{q,i\}=x\_\{q,i\}\.\(7\)
MaxNorm\-AC uniformly uses the maximum absolute raw advantage within the group for bounded recovery, and freezes the scale\-calibrated advantage as a fixed weight in the policy\-gradient loss:

sq\\displaystyle s\_\{q\}=max⁡\{max1≤j≤G⁡\|uq,j\|,τres\},\\displaystyle=\\max\\\{\\max\_\{1\\leq j\\leq G\}\|u\_\{q,j\}\|,\\tau\_\{\\mathrm\{res\}\}\\\},\(8\)cq,i\\displaystyle c\_\{q,i\}=uq,isq,\\displaystyle=\\frac\{u\_\{q,i\}\}\{s\_\{q\}\},wq,i\\displaystyle w\_\{q,i\}=sg⁡\(cq,i\)\.\\displaystyle=\\operatorname\{sg\}\\left\(c\_\{q,i\}\\right\)\.
Hereτres\\tau\_\{\\mathrm\{res\}\}is the reward\-resolution floor given by the Reward\-Resolution Protocol, andsg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes stop\-gradient; the denominator is not learned or optimized through backpropagation\. Sincesq≥maxj⁡\|uq,j\|s\_\{q\}\\geq\\max\_\{j\}\|u\_\{q,j\}\|, the scale\-calibrated advantage satisfies\|cq,i\|≤1\|c\_\{q,i\}\|\\leq 1, and the frozen weight entering the loss also satisfies\|wq,i\|≤1\|w\_\{q,i\}\|\\leq 1, while preserving the sign and within\-group ranking direction of the raw numerator\. MaxNorm\-AC only replaces the advantage scale denominator; it does not change the policy ratio, clipping, length normalization, or reward construction\. It bounds the unbounded reward\-branch reweighting that may arise in GRPO asσq→0\\sigma\_\{q\}\\to 0\.

Whenuq,i=aq,iRLOOu\_\{q,i\}=a\_\{q,i\}^\{\\mathrm\{RLOO\}\}, the expression corresponds to MaxNorm\-RLOO; whenuq,i=rq,i−r¯qu\_\{q,i\}=r\_\{q,i\}\-\\bar\{r\}\_\{q\}, it replaces the standard\-deviation denominator in GRPO or Dr\.GRPO\. We report MaxNorm\-RLOO as the representative instance because the RLOO and centered numerators differ by a positive constant when the floor is inactive, and we include MaxNorm\-Dr\.GRPO in Tables[1](https://arxiv.org/html/2609.19164#Sx10.T1)and[3](https://arxiv.org/html/2609.19164#Sx11.T3)to show that the result is not tied to the RLOO numerator\. In the example above,aq,1:4RLOO≈\[0\.0133,0,−0\.0133,0\]a^\{\\mathrm\{RLOO\}\}\_\{q,1:4\}\\approx\[0\.0133,0,\-0\.0133,0\]becomescq,1:4≈\[1,0,−1,0\]c\_\{q,1:4\}\\approx\[1,0,\-1,0\], while1/sq≤1/τres1/s\_\{q\}\\leq 1/\\tau\_\{\\mathrm\{res\}\}still bounds the prompt’s batch weight\.

## Algorithmic Placement and Training Recipe

To avoid ambiguity about the implementation location, this paper writes MaxNorm\-AC as an independent advantage\-calibration operator\. Figure[2](https://arxiv.org/html/2609.19164#Sx8.F2)depicts the intervention as a single scale cartridge placed after reward / advantage construction and before the policy\-gradient loss; KL regularization, policy ratio, clipping, token aggregation, and length normalization all retain the original definitions of the base algorithm\.

Algorithm 1: MaxNorm\-AC training steps\.For each prompt, sampleGGresponses, compute rewards, construct a raw numeratoruq,i=B⁡\(q,oq,i,\{rq,j\}j=1G\)u\_\{q,i\}=B\(q,o\_\{q,i\},\\\{r\_\{q,j\}\\\}\_\{j=1\}^\{G\}\)from the base group\-relative method, setsq=max⁡\{maxj⁡\|uq,j\|,τres\}s\_\{q\}=\\max\\\{\\max\_\{j\}\|u\_\{q,j\}\|,\\tau\_\{\\mathrm\{res\}\}\\\}, and replace the base loss advantage bywq,i=sg⁡\(uq,i/sq\)w\_\{q,i\}=\\operatorname\{sg\}\(u\_\{q,i\}/s\_\{q\}\)\.

In Eqs\. \(8\)–\(10\),uq,i,sq,cq,i,wq,iu\_\{q,i\},s\_\{q\},c\_\{q,i\},w\_\{q,i\}, sampled rollouts, rewards, andLnormL\_\{\\mathrm\{norm\}\}are fixed for the current update; gradients are taken only with respect to policy parametersθ\\thetathroughρq,i,t​\(θ\)\\rho\_\{q,i,t\}\(\\theta\),ρ¯q,i,t​\(θ\)\\bar\{\\rho\}\_\{q,i,t\}\(\\theta\), andℒKL​\(θ\)\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\(\\theta\)\. For a PPO / GRPO\-style clipped loss with ratioρq,i,t\\rho\_\{q,i,t\}and clipped ratioρ¯q,i,t\\bar\{\\rho\}\_\{q,i,t\}, the reward branch becomes

ℒpg​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{pg\}\}\(\\theta\)=−𝔼q​\[1G​∑i=1G1Lnorm​∑t=1\|oq,i\|ℓq,i,tAC​\(θ\)\],\\displaystyle=\-\\mathbb\{E\}\_\{q\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{L\_\{\\mathrm\{norm\}\}\}\\sum\_\{t=1\}^\{\|o\_\{q,i\}\|\}\\ell^\{\\mathrm\{AC\}\}\_\{q,i,t\}\(\\theta\)\\right\],\(9\)ℓq,i,tAC​\(θ\)\\displaystyle\\ell^\{\\mathrm\{AC\}\}\_\{q,i,t\}\(\\theta\)=min⁡\{ρq,i,t​\(θ\)​wq,i,ρ¯q,i,t​\(θ\)​wq,i\}\.\\displaystyle=\\min\\\{\\rho\_\{q,i,t\}\(\\theta\)w\_\{q,i\},\\bar\{\\rho\}\_\{q,i,t\}\(\\theta\)w\_\{q,i\}\\\}\.
HereLnormL\_\{\\mathrm\{norm\}\}is inherited from the base algorithm: response\-level averaging uses\|oq,i\|\|o\_\{q,i\}\|, while Dr\.GRPO uses the fixed budgetLmaxL\_\{\\max\}\. KL regularization remains outside the reward branch,

ℒ⁡\(θ\)=ℒpg​\(θ\)\+β​ℒKL​\(θ\),\\mathcal\{L\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{pg\}\}\(\\theta\)\+\\beta\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\(\\theta\),\(10\)
whereℒKL\\mathcal\{L\}\_\{\\mathrm\{KL\}\}can use the token\-level or sequence\-level KL already provided by the base algorithm\. MaxNorm\-AC does not applysqs\_\{q\}to the KL term; it changes reward/KL calibration through the reward branch only\. Thus it composes with Dr\.GRPO: MaxNorm\-AC handles the question\-level denominator, while the fixed length budget handles response\-level length normalization\.

## Theory Summary

The main\-text theory is a constructive failure\-mode analysis plus a bounded\-recovery criterion: under a fixed current batch, fixed scale denominators, and a local first\-order surrogate, it formalizes sub\-resolution jitter amplification, small\-gap reward\-branch decay, and prompt\-level batch reweighting, then gives the bounds that MaxNorm\-AC enforces on the same denominator interface\. Theτres\>0\\tau\_\{\\mathrm\{res\}\}\>0supplied by the Reward\-Resolution Protocol defines

sq=max⁡\{maxj⁡\|uq,j\|,τres\}\.s\_\{q\}=\\max\\\{\\max\_\{j\}\|u\_\{q,j\}\|,\\tau\_\{\\mathrm\{res\}\}\\\}\.\(11\)It immediately gives the three bounds used as diagnostics:

\|wq,i\|\\displaystyle\|w\_\{q,i\}\|≤1,\\displaystyle\\leq 1,1/sq\\displaystyle 1/s\_\{q\}≤1/τres,\\displaystyle\\leq 1/\\tau\_\{\\mathrm\{res\}\},βeff,q=β​sq\\displaystyle\\beta\_\{\\mathrm\{eff\},q\}=\\beta s\_\{q\}≥β​τres\.\\displaystyle\\geq\\beta\\tau\_\{\\mathrm\{res\}\}\.The first two bounds control the normalized advantage and prompt\-level amplification; the third is a fixed\-batch, fixed\-denominator local diagnostic on the original cardinal reward scale, not a claim that PPO/AdamW optimizes with prompt\-specificβ​sq\\beta s\_\{q\}\. In the main setting,τres=10−2\\tau\_\{\\mathrm\{res\}\}=10^\{\-2\}andβkl=0\.002\\beta\_\{\\mathrm\{kl\}\}=0\.002, givingβeff,q≥2×10−5\\beta\_\{\\mathrm\{eff\},q\}\\geq 2\\times 10^\{\-5\}for retained nonzero\-gap groups\. MaxNorm\-AC introduces bounded prompt\-level reweighting bias; because the same positive scale is shared within each prompt, it preserves within\-group signs, ordering, and relative\-magnitude ratios\.

Table 1:Per\-task benchmark gains restored in the main text\. The strongest core baseline is selected per model–task unit among RLOO \+ Protocol, GRPO, GRPO\+βq\\beta\_\{q\}\-comp, Dr\.GRPO, and REINFORCE\+\+; GRPO\+βq\\beta\_\{q\}\-comp is selected in all six units\. The prespecifiedp​90p90reference column exposes the percentile\-scale paired comparison, with MaxNorm\-RLOO higher on all six rows\. GOPO is the pure\-ranking boundary baseline\. MaxNorm\-Dr\.GRPO is the centered\-numerator check;Δ​vs​p​90​ref\.\\Delta\\text\{ vs \}p90\\text\{ ref\.\}reports the MaxNorm\-RLOO improvement over this reference\.This criterion also clarifies the limits of the prespecifiedp​90p90reference, MAD, Huber, and std\-floor\. Thep​90p90reference, MAD, and Huber are robust scale estimators: they reduce outlier influence, but their denominators need not cover the largest credible numerator and therefore do not constructively guarantee the normalized\-advantage bound\. std\-floor only truncates the inverse\-σq\\sigma\_\{q\}tail and likewise cannot bound the largest normalized advantage\. MaxNorm\-AC puts the maximum absolute raw numerator directly into the denominator, so the three bounds hold simultaneously\.

The theoretical claim is therefore a testable three\-way calibration criterion: for every retained nonzero group, the denominator jointly modulates reward strength, prompt weight, and KL calibration on the original reward scale, and should induce coupled changes in the1/sq1/s\_\{q\}tail, reward/KL ratio, contribution concentration, direction cosine, and matched\-KL ranking; tail clipping without improved alignment or matched\-KL performance is insufficient\. The supplementary material gives controlled numerical diagnostics, the AdamW scaling derivation, and first\-order sufficient conditions for bounded prompt reweighting to remain an improvement direction\.

## Empirical Results and Core Diagnostics

Table 2:Single\-column component ablation\. The Protocol column indicates whether the reward\-pipeline resolution gate is enabled; even when Protocol is disabled, reward bounding, component caps, data, and optimizer settings remain fixed\. The train\-timeδres\\delta\_\{\\mathrm\{res\}\}\-binning column only indicates whether the algorithm additionally merges sub\-resolution gaps before advantage construction\. GRPO and GRPO\+Protocol use the same standard\-deviation denominator and isolate the resolution gate; RLOO\+Protocol and MaxNorm\-RLOO no\-δres\\delta\_\{\\mathrm\{res\}\}\-binning share the three settings and isolate denominator calibration;†\\daggermarks the Protocol\-off but train\-time\-binned row, which has the same effective gaps as Full MaxNorm\-RLOO\. The Protocol/binning off row reports measured78\.0/0\.05878\.0/0\.058\.Tables[1](https://arxiv.org/html/2609.19164#Sx10.T1)–[2](https://arxiv.org/html/2609.19164#Sx11.T2)separate benchmark, diagnostic, and ablation evidence\. We use Qwen3\-32B and Qwen3\-Next\-80B\-A3B\-Thinking; train on decontaminated DeepMath\-103K and OpenCodeReasoning\([He et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib16);[Ahmad et al\. 2025](https://arxiv.org/html/2609.19164#bib.bib1)\); and evaluate on AIME25, HMMT25, and LiveCodeBench v6\([MAA Communications 2024](https://arxiv.org/html/2609.19164#bib.bib34);[Balunović et al\. 2026](https://arxiv.org/html/2609.19164#bib.bib4);[Jain et al\. 2024](https://arxiv.org/html/2609.19164#bib.bib19)\)\. All methods share five seeds, data, sampling budget, optimizer, learning rate,βkl\\beta\_\{\\mathrm\{kl\}\}, max length, batch size,G=16G=16, validation prompts, and checkpoint indices; baseline scale hyperparameters use the shared beta audit and prespecified method\-specific settings documented in the supplement, withp=90p=90fixed before training for thep​90p90reference andτres=δres\\tau\_\{\\mathrm\{res\}\}=\\delta\_\{\\mathrm\{res\}\}fixed outside validation search for MaxNorm\-RLOO\. Entries are seed means with seed\-bootstrap95%95\\%CIs, and the pairedp​90p90\-reference CI bootstraps six model–task differences\. Official results use the final same\-compute checkpoint; the supplement reports matched\-KL and saved\-checkpoint audits, a consolidated training and sampling protocol, OOD retention, and validation prompts\.

Table 3:Low\-variance diagnostics on updated groups withσq<10−2\\sigma\_\{q\}<10^\{\-2\}\. R/KL is‖gR‖/\(β​‖gKL‖\)\\\|g\_\{R\}\\\|/\(\\beta\\\|g\_\{\\mathrm\{KL\}\}\\\|\), and Dir\. is the reward/KL direction cosine\. Per\-task accuracy and pairedp​90p90\-reference comparisons are in Table[1](https://arxiv.org/html/2609.19164#Sx10.T1); checkpoint audits are in the supplement\.All official methods share the same bounded reward construction, component caps, and training budget; Table[2](https://arxiv.org/html/2609.19164#Sx11.T2)reports the Reward\-Resolution Protocol, train\-timeδres\\delta\_\{\\mathrm\{res\}\}\-binning, and exact zero\-gap skipping as row\-level switches\. Standard GRPO disables the resolution gate, whereas GRPO\+Protocol enables only that gate; when enabled,δres=10−2\\delta\_\{\\mathrm\{res\}\}=10^\{\-2\}is fixed before training by reward\-design resolution rather than validation search, and differences below this scale are treated as unresolved after component quantization, clipping, process\-score aggregation, and repeated verifier noise\. We audit this fixed gate instead of tuning it: nearbyτres∈\{5×10−3,10−2,2×10−2\}\\tau\_\{\\mathrm\{res\}\}\\in\\\{5\\times 10^\{\-3\},10^\{\-2\},2\\times 10^\{\-2\}\\\}gives gains\+4\.65,\+5\.23,\+4\.42\+4\.65,\+5\.23,\+4\.42, the main setting has1/sq≤1001/s\_\{q\}\\leq 100andβeff,q≥2×10−5\\beta\_\{\\mathrm\{eff\},q\}\\geq 2\\times 10^\{\-5\}, floor activation is11\.8%−20\.4%11\.8\\%\-20\.4\\%of nonzero updated groups, and the exact zero\-gap skip rate in the default setting is2\.4%2\.4\\%\. Thus the evidence is exposed as a fixed reliability gate, local sensitivity check, and transparent exact zero\-gap skip rate report, rather than a hidden trust threshold\.

The tables preserve the core evidence checks without overloading a single table\. First, Table[1](https://arxiv.org/html/2609.19164#Sx10.T1)shows that the gains are not concentrated in one model or benchmark and makes the pairedp​90p90\-reference comparison visible in the main text: MaxNorm\-RLOO is positive against this prespecified reference on all six prespecified model–task units, with an average improvement of about\+4\.77\+4\.77\. Full paired differences are reported in the supplement\. Second, Table[3](https://arxiv.org/html/2609.19164#Sx11.T3)is restricted to low\-variance diagnostics\. Across sampled diagnostic batches, the top\-25% GRPO prompts exceed60\.0%60\.0\\%of reward\-branch contribution mass, above the25%25\\%uniform baseline, indicating prompt\-level concentration and possible batch\-level variance amplification; MaxNorm\-RLOO mitigates this risk by bounding each prompt’s normalized weight; thep​90p90reference truncates the tail but leaves weaker direction alignment and higher KL P95 than MaxNorm\-RLOO\. Third, the supplement audits checkpoint choice under matched\-KL and same\-compute final\-checkpoint views; MaxNorm\-RLOO remains above thep​90p90reference in both views \(matched KL:76\.576\.5vs\.73\.173\.1; same\-compute final checkpoint:78\.378\.3vs\.73\.573\.5\)\. Fourth, in Table[2](https://arxiv.org/html/2609.19164#Sx11.T2), RLOO\+Protocol and MaxNorm\-RLOO no\-δres\\delta\_\{\\mathrm\{res\}\}\-binning share the same settings and isolate denominator calibration, raising Acc from71\.471\.4to78\.378\.3; MaxNorm\-Dr\.GRPO stays close to MaxNorm\-RLOO with lower KL P95, ruling out the numerator explanation; with themax⁡\|u\|\\max\|u\|denominator fixed, keeping zero\-gap KL preserves about89\.6%89\.6\\%of the default gain, and fully skipping zero\-gap groups adds only\+0\.5\+0\.5Acc\.

For sub\-resolution noise, the constructed diagnosticr=r0\+10−6​\[1,0,−1,0\]r=r\_\{0\}\+10^\{\-6\}\[1,0,\-1,0\]withδres=10−2\\delta\_\{\\mathrm\{res\}\}=10^\{\-2\}givesσq=10−6/2\\sigma\_\{q\}=10^\{\-6\}/\\sqrt\{2\}: GRPO maps the gap to\[1\.414,0,−1\.414,0\]\[1\.414,0,\-1\.414,0\]andβeff≈1\.4×10−9\\beta\_\{\\mathrm\{eff\}\}\\approx 1\.4\\times 10^\{\-9\}, whereas RLOO / Dr\.GRPO have anO⁡\(10−6\)O\(10^\{\-6\}\)reward branch and become KL dominated only when‖gR‖≪β​‖gKL‖\\\|g\_\{R\}\\\|\\ll\\beta\\\|g\_\{\\mathrm\{KL\}\}\\\|\. The Reward\-Resolution Protocol instead merges the group before scale normalization\. In diagnostic batches, a higher share of low\-variance groups corresponds to a larger average gain of MaxNorm\-AC over the strongest core baseline; the three share bins\[0,0\.05\),\[0\.05,0\.15\),\[0\.15,1\]\[0,0\.05\),\[0\.05,0\.15\),\[0\.15,1\]yield gains of\+0\.9,\+3\.8,\+7\.6\+0\.9,\+3\.8,\+7\.6, respectively\. Additional diagnostic tables, localτres\\tau\_\{\\mathrm\{res\}\}sensitivity, and zero\-variance boundary handling are in the supplementary material\.

## Discussion: Boundaries and Applicability

MaxNorm\-AC is a bounded intervention under a mechanistic diagnosis, not a universal robust normalizer\. Its evidential scope is verifier\-style RLVR where reward gaps are credible, low\-variance, and nonzero\. If reward magnitudes are untrustworthy and only the within\-group order is reliable, the applicability assumption of our method no longer holds; order\-only methods should be treated as external boundaries rather than objects whose superiority is established by this paper\. Subjective judge rewards require a pre\-calibrated minimum credible resolution from expert labels, repeated\-judge agreement, or reward\-design documentation; otherwise small gaps should be treated as noise and handled by uncertainty gating or order\-only objectives\.

The supplement treats ranking/listwise methods as boundary baselines and analyzes near ties, many ties, ranking noise, and meaningful cardinal gaps\. MaxNorm\-AC also does not replace length correction, KL scheduling, reward modeling, or zero\-variance signal construction\. Zero\-gap skipping should be monitored: if zero\-gap groups are frequent, this indicates insufficient relative supervision in the data stream or an inappropriate reward\-resolution setting, rather than a reason to optimize those groups with KL\-only updates\. Deployments should report mean/P95 KL,1/sq1/s\_\{q\}tails, prompt\-weight distributions, low\-variance group fraction, exact zero\-gap skip rate, PPO clip\-hit rate, and reward/KL gradient ratio together\.

#### Ethics and safety boundary\.

This paper only changes the advantage scale in RLVR training\. It does not introduce a new reward objective, nor does it replace safety filtering, code\-execution sandboxes, or human oversight\. Stronger math and code RLVR optimization may also improve capabilities that can be misused\. Therefore, high\-risk deployments should include safety constraints at the reward\-design stage, conduct safety evaluations after training, and retain unit tests, static scanning, and human oversight for code\-generation tasks\.

## Conclusion

This paper formulates credible low\-variance RLVR as a calibration imbalance in the advantage scale\. MaxNorm\-AC replaces the standard\-deviation denominator with the maximum absolute raw numerator under a reward\-resolution floor, preserving credible cardinal gaps while guaranteeing\|wq,i\|≤1\|w\_\{q,i\}\|\\leq 1and1/sq≤1/τres1/s\_\{q\}\\leq 1/\\tau\_\{\\mathrm\{res\}\}\. Across dense and MoE math/code tasks, MaxNorm\-AC improves same\-compute accuracy, exceeds the prespecifiedp​90p90reference under matched KL, and suppresses low\-variance inverse\-scale tails; it applies when cardinal reward gaps are credible and nonzero\.

## References

- Ahmad et al\. \(2025\)Ahmad, W\. U\.; Narenthiran, S\.; Majumdar, S\.; Ficek, A\.; Jain, S\.; Huang, J\.; Noroozi, V\.; and Ginsburg, B\. 2025\.OpenCodeReasoning: Advancing Data Distillation for Competitive Coding\.arXiv:2504\.01943\.
- Ahmadian et al\. \(2024\)Ahmadian, A\.; Cremer, C\.; Gallé, M\.; Fadaee, M\.; Kreutzer, J\.; Pietquin, O\.; Üstün, A\.; and Hooker, S\. 2024\.Back to Basics: Revisiting REINFORCE\-Style Optimization for Learning from Human Feedback in LLMs\.In Ku, L\.\-W\.; Martins, A\.; and Srikumar, V\., eds\.,*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 12248–12267\. Bangkok, Thailand: Association for Computational Linguistics\.
- Ao et al\. \(2026\)Ao, Y\.; Chen, L\.; Lee, B\. D\.; Wahd, A\. S\.; Czarnobai, A\.; Fürnstahl, P\.; Schölkopf, B\.; and Krause, A\. 2026\.Bounded Ratio Reinforcement Learning\.arXiv:2604\.18578\.
- Balunović et al\. \(2026\)Balunović, M\.; Dekoninck, J\.; Petrov, I\.; Jovanović, N\.; and Vechev, M\. 2026\.MathArena: Evaluating LLMs on Uncontaminated Math Competitions\.arXiv:2505\.23281\.
- by arXiv \(2026\)by arXiv, R\. 2026\.LamPO: A Lambda Style Policy Optimization for Reasoning Language Models\.arXiv:2605\.21235\.
- Chen et al\. \(2026\)Chen, P\.; Li, X\.; Li, Z\.; Chen, X\.; and Lin, T\. 2026\.Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO\.arXiv:2505\.11595\.
- Choi et al\. \(2026\)Choi, K\.; Saha, D\.; Kim, W\.; Agarwal, A\.; and Dwivedi, R\. 2026\.GOPO: Policy Optimization using Ranked Rewards\.arXiv:2602\.03876\.
- Dave et al\. \(2026\)Dave, R\.; Dubey, V\.; Deoghare, S\.; and Mishra, S\. 2026\.Selector\-Guided Autonomous Curriculum for One\-Shot Reinforcement Learning from Verifiable Rewards\.arXiv:2605\.01823\.
- Deng et al\. \(2026\)Deng, J\.; Wei, Z\.; Pang, L\.; Wu, J\.; Xu, S\.; Duan, Z\.; and Shen, H\. 2026\.Latent\-GRPO: Group Relative Policy Optimization for Latent Reasoning\.arXiv:2604\.27998\.
- Fontana et al\. \(2026\)Fontana, A\.; Simoni, M\.; Rossolini, G\.; Saracino, A\.; and Mori, P\. 2026\.On the Hidden Objective Biases of Group\-based Reinforcement Learning\.arXiv:2601\.05002\.
- Fu et al\. \(2026\)Fu, J\.; Zhao, X\.; Yao, C\.; Wang, H\.; Han, Q\.; and Xiao, Y\. 2026\.Reward Shaping to Mitigate Reward Hacking in RLHF\.arXiv:2502\.18770\.
- Ge et al\. \(2026\)Ge, C\.; Yin, C\. H\.; Liang, H\.; and Zhang, J\. 2026\.Why GRPO Needs Normalization: A Local\-Curvature Perspective on Adaptive Gradients\.arXiv:2601\.23135\.
- Girgis et al\. \(2026\)Girgis, R\.; de Schaetzen, R\.; Rowe, L\.; Robitaille, A\.; Pal, C\.; and Paull, L\. 2026\.Constrained Group Relative Policy Optimization\.arXiv:2602\.05863\.
- He et al\. \(2026a\)He, L\.; Qu, Q\.; Zhao, H\.; Wan, S\.; Wang, D\.; Yao, L\.; and Liu, T\. 2026a\.Unifying Stable Optimization and Reference Regularization in RLHF\.arXiv:2602\.11523\.
- He et al\. \(2026b\)He, X\.; Sun, Q\.; Cheng, A\.; Li, X\.; Ji, X\.; Lu, H\.; Huang, R\.; and Hu, Q\. 2026b\.Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation\.arXiv:2605\.21125\.
- He et al\. \(2025\)He, Z\.; Liang, T\.; Xu, J\.; Liu, Q\.; Chen, X\.; Wang, Y\.; Song, L\.; Yu, D\.; Liang, Z\.; Wang, W\.; Zhang, Z\.; Wang, R\.; Tu, Z\.; Mi, H\.; and Yu, D\. 2025\.DeepMath\-103K: A Large\-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning\.arXiv:2504\.11456\.
- Hu et al\. \(2025\)Hu, J\.; Liu, J\. K\.; Xu, H\.; and Shen, W\. 2025\.REINFORCE\+\+: Stabilizing Critic\-Free Policy Optimization with Global Advantage Normalization\.arXiv:2501\.03262\.
- Huang, Nguyen, and Zimmer \(2025\)Huang, B\.; Nguyen, T\.; and Zimmer, M\. 2025\.Tree\-OPO: Off\-policy Monte Carlo Tree\-Guided Advantage Optimization for Multistep Reasoning\.arXiv:2509\.09284\.
- Jain et al\. \(2024\)Jain, N\.; Han, K\.; Gu, A\.; Li, W\.\-D\.; Yan, F\.; Zhang, T\.; Wang, S\.; Solar\-Lezama, A\.; Sen, K\.; and Stoica, I\. 2024\.LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.arXiv:2403\.07974\.
- Jia et al\. \(2025\)Jia, W\.; Lu, J\.; Yu, H\.; Wang, S\.; Tang, G\.; Wang, A\.\-L\.; Yin, W\.; Yang, D\.; Nie, Y\.; Shan, B\.; Feng, H\.; Li, I\.; Yang, K\.; Wang, H\.; Tang, J\.; Fu, T\.; Jin, C\.; Feng, C\.; Lv, X\.; and Huang, C\. 2025\.MEML\-GRPO: Heterogeneous Multi\-Expert Mutual Learning for RLVR Advancement\.arXiv:2508\.09670\.
- Kim and Kang \(2026\)Kim, G\.; and Kang, S\. 2026\.CPC\-GRPO: Answer\-Free Reinforcement Learning with Cross\-Prompt Consensus Rewards\.In Liakata, M\.; Moreira, V\. P\.; Zhang, J\.; and Jurgens, D\., eds\.,*Findings of the Association for Computational Linguistics: ACL 2026*, 29733–29748\. San Diego, California, United States: Association for Computational Linguistics\.ISBN 979\-8\-89176\-395\-1\.
- Kim \(2026\)Kim, Y\. 2026\.MC\-GRPO: Median\-Centered Group Relative Policy Optimization for Small\-Rollout Reinforcement Learning\.arXiv:2601\.22582\.
- Kingma and Ba \(2017\)Kingma, D\. P\.; and Ba, J\. 2017\.Adam: A Method for Stochastic Optimization\.arXiv:1412\.6980\.
- Le, Van, and Le \(2026\)Le, T\.; Van, L\. N\.; and Le, T\. 2026\.Sharpness\-Guided Group Relative Policy Optimization via Probability Shaping\.arXiv:2511\.00066\.
- Le et al\. \(2026\)Le, T\.\-L\. V\.; Jeon, M\.; Vu, K\.; Lai, V\.; and Yang, E\. 2026\.No Prompt Left Behind: Exploiting Zero\-Variance Prompts in LLM Reinforcement Learning via Entropy\-Guided Advantage Shaping\.arXiv:2509\.21880\.
- Li et al\. \(2025\)Li, X\.; Li, Z\.; Kosuga, Y\.; and Bian, V\. 2025\.Optimizing Safe and Aligned Language Generation: A Multi\-Objective GRPO Approach\.arXiv:2503\.21819\.
- Liu and Xiao \(2026\)Liu, H\.; and Xiao, L\. 2026\.RE\-GRPO: Leveraging hard negative cases through large language model guided self training\.*Neurocomputing*, 669: 132543\.
- Liu et al\. \(2025a\)Liu, K\.; Liu, J\. K\.; Chen, M\.; and Liu, Y\. 2025a\.Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization\.arXiv:2510\.01555\.
- Liu et al\. \(2026\)Liu, S\.\-Y\.; Dong, X\.; Lu, X\.; Diao, S\.; Belcak, P\.; Liu, M\.; Chen, M\.\-H\.; Yin, H\.; Wang, Y\.\-C\. F\.; Cheng, K\.\-T\.; Choi, Y\.; Kautz, J\.; and Molchanov, P\. 2026\.GDPO: Group reward\-Decoupled Normalization Policy Optimization for Multi\-reward RL Optimization\.arXiv:2601\.05242\.
- Liu et al\. \(2025b\)Liu, T\.; Qin, Z\.; Wu, J\.; Shen, J\.; Khalman, M\.; Joshi, R\.; Zhao, Y\.; Saleh, M\.; Baumgartner, S\.; Liu, J\.; Liu, P\. J\.; and Wang, X\. 2025b\.LiPO: Listwise Preference Optimization through Learning\-to\-Rank\.In Chiruzzo, L\.; Ritter, A\.; and Wang, L\., eds\.,*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, 2404–2420\. Albuquerque, New Mexico: Association for Computational Linguistics\.ISBN 979\-8\-89176\-189\-6\.
- Liu et al\. \(2025c\)Liu, Z\.; Chen, C\.; Li, W\.; Qi, P\.; Pang, T\.; Du, C\.; Lee, W\. S\.; and Lin, M\. 2025c\.Understanding R1\-Zero\-Like Training: A Critical Perspective\.arXiv:2503\.20783\.
- Loshchilov and Hutter \(2019\)Loshchilov, I\.; and Hutter, F\. 2019\.Decoupled Weight Decay Regularization\.arXiv:1711\.05101\.
- Lu et al\. \(2025\)Lu, S\.; Wang, H\.; Chen, Z\.; and Tang, Y\. 2025\.URPO: A Unified Reward & Policy Optimization Framework for Large Language Models\.arXiv:2507\.17515\.
- MAA Communications \(2024\)MAA Communications\. 2024\.2024\-25 AIME Thresholds Are Available\.https://maa\.org/news/aime\-thresholds\-are\-available/\.
- Mansouri et al\. \(2026\)Mansouri, O\. E\.; Izzati, F\. A\.; Seddik, M\. E\. A\.; and Lahlou, S\. 2026\.Noise\-corrected GRPO: From Noisy Rewards to Unbiased Gradients\.arXiv:2510\.18924\.
- Mishra, Chakraborty, and Kapusuzoglu \(2026\)Mishra, A\.; Chakraborty, S\.; and Kapusuzoglu, B\. 2026\.On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse\.arXiv:2606\.29238\.
- Mitsuhashi et al\. \(2026\)Mitsuhashi, R\.; Chen, P\.; Tseng, I\.; Cekinmez, J\.; and Wu, A\. J\. 2026\.Quantifying Empirical Compute\-Supervision Tradeoffs in RLVR\.arXiv:2605\.25252\.
- Montero, Jurczyk, and Dhingra \(2026\)Montero, I\.; Jurczyk, T\.; and Dhingra, B\. 2026\.RVPO: Risk\-Sensitive Alignment via Variance Regularization\.arXiv:2605\.05750\.
- Mroueh \(2025\)Mroueh, Y\. 2025\.Reinforcement Learning with Verifiable Rewards: GRPO’s Effective Loss, Dynamics, and Success Amplification\.arXiv:2503\.06639\.
- Mroueh et al\. \(2025\)Mroueh, Y\.; Dupuis, N\.; Belgodere, B\.; Nitsure, A\.; Rigotti, M\.; Greenewald, K\.; Navratil, J\.; Ross, J\.; and Rios, J\. 2025\.Revisiting Group Relative Policy Optimization: Insights into On\-Policy and Off\-Policy Training\.arXiv:2505\.22257\.
- Mundada et al\. \(2026\)Mundada, G\.; Huang, Z\.; Surana, R\.; Yu, S\.; Zhang, J\. Y\.; Li, X\.; Yu, T\.; Yao, L\.; Shang, J\.; McAuley, J\.; and Wu, J\. 2026\.WS\-GRPO: Weakly\-Supervised Group\-Relative Policy Optimization for Rollout\-Efficient Reasoning\.arXiv:2602\.17025\.
- Nan et al\. \(2025\)Nan, G\.; Chen, S\.; Huang, J\.; Lu, M\.; Wang, D\.; Xie, C\.; Xiong, W\.; Zeng, X\.; Zhou, Q\.; Li, Y\.; and Xu, X\. 2025\.NGRPO: Negative\-enhanced Group Relative Policy Optimization\.arXiv:2509\.18851\.
- Pang, Luo, and Jin \(2026\)Pang, L\.; Luo, J\.; and Jin, R\. 2026\.TIC\-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback\.arXiv:2508\.02833\.
- Pikus, Tiwari, and Ye \(2025\)Pikus, B\.; Tiwari, P\. R\.; and Ye, B\. 2025\.Hard Examples Are All You Need: Maximizing GRPO Post\-Training Under Annotation Budgets\.arXiv:2508\.14094\.
- Qu et al\. \(2026\)Qu, Y\.; Wang, Q\.; Mao, Y\.; Zou, H\.; Jiang, Y\.; Li, Y\.; Xu, W\.; Cai, L\.; Liu, W\.; Bai, C\.; Yang, K\.; Chen, Y\.; Yang, S\.; and Ji, X\. 2026\.Listwise Policy Optimization: Group\-based RLVR as Target\-Projection on the LLM Response Simplex\.arXiv:2605\.06139\.
- Reddy, Yang, and Zhai \(2026\)Reddy, S\.; Yang, K\.; and Zhai, C\. 2026\.BiasGRPO: Stabilizing Bias Mitigation in High\-Variance Reward Landscapes via Group\-Relative Policy Optimization\.arXiv:2606\.04807\.
- Salmani\-Zarchi et al\. \(2026\)Salmani\-Zarchi, M\. M\.; Rahimi, Z\.; Faili, H\.; and Dousti, M\. J\. 2026\.MDP\-GRPO: Stabilized Group Relative Policy Optimization for Multi\-Constraint Instruction Following\.arXiv:2606\.06058\.
- Sane \(2025\)Sane, S\. 2025\.Hybrid Group Relative Policy Optimization: A Multi\-Sample Approach to Enhancing Policy Optimization\.arXiv:2502\.01652\.
- Shao et al\. \(2024\)Shao, Z\.; Wang, P\.; Zhu, Q\.; Xu, R\.; Song, J\.; Bi, X\.; Zhang, H\.; Zhang, M\.; Li, Y\. K\.; Wu, Y\.; and Guo, D\. 2024\.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv:2402\.03300\.
- Sun et al\. \(2026\)Sun, H\.; Wang, J\.; Song, Y\.; Lu, Y\.; Fang, B\.; Luo, Y\.; Yin, J\.; Zeng, P\.; Zhang, M\.; Zhang, T\.; Wang, X\.; and Lu, S\. 2026\.Power Reinforcement Post\-Training of Text\-to\-Image Models with Super\-Linear Advantage Shaping\.arXiv:2605\.10937\.
- Tan et al\. \(2026\)Tan, Z\.; Yu, Z\.; Lin, B\.; Geng, Z\.; Geng, H\.; Zhang, Y\.; Zhang, M\.; Chen, Y\.; Hu, S\.; Yin, Z\.; Zhang, C\.; and Bai, L\. 2026\.PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization\.arXiv:2603\.26535\.
- Wan et al\. \(2026\)Wan, X\.; Wang, Y\.; Huang, W\.; and Sun, M\. 2026\.Buffer Matters: Unleashing the Power of Off\-Policy Reinforcement Learning in Large Language Model Reasoning\.arXiv:2602\.20722\.
- Wang et al\. \(2026a\)Wang, J\.; Peters, H\.; Butt, A\. A\.; Hashemi, N\.; Hashemi, A\.; Ghari, P\. M\.; Hoover, J\.; Rae, J\.; and Dehghani, M\. 2026a\.Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment\.arXiv:2603\.10009\.
- Wang et al\. \(2026b\)Wang, J\.; Zhu, L\.; Weng, T\.; Wu, S\.\-L\.; Tan, H\.; Chen, J\.; Tao, C\.; Bai, H\.; Hou, L\.; Shang, L\.; and Zhang, X\.\-P\. 2026b\.GRPO\-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning\.arXiv:2604\.20659\.
- Wang et al\. \(2026c\)Wang, P\.\-Y\.; Li, Z\.; Xu, T\.; Yang, B\.; Liu, T\.\-S\.; Wang, C\.; Chen, X\.\-H\.; Li, Y\.\-C\.; Yang, T\.; Chen, C\.; and Yu, Y\. 2026c\.Off\-Policy Value\-Based Reinforcement Learning for Large Language Models\.arXiv:2603\.23355\.
- Wang et al\. \(2025\)Wang, Y\.; Zhao, J\.; Zhao, C\.; Guan, S\.; Penn, G\.; and Liu, S\. 2025\.λ\\lambda\-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences\.arXiv:2510\.06870\.
- Wang et al\. \(2026d\)Wang, Z\.; Lou, X\.; Wu, M\.; Wen, Z\.; and Zhang, J\. 2026d\.Calibration\-Aware Policy Optimization for Reasoning LLMs\.In Liakata, M\.; Moreira, V\. P\.; Zhang, J\.; and Jurgens, D\., eds\.,*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 18375–18390\. San Diego, California, United States: Association for Computational Linguistics\.ISBN 979\-8\-89176\-390\-6\.
- Wen et al\. \(2025\)Wen, X\.; Liu, Z\.; Zheng, S\.; Ye, S\.; Wu, Z\.; Wang, Y\.; Xu, Z\.; Liang, X\.; Li, J\.; Miao, Z\.; Bian, J\.; and Yang, M\. 2025\.Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs\.arXiv:2506\.14245\.
- Xiao, Zhang, and Cao \(2025\)Xiao, C\.; Zhang, M\.; and Cao, Y\. 2025\.BNPO: Beta Normalization Policy Optimization\.arXiv:2506\.02864\.
- Xu et al\. \(2024\)Xu, S\.; Fu, W\.; Gao, J\.; Ye, W\.; Liu, W\.; Mei, Z\.; Wang, G\.; Yu, C\.; and Wu, Y\. 2024\.Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study\.In Salakhutdinov, R\.; Kolter, Z\.; Heller, K\.; Weller, A\.; Oliver, N\.; Scarlett, J\.; and Berkenkamp, F\., eds\.,*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, 54983–54998\. PMLR\.
- Xu and Ding \(2025\)Xu, Z\.; and Ding, Z\. 2025\.Single\-stream Policy Optimization\.arXiv:2509\.13232\.
- Yang et al\. \(2026\)Yang, F\.; Chen, Z\.; Wang, X\.; Lu, X\.; Chai, J\.; Yin, G\.; Lin, W\.; Ma, S\.; Zhuang, F\.; Wang, D\.; Yang, Y\.; Li, J\.; and Ban, Y\. 2026\.Your Group\-Relative Advantage Is Biased\.arXiv:2601\.08521\.
- Yao et al\. \(2026\)Yao, J\.; Wang, R\.; Bai, H\.; and Zhang, T\. 2026\.Future\-KL Regularized GRPO: Process\-Level Credit Assignment fromff\-Divergence Regularization\.arXiv:2601\.10201\.
- Yu et al\. \(2025\)Yu, Q\.; Zhang, Z\.; Zhu, R\.; Yuan, Y\.; Zuo, X\.; Yue, Y\.; Dai, W\.; Fan, T\.; Liu, G\.; Liu, L\.; Liu, X\.; Lin, H\.; Lin, Z\.; Ma, B\.; Sheng, G\.; Tong, Y\.; Zhang, C\.; Zhang, M\.; Zhang, W\.; Zhu, H\.; Zhu, J\.; Chen, J\.; Chen, J\.; Wang, C\.; Yu, H\.; Song, Y\.; Wei, X\.; Zhou, H\.; Liu, J\.; Ma, W\.\-Y\.; Zhang, Y\.\-Q\.; Yan, L\.; Qiao, M\.; Wu, Y\.; and Wang, M\. 2025\.DAPO: An Open\-Source LLM Reinforcement Learning System at Scale\.arXiv:2503\.14476\.
- Yu et al\. \(2026a\)Yu, S\.; Li, L\.; Zhao, W\.; and Yang, Z\. 2026a\.EP\-GRPO: Entropy\-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance\.arXiv:2605\.04960\.
- Yu et al\. \(2026b\)Yu, S\.; Li, L\.; Zhao, W\.; and Yang, Z\. 2026b\.ERPO: Token\-Level Entropy\-Regulated Policy Optimization for Large Reasoning Models\.arXiv:2603\.28204\.
- Yue et al\. \(2025\)Yue, C\.; Dong, C\.; Gao, Y\.; He, H\.; Chai, J\.; Yin, G\.; and Lin, W\. 2025\.Promoting Efficient Reasoning with Verifiable Stepwise Reward\.arXiv:2508\.10293\.
- Zeng et al\. \(2026\)Zeng, G\.; Zhou, Z\.; Arora, D\.; and Zanette, A\. 2026\.Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards\.arXiv:2511\.03710\.
- Zhai et al\. \(2023\)Zhai, Y\.; Zhang, H\.; Lei, Y\.; Yu, Y\.; Xu, K\.; Feng, D\.; Ding, B\.; and Wang, H\. 2023\.Uncertainty\-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles\.arXiv:2401\.00243\.
- Zhang et al\. \(2025\)Zhang, K\.; Hong, Y\.; Bao, J\.; Jiang, H\.; Song, Y\.; Hong, D\.; and Xiong, H\. 2025\.GVPO: Group Variance Policy Optimization for Large Language Model Post\-Training\.arXiv:2504\.19599\.
- Zheng et al\. \(2025\)Zheng, C\.; Liu, S\.; Li, M\.; Chen, X\.\-H\.; Yu, B\.; Gao, C\.; Dang, K\.; Liu, Y\.; Men, R\.; Yang, A\.; Zhou, J\.; and Lin, J\. 2025\.Group Sequence Policy Optimization\.arXiv:2507\.18071\.
- Zheng et al\. \(2023\)Zheng, R\.; Dou, S\.; Gao, S\.; Hua, Y\.; Shen, W\.; Wang, B\.; Liu, Y\.; Jin, S\.; Liu, Q\.; Zhou, Y\.; Xiong, L\.; Chen, L\.; Xi, Z\.; Xu, N\.; Lai, W\.; Zhu, M\.; Chang, C\.; Yin, Z\.; Weng, R\.; Cheng, W\.; Huang, H\.; Sun, T\.; Yan, H\.; Gui, T\.; Zhang, Q\.; Qiu, X\.; and Huang, X\. 2023\.Secrets of RLHF in Large Language Models Part I: PPO\.arXiv:2307\.04964\.
- Zhong et al\. \(2026\)Zhong, H\.; Zhai, J\.; Song, L\.; Bian, J\.; Liu, Q\.; and Tan, T\. 2026\.RC\-GRPO: Reward\-Conditioned Group Relative Policy Optimization for Multi\-Turn Tool Calling Agents\.arXiv:2602\.03025\.
- Zhou et al\. \(2026\)Zhou, H\.; Ye, K\.; Xu, E\.; Zhu, J\.; Yang, Y\.; Gong, S\.; and Shi, C\. 2026\.Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U\-Statistic\.arXiv:2603\.01162\.