用于可验证奖励离策略强化学习的吉布斯间隙策略优化

# Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Source: [https://arxiv.org/html/2610.20814](https://arxiv.org/html/2610.20814)
###### Abstract

Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy\. Correcting this mismatch with learner\-to\-behavior importance weights can introduce high variance for long responses\. We introduce Gibbs\-Gap Policy Optimization \(GGPO\), which minimizes a log\-mean\-exp contrast of KL\-adjusted rewards among responses to the same prompt, without learner\-to\-behavior importance weights\. We derive GGPO from KL\-regularized policy optimization: on policy, its population loss is exactly the KL\-regularized optimality gap, and under suitable coverage the learner’s single\-response expected reward is at least the behavior policy’s best\-of\-KKreward, minus terms for the raw finite\-group loss, regularization, and finite training data\. We characterize reference refresh through exact Gibbs\-target composition and a finite\-data final\-policy reward bound under local reference control and tokenwise behavior mismatch\. This refresh is empirically important: in an ablation, fixed\-reference GGPO scores32\.9632\.96on the five\-benchmark problem\-weighted average, versus46\.5846\.58with the default refresh schedule\. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training\-\-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows\.111Code is available at[https://github\.com/thunderous77/GGPO](https://github.com/thunderous77/GGPO)\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) improves language\-model reasoning by using automatic checks to evaluate task outcomes\([Guo et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib16)\)\. In group\-based methods, the model generates several responses to each prompt, and differences in their rewards provide a learning signal\([Shao et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib40);[Yu et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib47)\)\. These groups also suggest a natural performance target\. If selecting the highest\-reward response from a group improves on drawing a single response, we would like to train a learner whose individual responses approach the reward of that selection procedure\. This motivation also underlies reward\-ranked fine\-tuning methods such as RAFT\([Dong et al\., 2023](https://arxiv.org/html/2610.20814#bib.bib11)\)and reasoning variants such as RAFT\+\+\([Xiong et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib45)\)\. In practice, however, the policy that generates the group may differ from the learner being trained\. During asynchronous generation, responses can arrive from an earlier policy version; even within a rollout batch, sequential learner updates can make the remaining responses stale\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib33);[Zheng et al\., 2025a](https://arxiv.org/html/2610.20814#bib.bib51)\)\. We refer to the policy generating these responses as the behavior policyπbeh\\pi\_\{\\mathrm\{beh\}\}, and to the policy being trained as the learnerπθ\\pi\_\{\\theta\}\. The distinction is also relevant in offline training, where sampled responses remain fixed while the learner continues to change\. Moreover, synchronizing policy parameters does not necessarily eliminate the mismatch: training and inference can use different implementations or numerical precision\([MiniMax et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib30);[Zhong et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib53)\)\.

A standard way to account for this mismatch is importance sampling, which weights behavior\-policy data by the learner\-to\-behavior likelihood ratioπθ/πbeh\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{beh\}\}\. For an autoregressive response, this ratio is a product of conditional token ratios\. Small differences at individual token positions can therefore accumulate over a long response, producing highly variable weights and making the update sensitive to a few sampled responses\. Practical methods control these contributions through clipping, truncation, or masking\([Schulman et al\., 2017](https://arxiv.org/html/2610.20814#bib.bib39);[Yu et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib47);[Zhong et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib53);[Ling Team et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib26);[Zheng et al\., 2025b](https://arxiv.org/html/2610.20814#bib.bib52)\)\. These choices involve both where to apply the correction, such as at the token or response level, and how strongly to limit it\. They can reduce variance, but also introduce bias\. This tradeoff motivates an alternative: learn directly from behavior\-policy response groups through an objective that retains a quantitative connection to the learner’s reward, without requiring learner\-to\-behavior importance weights in the update\.

We introduce Gibbs\-Gap Policy Optimization \(GGPO\), which learns from contrasts in KL\-adjusted rewards among responses to the same prompt, without learner\-to\-behavior importance weights\. Under bounded coverage, its finite\-group loss controls the gap between the learner’s single\-response expected reward and the reward obtained by best\-of\-KKselection from the behavior policy\. Thus the response groups serve both as training data and as a reward benchmark for the learner\. Groups of orderKKsuffice for this comparison, and a uniform generalization bound extends it to finite data\.

The construction follows from KL\-regularized duality, which identifies a Gibbs policy whose adjusted reward is constant across responses\. GGPO measures departures from this condition through contrasts within each group\. On policy, its population loss equals the regularized optimality gap\. For finite groups, we establish the reward comparison directly, without requiring accurate estimation of that population loss\.

A fixed reference defines a single Gibbs target\. Refreshing the reference to the current learner changes that target, allowing successive rounds to build on earlier progress\. We characterize how exact targets compose across rounds and bound the reward of averaged outputs when updates are imperfect, under coverage and bounded adjusted\-reward spread\. Under binary rewards and local reference control, a finite\-data guarantee bounds the final policy’s reward using aggregate tokenwise mismatch and achieved fitting accuracy\. Under its assumptions, the expected reward of one response from the trained policy can approach that of selecting the best ofKKresponses from the initial policy\. Section[5](https://arxiv.org/html/2610.20814#S5)summarizes this guarantee; Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1)gives the precise result\.

GGPO builds on earlier work on off\-policy consistency\. AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\)minimizes the variance of the same adjusted reward and identifies the same regularized optimum under common support\. Although the two objectives share this target, they measure departures from it differently\. We compare their objectives and guarantees in Section[2](https://arxiv.org/html/2610.20814#S2)and evaluate GGPO against GRPO and AGRO under training–inference mismatch, controlled policy staleness, and offline training\.

Our contributions are:

- •An off\-policy group objective with a direct reward guarantee\.GGPO minimizes the log\-mean\-exp contrast of KL\-adjusted rewards within behavior\-policy response groups, without learner\-to\-behavior importance weights\. On policy, its population value is exactly the KL\-regularized optimality gap\. Off policy, coverage bounds that gap\. Its raw finite\-group loss bounds the gap between the learner’s single\-response expected reward and the behavior policy’s best\-of\-KKreward, with explicit terms for the loss, regularization, reference–behavior mismatch, and group size\. A uniform empirical bound and oracle inequality extend this comparison to finite data\.
- •A characterization of periodic reference refresh\.When each round exactly fits its Gibbs target,TTrefresh rounds at temperatureβ\\betaimplement one Gibbs update at temperatureβ/T\\beta/T\. This explains how refresh changes the target across rounds\. For imperfect updates, a finite\-data guarantee controls the final policy reward using aggregate tokenwise mismatch and achieved inner optimization accuracy\. An ablation shows that periodic refresh substantially improves GGPO under the default training setting\.
- •Evaluation across behavior–learner mismatch regimes\.On five mathematical\-reasoning benchmarks and two model sizes, GGPO outperforms GRPO and its closest objective\-level comparator AGRO under training–inference mismatch\. It maintains its accuracy as the synchronization delay grows fromN=8N=8toN=128N=128, while GRPO and AGRO decline\. AtN=128N=128, token\-level TIS improves GRPO, while GGPO achieves higher accuracy under the same synchronization delay\. On a fixed offline rollout pool, GGPO has the highest MATH\-500 accuracy at every checkpoint from 160 updates onward\.

## 2Related work

#### KL\-regularized and direct policy objectives\.

Gibbs variational identities connect KL regularization to exponential tilting\([Richemond and Maginnis, 2017](https://arxiv.org/html/2610.20814#bib.bib35);[Geist et al\., 2019](https://arxiv.org/html/2610.20814#bib.bib15)\)\. These identities also provide a starting point for direct policy objectives\. Objectives derived from related policy identities include DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2610.20814#bib.bib34)\), InfoNCA/NCA\([Chen et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib8)\), REBEL\([Gao et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib14)\), andA∗A^\{\*\}\-PO\([Brantley et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib7)\)\. ConSPO\([Zhang et al\., 2026a](https://arxiv.org/html/2610.20814#bib.bib49)\)contrasts verified positive and negative responses using length\-normalized sequence log probabilities in an InfoNCE\-style objective\. GGPO instead applies a log\-mean\-exp gap to KL\-adjusted rewards\. In this construction, the behavior policy generates the samples, while a distinct reference policy provides the KL anchor\. InfoNCA uses learner\-to\-reference logits with reward\-softmax labels; GGPO instead minimizes a KL divergence from uniform group weights to an adjusted\-reward tilt\.

#### Group RLVR and behavior–learner mismatch\.

GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib40)\)uses within\-group advantages in a PPO\([Schulman et al\., 2017](https://arxiv.org/html/2610.20814#bib.bib39)\)surrogate\. Building on this group\-based approach, DAPO\([Yu et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib47)\), Dr\.GRPO\([Liu et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib27)\), and GSPO\([Zheng et al\., 2025b](https://arxiv.org/html/2610.20814#bib.bib52)\)modify sampling, normalization, or PPO\-style ratio aggregation\. MaxRL\([Tajwar et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib42)\)derives success\-normalized on\-policy updates from a maximum\-likelihood formulation\. Its centered estimator is related to a special binary\-reward limit of GGPO’s update coefficients, detailed in Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4)\.

When the behavior policy differs from the learner, the update must also account for how the responses were sampled\. Under this mismatch, asynchronous RLHF\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib33)\)studies staleness and IMPALA\([Espeholt et al\., 2018](https://arxiv.org/html/2610.20814#bib.bib12)\)uses V\-trace\. In LLM training,[Zhong et al\. \(2026\)](https://arxiv.org/html/2610.20814#bib.bib53)studies truncated old\-trainer\-to\-rollout correction weights alongside a PPO ratio whose denominator is the old trainer policy\. This correction and PPO denominator serve different roles from GGPO’s KL reference\. Other methods mask ratio\-based contributions\([Ling Team et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib26);[Li et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib24)\)or restructure learner\-to\-behavior ratios\([Zhang et al\., 2026b](https://arxiv.org/html/2610.20814#bib.bib50)\)\. These corrections trade importance\-weight variance against clipping, truncation, or masking bias\([Huang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib19)\)\. GGPO does not evaluate learner\-to\-behavior correction ratios, while its population guarantee depends on behavior coverage\.

In concurrent work,[Marek and Ryabinin \(2026\)](https://arxiv.org/html/2610.20814#bib.bib29)center token\-level policy scores under the sampling distribution to remove an additional drift term induced by training–inference mismatch\. This additive correction can be combined with importance weighting\. GGPO derives a response\-group objective from KL\-regularized duality and contrasts KL\-adjusted rewards using a separately maintained reference\. Its update uses likelihoods of sampled responses under the learner and reference, rather than the sampler’s next\-token distribution\.

#### Objectives without learner\-to\-behavior importance weighting\.

PCL/Trust\-PCL\([Nachum et al\., 2017](https://arxiv.org/html/2610.20814#bib.bib31);[Nachum et al\., 2018](https://arxiv.org/html/2610.20814#bib.bib32)\)optimize squared soft consistency residuals on off\-policy trajectories\. In language\-model training, Kimi k1\.5\([Kimi Team et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib20)\)uses squared Gibbs\-consistency regression with an iteratively updated reference\. Contrastive Policy Gradient\([Flet\-Berliac et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib13)\)uses a pairwise quadratic objective under fixed sampling distributions\. OAPL\([Ritter et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib37)\)regresses an estimated optimal advantage using the behavior policy as its KL anchor; its learner\-to\-behavior log ratio is a regression term, rather than an importance weight\. AsymRE\([Arnal et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib3)\)scales positive and negative feedback asymmetrically\. DRO\([Richemond et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib36)\)applies a squared penalty to the optimality residual while jointly estimating the value offset\. These methods show that off\-policy learning can be approached through consistency conditions, without learner\-to\-behavior correction ratios\. GGPO follows this general approach, using a contrastive loss whose value can be related directly to reward\.

#### Relation to AGRO\.

AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\)is our closest objective\-level comparator and an earlier example with an off\-policy consistency guarantee\. It minimizes the behavior variance of the same KL\-adjusted reward\. Theorem 2 of[Tang et al\. \(2026\)](https://arxiv.org/html/2610.20814#bib.bib43)establishes that the regularized optimal policy is the unique global minimizer under common support, shared by GGPO\.

GGPO’s distinction is the direct alignment of its loss value with the KL\-regularized objective\. Its exponential form is derived from the exact duality residual\. This gives a direct interpretation of the loss value: at the population level and when behavior equals learner, the loss equals the regularized optimality gap for every positive candidate policy\. Off policy, the connection becomes a coverage\-dependent upper bound\. AGRO’s variance loss, scaled by1/β1/\\beta, is the local quadratic approximation to this population contrastive loss when adjusted\-reward spread is small relative toβ\\beta\(Appendix[A\.6](https://arxiv.org/html/2610.20814#A1.SS6)\)\. GGPO’s finite\-group loss additionally admits a direct best\-of\-KKreward bound and an empirical generalization bound\. The finite\-group variance comparison also yields a reward bound in terms of AGRO’s population variance objective \(Eq\. \([39](https://arxiv.org/html/2610.20814#A1.E39)\)\)\. Appendices[A\.4](https://arxiv.org/html/2610.20814#A1.SS4)and[A\.6](https://arxiv.org/html/2610.20814#A1.SS6)give the gradient and finite\-group weighting comparisons\. The experiments compare the practical consequences under matched settings\.

We also acknowledge related unpublished work by Audrey Huang and collaborators on coverage collapse and reference re\-anchoring in policy optimization after pre\-training \(private communication with Nan Jiang, October 2026\)\.

## 3From KL regularization to a contrastive loss

We treat a complete response as one action in a contextual bandit\. Fix a promptxx, a finite action space𝒜\\mathcal\{A\}, rewardr⁡\(x,a\)r\(x,a\), temperatureβ\>0\\beta\>0, and a reference policyπref\\pi\_\{\\mathrm\{ref\}\}with positive support\. Define

Vβ\(π;x\)=𝔼a∼πr\(x,a\)−βDKL\(π∥πref\)\.V\_\{\\beta\}\(\\pi;x\)=\\mathbb\{E\}\_\{a\\sim\\pi\}r\(x,a\)\-\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{\\mathrm\{ref\}\}\)\.\(1\)LetVβ∗​\(x\)=maxπ⁡Vβ​\(π,x\)V\_\{\\beta\}^\{\*\}\(x\)=\\max\_\{\\pi\}V\_\{\\beta\}\(\\pi;x\)andπ∗\(⋅∣x\)=argmaxπVβ\(π;x\)\\pi^\{\*\}\(\\cdot\\mid x\)=\\arg\\max\_\{\\pi\}V\_\{\\beta\}\(\\pi;x\)over all policiesπ\\pion𝒜\\mathcal\{A\}\. The unique maximizer is the Gibbs policy

π∗​\(a∣x\)=πref​\(a∣x\)​er⁡\(x,a\)/β𝔼a′∼πref​er⁡\(x,a′\)/β\.\\pi^\{\*\}\(a\\mid x\)=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(a\\mid x\)e^\{r\(x,a\)/\\beta\}\}\{\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\pi\_\{\\mathrm\{ref\}\}\}e^\{r\(x,a^\{\\prime\}\)/\\beta\}\}\.\(2\)The Gibbs variational identity\([Richemond and Maginnis, 2017](https://arxiv.org/html/2610.20814#bib.bib35)\)gives a useful way to recognize this optimum\. Whenπθ\\pi\_\{\\theta\}is the Gibbs policy, its adjusted rewardRθ​\(x,a\)=r⁡\(x,a\)−β​log⁡\[πθ​\(a∣x\)/πref​\(a∣x\)\]R\_\{\\theta\}\(x,a\)=r\(x,a\)\-\\beta\\log\[\\pi\_\{\\theta\}\(a\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\\mid x\)\]is constant across responses to the same prompt\. This suggests learning by comparing adjusted rewards: if they differ among sampled responses, the learner has not yet satisfied this optimality condition\.

We can test this condition using behavior\-policy samples, even when the behavior policy differs from the learner\. To quantify the differences, define the population contrastive loss

𝒥beh​\(πθ,x\)=β​log⁡𝔼a∼πbeh​eRθ​\(x,a\)/β−𝔼a∼πbeh​Rθ​\(x,a\)\.\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\(\\pi\_\{\\theta\};x\)=\\beta\\log\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\}e^\{R\_\{\\theta\}\(x,a\)/\\beta\}\-\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\}R\_\{\\theta\}\(x,a\)\.\(3\)The following proposition states its dual interpretation and its relation to the original KL\-regularized objective\.

###### Proposition 1\(Population duality and regularized optimality\)\.

Supposeπbeh\\pi\_\{\\mathrm\{beh\}\},πθ\\pi\_\{\\theta\}, andπref\\pi\_\{\\mathrm\{ref\}\}are positive on𝒜\\mathcal\{A\}\. Defineh⁡\(z\)=ez−z−1h\(z\)=e^\{z\}\-z\-1andqθ​\(a∣x\)∝πbeh​\(a∣x\)​eRθ​\(x,a\)/βq\_\{\\theta\}\(a\\mid x\)\\propto\\pi\_\{\\mathrm\{beh\}\}\(a\\mid x\)e^\{R\_\{\\theta\}\(x,a\)/\\beta\}\. Then

𝒥beh\(πθ;x\)=minν∈ℝβ𝔼a∼πbehh\(Rθ​\(x,a\)−νβ\)=βDKL\(πbeh∥qθ\)≥0\.\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\(\\pi\_\{\\theta\};x\)=\\min\_\{\\nu\\in\\mathbb\{R\}\}\\beta\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\}h\\\!\\left\(\\frac\{R\_\{\\theta\}\(x,a\)\-\\nu\}\{\\beta\}\\right\)=\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{beh\}\}\\\|q\_\{\\theta\}\)\\geq 0\.\(4\)WithCx=maxa⁡πθ​\(a∣x\)/πbeh​\(a∣x\)C\_\{x\}=\\max\_\{a\}\\pi\_\{\\theta\}\(a\\mid x\)/\\pi\_\{\\mathrm\{beh\}\}\(a\\mid x\),

Vβ∗\(x\)−Vβ\(πθ;x\)=βDKL\(πθ∥π∗\)≤Cx𝒥beh\(πθ;x\)\.V\_\{\\beta\}^\{\*\}\(x\)\-V\_\{\\beta\}\(\\pi\_\{\\theta\};x\)=\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi^\{\*\}\)\\leq C\_\{x\}\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\(\\pi\_\{\\theta\};x\)\.\(5\)Over all positive policies, the minimum contrastive loss is zero and its unique minimizer isπ∗\\pi^\{\*\}\. Whenπbeh=πθ\\pi\_\{\\mathrm\{beh\}\}=\\pi\_\{\\theta\},𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}equals the exact KL\-regularized optimality gap\.

The ratioCxC\_\{x\}enters the analysis but is not evaluated as an update weight\. For long responses, the worst\-case sequence ratioCxC\_\{x\}in Eq\. \([5](https://arxiv.org/html/2610.20814#S3.E5)\) can be large\. Lemma[17](https://arxiv.org/html/2610.20814#Thmtheorem17)in Appendix[B](https://arxiv.org/html/2610.20814#A2)gives an alternative bound that converts finite\-group loss into a proximal objective gap using aggregate tokenwise mismatch\. Under its local reference assumptions, this can reduce the coverage dependence of the error bound without requiring a smallerCxC\_\{x\}\. Appendix[A\.1](https://arxiv.org/html/2610.20814#A1.SS1)proves the proposition\. Training uses finite groups, whose expected loss differs from𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}and requires the separate guarantee in Section[4\.4](https://arxiv.org/html/2610.20814#S4.SS4)\.

## 4Gibbs\-Gap Policy Optimization

We first describe the finite\-group objective, its gradient, and the training procedure\. We then state the reward guarantees for finite groups and periodic reference refresh\.

### 4\.1From the population objective to finite groups

Givenmmresponsesa1:m∼πbeh\(⋅∣x\)a\_\{1:m\}\\sim\\pi\_\{\\mathrm\{beh\}\}\(\\cdot\\mid x\), letri:=r⁡\(x,ai\)r\_\{i\}:=r\(x,a\_\{i\}\)and define their KL\-adjusted rewardsRi:=ri−β​log⁡\[πθ​\(ai∣x\)/πref​\(ai∣x\)\]=Rθ​\(x,ai\)R\_\{i\}:=r\_\{i\}\-\\beta\\log\[\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\\mid x\)\]=R\_\{\\theta\}\(x,a\_\{i\}\)\. Replacing the behavior expectations in Eq\. \([3](https://arxiv.org/html/2610.20814#S3.E3)\) with their empirical averages yields the single\-group loss

𝒥^m\(πθ;x,a1:m\)=βlog\(1m∑i=1meRi/β\)−1m∑i=1mRi≥0\.\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};x,a\_\{1:m\}\)=\\beta\\log\\\!\\left\(\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}e^\{R\_\{i\}/\\beta\}\\right\)\-\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}R\_\{i\}\\geq 0\.\(6\)This log\-mean\-exp gap compares the adjusted rewards within the sampled group\. It is zero if and only if all responses in the group have the same adjusted reward, matching the optimality condition used to construct the population objective\.

Letρ\\rhobe the prompt distribution\. For fixedπbeh\\pi\_\{\\mathrm\{beh\}\}andπref\\pi\_\{\\mathrm\{ref\}\}, the expected finite\-group target and its prompt average are

Hm​\(πθ,πbeh,x\)\\displaystyle H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x\):=𝔼a1:m∼πbehm𝒥^m\(πθ;x,a1:m\),\\displaystyle:=\\mathbb\{E\}\_\{a\_\{1:m\}\\sim\\pi\_\{\\mathrm\{beh\}\}^\{m\}\}\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};x,a\_\{1:m\}\),\(7\)Lm​\(πθ\)\\displaystyle L\_\{m\}\(\\pi\_\{\\theta\}\):=𝔼X∼ρ​Hm​\(πθ,πbeh,X\)\.\\displaystyle:=\\mathbb\{E\}\_\{X\\sim\\rho\}H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},X\)\.We suppress the fixedβ\\betaand reference in the loss notation, writingHm​\(πθ,πbeh,x,πref\)H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x;\\pi\_\{\\mathrm\{ref\}\}\)when specifying the reference is necessary\. Although the group loss replaces expectations by sample averages, its expectation is not generally the population loss: the logarithm is nonlinear\. For i\.i\.d\. groups,Hm≤𝒥behH\_\{m\}\\leq\\mathcal\{J\}\_\{\\mathrm\{beh\}\}, the two share the same zero set for everym≥2m\\geq 2, andHm→𝒥behH\_\{m\}\\to\\mathcal\{J\}\_\{\\mathrm\{beh\}\}as the group size grows \(Proposition[5](https://arxiv.org/html/2610.20814#Thmtheorem5)\)\. Sharing the same zero set means that both losses identify the same unrestricted target\. It does not, however, make their values interchangeable in a reward bound\. We therefore compare finite\-group loss directly with a reward benchmark\.

### 4\.2Gradient and normalized training loss

Minimizing the group loss gives a weighted contrast between responses\. To see how each response contributes to the update, define the within\-group tilt

qi=eRi/β∑jeRj/β\.q\_\{i\}=\\frac\{e^\{R\_\{i\}/\\beta\}\}\{\\sum\_\{j\}e^\{R\_\{j\}/\\beta\}\}\.\(8\)LetUmU\_\{m\}denote the uniform distribution on themmsample indices\. Then𝒥^m/β=DKL\(Um∥q\)\\widehat\{\\mathcal\{J\}\}\_\{m\}/\\beta=D\_\{\\mathrm\{KL\}\}\(U\_\{m\}\\\|q\), so the finite\-group loss penalizes deviations of the adjusted\-reward tilt from uniformity\. Holding rewards, actions, and reference likelihoods fixed during an actor update gives

∇θ𝒥^m=−β∑i=1m\(qi−1m\)∇θlogπθ\(ai∣x\)\.\\nabla\_\{\\theta\}\\widehat\{\\mathcal\{J\}\}\_\{m\}=\-\\beta\\sum\_\{i=1\}^\{m\}\\left\(q\_\{i\}\-\\frac\{1\}\{m\}\\right\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)\.\(9\)Under gradient descent, responses withqi\>1/mq\_\{i\}\>1/mreceive positive update coefficients, while those withqi<1/mq\_\{i\}<1/mreceive negative coefficients\. The behavior policy determines which responses appear in the group; the update uses no learner\-to\-behavior correction weight\. The coefficient can also be written

qi=softmaxi⁡\(riβ−log⁡πθ​\(ai∣x\)πref​\(ai∣x\)\)\.q\_\{i\}=\\softmax\_\{i\}\\\!\\left\(\\frac\{r\_\{i\}\}\{\\beta\}\-\\log\\frac\{\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\\mid x\)\}\\right\)\.\(10\)The reward term in Eq\. \([10](https://arxiv.org/html/2610.20814#S4.E10)\) separates reward classes, while the learner\-to\-reference ratio redistributes weight among similarly rewarded responses\. This separates two effects in the update: rewarding successful responses and adjusting their relative likelihoods with respect to the reference\. With binary rewards and smallβ\\beta,qqconcentrates on correct responses\. Incorrect responses receive the same limiting negative coefficient, while the weights among correct responses follow a softmax of their negative learner\-to\-reference log ratios\. GGPO therefore gives larger coefficients to correct responses whose likelihood has grown less relative to the reference, and the net coefficient summed over correct responses is largest when only one response in the group is correct\. Figure[6](https://arxiv.org/html/2610.20814#A4.F6)compares these coefficients analytically\. At the population level, the frozen\-behavior gradient of𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}contrasts the behavior distribution with its induced exponential tilt\. Whenπbeh=πθ\\pi\_\{\\mathrm\{beh\}\}=\\pi\_\{\\theta\},𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}equals the exact regularized gap, while its frozen\-behavior gradient follows the forward\-KL/cross\-entropy direction towardπ∗\\pi^\{\*\}\. Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4)gives the population relation, limiting cases, and coefficient analysis; Appendix[C\.4](https://arxiv.org/html/2610.20814#A3.SS4)reports weight\-concentration diagnostics\.

For a minibatch ofGmbG\_\{\\mathrm\{mb\}\}retained groups, let𝒥^mg:=𝒥^m\(πθ;xg,a1:mg\)\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}:=\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};x\_\{g\},a^\{g\}\_\{1:m\}\)\. To reuse GRPO’s training hyperparameters, we use

LGGPO=\(1Gmb​∑g=1Gmb𝒥^mg\)​1β​ℓ¯​m−1,L\_\{\\mathrm\{GGPO\}\}=\\left\(\\frac\{1\}\{G\_\{\\mathrm\{mb\}\}\}\\sum\_\{g=1\}^\{G\_\{\\mathrm\{mb\}\}\}\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}\\right\)\\frac\{1\}\{\\beta\\bar\{\\ell\}\\sqrt\{m\-1\}\},\(11\)whereℓ¯\\bar\{\\ell\}is the minibatch mean response length\. This positive normalization leaves the direction of each realized minibatch gradient unchanged\. It removes the exteriorβ\\betaand converts the sequence\-level group average to a token\-mean coefficient scale, allowing us to use the same training hyperparameters across methods\. Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4)gives the full calculation\.

Training retains only mixed\-reward groups, so the minibatches used by the optimizer differ from the unfiltered groups in the analysis\. For deterministic binary rewards, filtering and positive normalization preserve the zero set on behavior support for each prompt with0<Pra∼πbeh\(⋅∣x\)\[r\(x,a\)=1\]<10<\\Pr\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\(\\cdot\\mid x\)\}\[r\(x,a\)=1\]<1\(Proposition[9](https://arxiv.org/html/2610.20814#Thmtheorem9)\)\. The resulting training objective averages over the retained prompt distribution\. The population and empirical best\-of\-KKbounds concern the raw, unfiltered loss\.

### 4\.3Practical GGPO algorithm

LetNNbe the number of learner minibatches consumed from one frozen behavior collection\. This number is the synchronization delayNN: the learner continues updating while the behavior collection remains fixed\. We synchronize the rollout parameters with the learner, collectN​GmbNG\_\{\\mathrm\{mb\}\}mixed\-reward groups, and partition them intoNNdisjoint minibatches\. Each minibatch receives one optimizer update and is consumed once\. Parameter synchronization does not remove the training–inference mismatch discussed in Section[6](https://arxiv.org/html/2610.20814#S6)\.

Periodic reference refresh is part of the GGPO training algorithm used in our main experiments\. EveryTrefT\_\{\\mathrm\{ref\}\}optimizer updates, we setπref←πθ\\pi\_\{\\mathrm\{ref\}\}\\leftarrow\\pi\_\{\\theta\}and hold it fixed between refreshes\. This resets the learner\-to\-reference log ratio in Eq\. \([10](https://arxiv.org/html/2610.20814#S4.E10)\), keeping the anchor local to the current training stage \(Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4.SSS0.Px3)\)\. AGRO shares the same reference\-dependent Gibbs target, so this motivation applies to its reference updates as well\.

Algorithm[1](https://arxiv.org/html/2610.20814#algorithm1)gives the original, unclipped training procedure\. The optional token\-clipping variant is described below\. The reference schedule uses the global optimizer\-update counter, independently of collection boundaries\. If a refresh occurs within a collection, the remaining responses stay fixed and the next minibatch uses the new reference\. Collection allows at most 32 generation batches, including the initial batch, and stops with an error if the target cannot be filled \(Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\)\.

Algorithm 1Gibbs\-Gap Policy Optimization with periodic reference refreshInput:Initial policyπ0\\pi\_\{0\}, training prompts𝒟\\mathcal\{D\}, binary verifierrr; group sizem≥2m\\geq 2, groups per minibatchGmbG\_\{\\mathrm\{mb\}\}, synchronization delayNN; temperatureβ\>0\\beta\>0, reference intervalTrefT\_\{\\mathrm\{ref\}\}, total optimizer updatesUU\.

Output:Final learnerπθ\\pi\_\{\\theta\}\.

1:Initializeπθ←π0\\pi\_\{\\theta\}\\leftarrow\\pi\_\{0\},πref←π0\\pi\_\{\\mathrm\{ref\}\}\\leftarrow\\pi\_\{0\}, andu←0u\\leftarrow 0\.2:whileu<Uu<Udo3:Synchronize rollout parameters withπθ\\pi\_\{\\theta\}; freeze the resultingπbeh\\pi\_\{\\mathrm\{beh\}\}\.4:Initialize the ordered collection𝒞←∅\\mathcal\{C\}\\leftarrow\\varnothing\.5:while\|𝒞\|<N​Gmb\|\\mathcal\{C\}\|<NG\_\{\\mathrm\{mb\}\}do6:Take the next prompt batch from the shuffled dataloader of𝒟\\mathcal\{D\}\.7:Samplemmresponses per prompt underπbeh\\pi\_\{\\mathrm\{beh\}\}\.8:Evaluateri=r⁡\(x,ai\)r\_\{i\}=r\(x,a\_\{i\}\); append complete groups with0<∑iri<m0<\\sum\_\{i\}r\_\{i\}<mto𝒞\\mathcal\{C\}\.9:end while10:Keep the firstN​GmbNG\_\{\\mathrm\{mb\}\}groups; partition intoNNdisjoint minibatches\.11:for eachminibatch ofGmbG\_\{\\mathrm\{mb\}\}groupsdo12:ifu=Uu=Uthen returnπθ\\pi\_\{\\theta\}\.13:For every response, compute masked sequence log probabilities:14:sg,i←∑tMg,i,t​log⁡πθ​\(ai,tg∣xg,ai,<tg\)s\_\{g,i\}\\leftarrow\\sum\_\{t\}M\_\{g,i,t\}\\log\\pi\_\{\\theta\}\(a^\{g\}\_\{i,t\}\\mid x\_\{g\},a^\{g\}\_\{i,<t\}\);15:sg,iref←∑tMg,i,t​log⁡πref​\(ai,tg∣xg,ai,<tg\)s^\{\\mathrm\{ref\}\}\_\{g,i\}\\leftarrow\\sum\_\{t\}M\_\{g,i,t\}\\log\\pi\_\{\\mathrm\{ref\}\}\(a^\{g\}\_\{i,t\}\\mid x\_\{g\},a^\{g\}\_\{i,<t\}\)\.16:FormRg,i←r⁡\(xg,aig\)−β⁡\(sg,i−sg,iref\)R\_\{g,i\}\\leftarrow r\(x\_\{g\},a^\{g\}\_\{i\}\)\-\\beta\(s\_\{g,i\}\-s^\{\\mathrm\{ref\}\}\_\{g,i\}\)\.17:For each group,𝒥^mg←β​log⁡\(1m​∑ieRg,i/β\)−1m​∑iRg,i\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}\\leftarrow\\beta\\log\(\\frac\{1\}\{m\}\\sum\_\{i\}e^\{R\_\{g,i\}/\\beta\}\)\-\\frac\{1\}\{m\}\\sum\_\{i\}R\_\{g,i\}\.18:Computeℓ¯←\(Gmb​m\)−1​∑g,i,tMg,i,t\\bar\{\\ell\}\\leftarrow\(G\_\{\\mathrm\{mb\}\}m\)^\{\-1\}\\sum\_\{g,i,t\}M\_\{g,i,t\}\.19:SetL←∑g𝒥^mg/\(Gmb​β​ℓ¯​m−1\)L\\leftarrow\\sum\_\{g\}\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}/\(G\_\{\\mathrm\{mb\}\}\\beta\\bar\{\\ell\}\\sqrt\{m\-1\}\)\.20:Take one AdamW update using∇θL\\nabla\_\{\\theta\}L; setu←u\+1u\\leftarrow u\+1\.21:ifumodTref=0u\\bmod T\_\{\\mathrm\{ref\}\}=0then22:Copy the current learner parameters toπref\\pi\_\{\\mathrm\{ref\}\}\.23:end if24:end for25:end while26:returnπθ\\pi\_\{\\theta\}\.
Optional token clipping: replace the adjusted reward in line[1](https://arxiv.org/html/2610.20814#algorithm1)by Eq\. \([12](https://arxiv.org/html/2610.20814#S4.E12)\) and the loss in lines[1](https://arxiv.org/html/2610.20814#algorithm1)–[1](https://arxiv.org/html/2610.20814#algorithm1)by the detached\-coefficient surrogate in Eq\. \([65](https://arxiv.org/html/2610.20814#A5.E65)\)\.

HereMg,i,tM\_\{g,i,t\}is the valid\-response\-token mask\. Rewards, sampled tokens, reference probabilities, and masks are held fixed in differentiation; gradients flow directly through the learner probabilities in the log\-mean\-exp loss\. The weightsqiq\_\{i\}explain this gradient but need not be constructed as a separate policy\-gradient surrogate\. We evaluate the log\-mean\-exp stably, and computeℓ¯\\bar\{\\ell\}across the complete learner minibatch, including all devices and microbatches\.

#### Optional token clipping\.

The main experiments use the original update in Algorithm[1](https://arxiv.org/html/2610.20814#algorithm1)\. The clipping experiment in Section[6\.6](https://arxiv.org/html/2610.20814#S6.SS6)replaces the adjusted reward in line[1](https://arxiv.org/html/2610.20814#algorithm1)by

R¯g,i=r⁡\(xg,aig\)−β​∑tMg,i,t​clip⁡\(log⁡πθ​\(ai,tg∣xg,ai,<tg\)πref​\(ai,tg∣xg,ai,<tg\),−cclip,cclip\)\.\\bar\{R\}\_\{g,i\}=r\(x\_\{g\},a\_\{i\}^\{g\}\)\-\\beta\\sum\_\{t\}M\_\{g,i,t\}\\operatorname\{clip\}\\\!\\left\(\\log\\frac\{\\pi\_\{\\theta\}\(a^\{g\}\_\{i,t\}\\mid x\_\{g\},a^\{g\}\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(a^\{g\}\_\{i,t\}\\mid x\_\{g\},a^\{g\}\_\{i,<t\}\)\},\-c\_\{\\mathrm\{clip\}\},c\_\{\\mathrm\{clip\}\}\\right\)\.\(12\)Herecclip\>0c\_\{\\mathrm\{clip\}\}\>0is the token log\-ratio clipping radius; the experiment usescclip=0\.05c\_\{\\mathrm\{clip\}\}=0\.05\. These adjusted rewards determine detached response coefficients for the surrogate in Eq\. \([65](https://arxiv.org/html/2610.20814#A5.E65)\), replacing the loss construction in lines[1](https://arxiv.org/html/2610.20814#algorithm1)–[1](https://arxiv.org/html/2610.20814#algorithm1)\. Line[1](https://arxiv.org/html/2610.20814#algorithm1)then updates with this surrogate\. The coefficients are clipped through their adjusted rewards, but the surrogate is differentiated using unclipped learner log probabilities\. Consequently, saturated tokens retain their score gradients\. Direct differentiation through a clipped log\-mean\-exp loss would give a different update\. The appendix specifies the coefficients and normalization\.

The default setting usesGmb=32G\_\{\\mathrm\{mb\}\}=32,m=8m=8,N=8N=8,β=0\.01\\beta=0\.01, andTref=160T\_\{\\mathrm\{ref\}\}=160\. Appendices[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)and[C\.4](https://arxiv.org/html/2610.20814#A3.SS4)specify the refill budget, masking, EOS, padding, and truncation details\. Fixed\-pool offline training replaces behavior collection with successive minibatches from a precollected pool; each is still consumed once, and reference refresh follows the same optimizer\-update schedule\.

### 4\.4Best\-of\-KKreward and the finite\-group loss

For each prompt, best\-of\-KKsampling drawsKKindependent responses from a policyπ\\piand returns the one with the highest verifier reward\. Its expected reward, averaged over the prompt distributionρ\\rho, is

JK\(π\):=𝔼X∼ρ𝔼A1:K∼π\(⋅∣X\)K\[maxi≤Kr\(X,Ai\)\]\.J\_\{K\}\(\\pi\):=\\mathbb\{E\}\_\{X\\sim\\rho\}\\mathbb\{E\}\_\{A\_\{1:K\}\\sim\\pi\(\\cdot\\mid X\)^\{K\}\}\\left\[\\max\_\{i\\leq K\}r\(X,A\_\{i\}\)\\right\]\.\(13\)The comparison of interest is between this selection procedure and drawing a single response from the learner\. A single response from a policyπ\\pihas expected rewardJ\(π\)=𝔼X∼ρ,A∼π\(⋅∣X\)r\(X,A\)J\(\\pi\)=\\mathbb\{E\}\_\{X\\sim\\rho,A\\sim\\pi\(\\cdot\\mid X\)\}r\(X,A\); in particular,J1​\(π\)=J​\(π\)J\_\{1\}\(\\pi\)=J\(\\pi\)\. The selection budgetKKis distinct from the synchronization delayNN\(Section[4\.3](https://arxiv.org/html/2610.20814#S4.SS3)\)\. For rewards in\[0,1\]\[0,1\], Proposition[6](https://arxiv.org/html/2610.20814#Thmtheorem6)\(Appendix[A\.2](https://arxiv.org/html/2610.20814#A1.SS2)\) gives the bound

JK​\(πbeh\)−J⁡\(πθ\)≤κK,m​\{β⁡\(log⁡m\+Γ\)\+C​Lm​\(πθ\)\}\.J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\}\)\-J\(\\pi\_\{\\theta\}\)\\leq\\kappa\_\{K,m\}\\\{\\beta\(\\log m\+\\Gamma\)\+CL\_\{m\}\(\\pi\_\{\\theta\}\)\\\}\.\(14\)HereΓ\\Gammabounds the spread \(maximum minus minimum\) oflog⁡\(πref/πbeh\)\\log\(\\pi\_\{\\mathrm\{ref\}\}/\\pi\_\{\\mathrm\{beh\}\}\)across responses to a fixed prompt and vanishes whenπref=πbeh\\pi\_\{\\mathrm\{ref\}\}=\\pi\_\{\\mathrm\{beh\}\}, whileCCuniformly boundsπθ/πbeh\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{beh\}\}over the hypothesis class\. For the individual candidateπθ=πbeh\\pi\_\{\\theta\}=\\pi\_\{\\mathrm\{beh\}\}, the ratio is one; a class\-wide bound may still exceed one\. The tail\-comparison multiplierκK,m\\kappa\_\{K,m\}, defined in Appendix[A\.2](https://arxiv.org/html/2610.20814#A1.SS2), satisfiesκK,m≤max⁡\{1,K/\(m−1\)\}\\kappa\_\{K,m\}\\leq\\max\\\{1,K/\(m\-1\)\\\}\. In particular,m≥K\+1m\\geq K\+1givesκK,m=1\\kappa\_\{K,m\}=1\. Theorem[7](https://arxiv.org/html/2610.20814#Thmtheorem7)in Appendix[A\.3](https://arxiv.org/html/2610.20814#A1.SS3)converts it into an empirical bound based on raw loss evaluated on i\.i\.d\. unfiltered groups\.

### 4\.5Reference refresh guarantees

A fixed reference defines a single KL\-regularized Gibbs target in Eq\. \([2](https://arxiv.org/html/2610.20814#S3.E2)\), independent of group size\. Reference refresh changes this target by using the current learner as the next reference\. Exact target composition makes this effect explicit: successive refreshes increase the effective reward tilt\.

#### Refresh composes temperatures\.

Letπ0\\pi\_\{0\}be the initial policy andπt\\pi\_\{t\}the output of reference\-update roundtt\. If each round exactly attains its Gibbs target relative to the previous output, Eq\. \([2](https://arxiv.org/html/2610.20814#S3.E2)\) gives

πT​\(a∣x\)∝π0​\(a∣x\)​eT​r​\(x,a\)/β\.\\pi\_\{T\}\(a\\mid x\)\\propto\\pi\_\{0\}\(a\\mid x\)e^\{Tr\(x,a\)/\\beta\}\.\(15\)ThusTTexact rounds at temperatureβ\\betaimplement one Gibbs update at temperatureβ/T\\beta/T\. The following result extends the best\-of\-KKcomparison to reference refresh when the round losses are nonzero\.

###### Proposition 2\(Reference refresh with imperfect rounds, informal\)\.

For rewards in\[0,1\]\[0,1\], considerTTreference\-update rounds with group sizem≥2m\\geq 2\. Assume positive policies, uniform behavior coverage of both the round outputs and the best\-of\-KKcomparator, and bounded regularized residual range, as specified in Proposition[11](https://arxiv.org/html/2610.20814#Thmtheorem11), Appendix[A\.5](https://arxiv.org/html/2610.20814#A1.SS5)\. Letπ¯T\\bar\{\\pi\}\_\{T\}be the uniform mixture of the round outputs andL¯T\\bar\{L\}\_\{T\}their average population raw GGPO loss\. Then

JK​\(π0\)−J⁡\(π¯T\)≤β​log⁡KT\+O⁡\(β​L¯T\),J\_\{K\}\(\\pi\_\{0\}\)\-J\(\\bar\{\\pi\}\_\{T\}\)\\leq\\frac\{\\beta\\log K\}\{T\}\+O\\\!\\left\(\\sqrt\{\\beta\\bar\{L\}\_\{T\}\}\\right\),where the hidden constant depends only on the coverage and residual\-range bounds and the group size\.

The mixtureπ¯T\\bar\{\\pi\}\_\{T\}selects a round uniformly and samples one response from its output policy\. The lossL¯T\\bar\{L\}\_\{T\}averages the population group loss in Eq\. \([7](https://arxiv.org/html/2610.20814#S4.E7)\) across rounds, using behaviorπbeh,t\\pi\_\{\\mathrm\{beh\},t\}and referenceπt−1\\pi\_\{t\-1\}in roundtt\. HereTTcounts reference updates, whileKKis the number of initial\-policy responses used by the best\-of\-KKcomparator\. More refresh rounds reduce the initial\-reference penaltyβ​log⁡K/T\\beta\\log K/T, while accurate fitting controls the remaining error\. The regularized residual is reward minusβ\\betatimes the log ratio of the round output to its reference\. Its range is the spread of these values across responses to a fixed prompt; bounding that range and the coverage ratios controls the constant hidden in the loss term\. Appendix[A\.5](https://arxiv.org/html/2610.20814#A1.SS5)gives the precise statement and proof\.

The distinction between the mixture and the final policy matters for deployment\. At zero loss, exact composition gives the sameβ​log⁡K/T\\beta\\log K/Tbound for the final outputπT\\pi\_\{T\}\. For imperfect updates, the next section gives a final\-policy guarantee under binary rewards and aggregate tokenwise mismatch control\.

## 5Final\-policy reward under reference refresh

Reference refresh changes the Gibbs target as training progresses\. With binary rewards, each exact update multiplies the odds of a correct response bye1/βe^\{1/\\beta\}\. Thus a correct response that already has positive probability under the initial policy can receive progressively more probability over successive rounds\. In practice, each round only approximates its Gibbs target, so the question is whether these fitting errors allow the improvement to continue\. The result below connects the accuracy of individual updates to the final policy’s single\-response reward\.

This requires relating two distributions: GGPO’s loss is measured on responses from the behavior policy, while reward is evaluated under the returned learner\. The bounds in Eqs\. \([5](https://arxiv.org/html/2610.20814#S3.E5)\) and \([14](https://arxiv.org/html/2610.20814#S4.E14)\) use worst\-case sequence coverage to make this connection\. Here we use moments of the learner\-to\-behavior response likelihood ratio to relate the two distributions\. Small conditional token log ratios can keep these moments moderate even when the worst\-case sequence ratio is large\. The guarantee accounts for imperfect fitting and finite data\.

#### Analyzed procedure\.

Start fromπ0\\pi\_\{0\}\. In roundtt, freeze referenceπt−1\\pi\_\{t\-1\}, behaviorπbeh,t\\pi\_\{\\mathrm\{beh\},t\}, and candidate classΠt\\Pi\_\{t\}before sampling\. Drawnnfresh, independent prompts fromρ\\rho\. With group sizem≥2m\\geq 2, samplemmindependent responses underπbeh,t\\pi\_\{\\mathrm\{beh\},t\}for each prompt draw, without filtering\. LetL^t​\(π\)\\widehat\{L\}\_\{t\}\(\\pi\)be the average raw GGPO loss in Eq\. \([6](https://arxiv.org/html/2610.20814#S4.E6)\) over thesenngroups\. Fit this loss overΠt\\Pi\_\{t\}, assuming the returned policyπt∈Πt\\pi\_\{t\}\\in\\Pi\_\{t\}satisfies

L^t​\(πt\)−infπ∈ΠtL^t​\(π\)≤ξ\.\\widehat\{L\}\_\{t\}\(\\pi\_\{t\}\)\-\\inf\_\{\\pi\\in\\Pi\_\{t\}\}\\widehat\{L\}\_\{t\}\(\\pi\)\\leq\\xi\.Hereξ≥0\\xi\\geq 0bounds the achieved empirical optimization error in each round, as specified in Eq\. \([42](https://arxiv.org/html/2610.20814#A2.E42)\)\. Useπt\\pi\_\{t\}as the next reference\. Data may be reused within a round\. LetTTdenote the total number of reference rounds\.

###### Theorem 3\(Final\-policy reward guarantee, informal\)\.

Consider the reference\-update procedure above with binary rewards and responses of at mostHHtokens\. Assume common positive support, a sufficiently large temperatureβ\\betaof orderHH, and small conditional token log ratios relative to each reference\. Letπ⋆\\pi\_\{\\star\}be a comparator whose success probability at each prompt is at mostC⋆C\_\{\\star\}times that of the initial policyπ0\\pi\_\{0\}\. Assume a uniform bound on a moment of order at least two of the learner\-to\-behavior response ratio, or on the ratio itself\. Assume the representation, achieved optimization accuracy, and loss\-cover conditions of Theorem[13](https://arxiv.org/html/2610.20814#Thmtheorem13)\.

For any target error0<ε<10<\\varepsilon<1and failure probability0<δ<10<\\delta<1, suitable numbers of reference rounds and fresh prompt draws give, with probability at least1−δ1\-\\deltaover the sampled prompts and responses,

J\(πT\)≥J\(π⋆\)−ε−ℰfit,J\(π\)=𝔼X∼ρ,y∼π\(⋅∣X\)r\(X,y\)\.J\(\\pi\_\{T\}\)\\geq J\(\\pi\_\{\\star\}\)\-\\varepsilon\-\\mathcal\{E\}\_\{\\rm fit\},\\qquad J\(\\pi\)=\\mathbb\{E\}\_\{X\\sim\\rho,y\\sim\\pi\(\\cdot\\mid X\)\}r\(X,y\)\.Hereℰfit\\mathcal\{E\}\_\{\\rm fit\}is the fitting residual in Eq\. \([51](https://arxiv.org/html/2610.20814#A2.E51)\) or \([53](https://arxiv.org/html/2610.20814#A2.E53)\), according to the coverage condition\. For fixed moment order and coverage coefficient, sufficient prompt and response counts are polynomial inHH,C⋆C\_\{\\star\}, and1/ε1/\\varepsilon, with logarithmic dependence on the loss\-cover size and1/δ1/\\delta\.

Theorem[13](https://arxiv.org/html/2610.20814#Thmtheorem13)in Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1)gives the full finite\-moment statement, including the assumptions, sample budget, and fitting residual\. Remark[14](https://arxiv.org/html/2610.20814#Thmtheorem14)gives the bounded\-coverage case\.

#### Interpretation of the theorem\.

The guarantee concerns the expected reward of a single response from the final policyπT\\pi\_\{T\}\. This expected reward is at least the comparator’s reward minusε\+ℰfit\\varepsilon\+\\mathcal\{E\}\_\{\\rm fit\}\. The coverage constantC⋆C\_\{\\star\}describes how much improvement is requested relative to initialization\. For example, oracle best\-of\-KKsampling fromπ0\\pi\_\{0\}satisfies the coverage condition withC⋆=KC\_\{\\star\}=K, because the probability of finding a correct response inKKdraws is at mostKKtimes the single\-draw success probability\. Thus the expected reward of one response from the trained policy can approach that of selecting the best ofKKresponses from the initial policy\. The training cost is accounted for separately\.

The target errorε\\varepsilonis controlled by the number of reference rounds and fresh prompt draws\. The residualℰfit\\mathcal\{E\}\_\{\\rm fit\}captures representation error, achieved empirical optimization error, and loss\-cover approximation error, whose combined size isεA\\varepsilon\_\{A\}in Eq\. \([43](https://arxiv.org/html/2610.20814#A2.E43)\)\. Increasing the sample size alone does not remove persistent fitting error\. For fixed temperature and coverage, the bound multiplies fitting error by a factor quadratic inTT\. Longer runs therefore require more accurate fits to keep the same reward bound\. Under bounded coverage, the residual is linear inεA\\varepsilon\_\{A\}\. At finite orderpp, it also includes a term proportional toεA1−1/p\\varepsilon\_\{A\}^\{1\-1/p\}, whose coefficient decreases as the learner\-to\-behavior coverage moment approaches one\. The residual vanishes when the three fitting errors are zero\.

The sample bound countsnnindependent prompt draws per round, each withmmresponses, givingT​nTnprompt groups andm​T​nmTnscored responses\. Prompts may repeat under independent sampling, but additional groups on a fixed prompt pool do not supply fresh prompt draws\. Loss\-class complexity enters because the learner is chosen using the data: the loss cover approximates all candidate policies’ group\-loss functions uniformly, and its size measures how many functions are needed at the chosen accuracy\. Under these sampling and policy constraints, the guarantee applies to any solver attaining the stated empirical accuracy; it does not establish convergence of AdamW\. Under the token conditions in Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1), the moment coefficient is bounded bycν=exp⁡\(\(1\+ν−1\)​Vlog\)c\_\{\\nu\}=\\exp\(\(1\+\\nu^\{\-1\}\)V\_\{\\log\}\)for0<ν≤10<\\nu\\leq 1, whereVlogV\_\{\\log\}bounds the sum of conditional token log\-ratio second moments along each response path\. The statistical group budget has accuracy dependence1/ε\+cν/ε1\+ν1/\\varepsilon\+c\_\{\\nu\}/\\varepsilon^\{1\+\\nu\}, apart from round and problem factors\. Bounded full\-response coverage gives the direct1/ε1/\\varepsilondependence\. A smallerν\\nureduces the accuracy exponent but can increasecνc\_\{\\nu\}and requires tighter token ratios\. The benefit therefore depends on both the token mismatch and the chosen moment order\.

#### Behavior lag and practical clipping\.

If behavior is a stored policy at mostddreference rounds old, local reference control gives\|log⁡\(πt,h/πbeh,t,h\)\|≤d/β\|\\log\(\\pi\_\{t,h\}/\\pi\_\{\\mathrm\{beh\},t,h\}\)\|\\leq d/\\betaand allowsVlog=H​d2/β2V\_\{\\log\}=Hd^\{2\}/\\beta^\{2\}\. This supplies the stated log\-moment coefficient whend/β≤min⁡\{log⁡2,3​ν/2\}d/\\beta\\leq\\min\\\{\\log 2,3\\nu/2\\\}\. The same lag bound also gives direct sequence coverage𝒞∞≤eH​d/β\\mathcal\{C\}\_\{\\infty\}\\leq e^\{Hd/\\beta\}, which can be used in the bounded\-coverage branch without this token cap\. In particular, sampling from the current reference hasd=1d=1and𝒞∞≤eH/β<e\\mathcal\{C\}\_\{\\infty\}\\leq e^\{H/\\beta\}<\\sqrt\{e\}under the stated temperature condition, so its group budget has linear accuracy dependence apart from round and problem factors\. The lag counts reference rounds rather than the experimental synchronization delayNN\. Appendix[B\.6](https://arxiv.org/html/2610.20814#A2.SS6)gives the relaxed constraint and an alternative bound from token spans\.

The theorem’s small token log ratios keep every candidate policy close to its frozen reference at each response prefix\. The precise constraint in Eq\. \([40](https://arxiv.org/html/2610.20814#A2.E40)\) bounds the actual conditional probabilities and permits the exact Gibbs target, as shown in Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1)\. This motivates practical token clipping, which instead limits log\-ratio contributions to response coefficients, with some bias, and does not enforce the same policy constraint\. Section[6\.6](https://arxiv.org/html/2610.20814#S6.SS6)evaluates this stabilization, and Appendix[E](https://arxiv.org/html/2610.20814#A5)specifies its implementation\. The theorem analyzes constrained fitting of the raw loss\.

## 6Experiments

We compare GRPO, AGRO, and GGPO in default training \(N=8N=8\), including pass@KKand sampling cost, then examine reference refresh, controlled policy staleness \(N=8N=8to128128\), offline training on fixed data, and optional token clipping\. These settings examine how the methods respond to behavior–learner mismatch\. Training–inference mismatch remains present even immediately after the rollout parameters are synchronized with the learner\. Appendices[D](https://arxiv.org/html/2610.20814#A4)and[C](https://arxiv.org/html/2610.20814#A3)give additional results and implementation details\.

### 6\.1Experimental setup

Models, data, and evaluation\.We evaluate mathematical reasoning with Qwen3\-1\.7B\-Base and Qwen3\-4B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib46)\)\. Training uses ALP\-Math, which merges a difficulty\-filtered OpenR1\-derived corpus\([2025](https://arxiv.org/html/2610.20814#bib.bib1)\)with the math split of Guru\-RL\([Cheng et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib10)\)\. Following standard RLVR protocols\([Guo et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib16)\), rewards are binary and verified by symbolic equivalence with Math\-Verify\([Kydlíček, 2025](https://arxiv.org/html/2610.20814#bib.bib22)\)\. Under the inherited training configuration described below, progress is monitored on MATH\-500\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.20814#bib.bib18);[Lightman et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib25)\)\. Monitoring results were not used for hyperparameter or checkpoint selection\. For the final comparison, we report mean@32, the mean accuracy over 32 newly sampled responses per problem, on MATH\-500,[AIME 2024](https://huggingface.co/datasets/math-ai/aime24),[AIME 2025](https://huggingface.co/datasets/math-ai/aime25), OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib17)\), and Minerva Math\([Lewkowycz et al\., 2022](https://arxiv.org/html/2610.20814#bib.bib23)\)\. Mean@32 estimates single\-response mean reward under the evaluation rollout policy\. Training curves also report MATH\-500 best@32, the fraction of problems with at least one correct response among the 32 samples, equivalent to pass@32 for binary rewards \(Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\)\. The problem\-weighted average over the five benchmarks gives every evaluated problem equal weight \(counts in Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\)\.

Seeds and checkpoint reporting\.All five\-benchmark tables report final checkpoints; default and controlled staleness runs use 8,000 optimizer updates\. Each checkpoint is evaluated once with 32 responses per problem\. We report means and sample standard deviations over training seeds 42, 1234, and 2027 for the Qwen3\-1\.7B\-Base default comparison \(Table[1](https://arxiv.org/html/2610.20814#S6.T1), Figure[1](https://arxiv.org/html/2610.20814#S6.F1)\), offline accuracy \(Table[5](https://arxiv.org/html/2610.20814#A4.T5), Figure[4](https://arxiv.org/html/2610.20814#S6.F4), left\), andN=128N=128staleness, including GRPO\+TIS \(Table[4](https://arxiv.org/html/2610.20814#A4.T4)\)\. Curves summarize each logged or evaluated update\. Other conditions and offline diagnostics use seed 42\. When a comparison reuses default results, those results retain their three\-seed means, as noted in table captions\.

Baselines\.AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\)is our direct objective\-level comparator: it shares GGPO’s KL\-adjusted reward, Gibbs target, and independently controlled reference schedule\. OAPL\([Ritter et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib37)\)ties its KL anchor and value target to the rollout policy, coupling target refresh to behavior synchronization\. Matching our independent reference schedule would therefore require modifying its target estimator\. We discuss OAPL as related work and focus the controlled objective comparison on AGRO, with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib40)\)and GRPO\+TIS providing policy\-gradient baselines\. In the default runs, GRPO uses token\-level learner\-to\-behavior PPO ratios with the log probabilities recorded during rollout as the denominator and no additional off\-policy correction\. We include GRPO\+TIS\([Zhong et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib53)\)under the largest synchronization delay,N=128N=128, and in fixed\-data offline training to evaluate importance\-corrected policy\-gradient updates in these settings\. It uses token\-level truncated importance sampling with a cap of 2 \(Appendix[C\.2](https://arxiv.org/html/2610.20814#A3.SS2.SSS0.Px3)\)\.

Hyperparameters\.By default, we setβ=0\.01\\beta=0\.01for all methods, following the offline experiments of AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\), and drawm=8m=8responses per prompt\. Appendix[D\.6](https://arxiv.org/html/2610.20814#A4.SS6)also evaluatesβ=0\.001\\beta=0\.001, used in AGRO’s on\-policy experiments\. The methods share the same batch geometry and prompt order, and reuse the learning rate and optimizer settings of our GRPO baseline\. They also use the same dynamic difficulty\-filtering rule\([Yu et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib47);[Bae et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib4)\)\. By default, our GRPO implementation keeps its initial reference fixed, while AGRO and GGPO refresh the reference everyTref=160T\_\{\\mathrm\{ref\}\}=160optimizer updates, matching our checkpointing cadence \(Section[4\.3](https://arxiv.org/html/2610.20814#S4.SS3)\)\. Appendices[D\.5](https://arxiv.org/html/2610.20814#A4.SS5)and[D\.6](https://arxiv.org/html/2610.20814#A4.SS6)compare reference cadences and group weights\. The complete reference, batch, decoding, refill, and optimizer settings are given in Appendix[C](https://arxiv.org/html/2610.20814#A3)\.

Training budget\.Method comparisons match optimizer update counts and retained responses per update\. Mixed\-reward filtering and refill can nevertheless require different numbers of generated responses across methods\. Figure[2](https://arxiv.org/html/2610.20814#S6.F2)reports this sampling demand\. Response counts do not equalize generated tokens or GPU\-hours, since those costs also depend on response length and execution costs\.

### 6\.2Default training setting

Under default training \(N=8N=8; Section[4\.3](https://arxiv.org/html/2610.20814#S4.SS3)\), all three methods’ token\-mean K3 stays on the order of10−310^\{\-3\}throughout training \(Figure[1](https://arxiv.org/html/2610.20814#S6.F1), panel 1\)\. Table[1](https://arxiv.org/html/2610.20814#S6.T1)compares their final checkpoints at both model sizes\.

Table 1:Comparison under the default training setting \(N=8N=8\)\.Final\-checkpoint mean@32 accuracy after 8,000 optimizer updates on five mathematical\-reasoning benchmarks \(%\)\. The final column is the problem\-weighted average over the five benchmarks, which gives every evaluated problem equal weight\. For Qwen3\-1\.7B\-Base, values are the mean±\\pmsample standard deviation across training seeds 42, 1234, and 2027\. For Avg\., these statistics are computed across the per\-seed problem\-weighted averages\. Qwen3\-4B\-Base uses seed 42\. Bold marks the best value in each model\-size block\.#### Results\.

GGPO achieves the highest problem\-weighted average at both model sizes \(Table[1](https://arxiv.org/html/2610.20814#S6.T1)\)\. On Qwen3\-1\.7B\-Base, it exceeds AGRO by 1\.5 points and GRPO by 3\.9 points\. The single\-seed Qwen3\-4B\-Base comparison shows the same ordering of problem\-weighted accuracy at the larger model size\. Figure[1](https://arxiv.org/html/2610.20814#S6.F1)shows that GGPO initially improves more slowly on MATH\-500 mean@32, but overtakes both baselines later in training\. GGPO maintains higher policy entropy, while its best@32 continues to improve later in training\. Appendix[D\.1](https://arxiv.org/html/2610.20814#A4.SS1)reports performance stratified by the base model’s per\-problem success rate\.

Figure 1:Training dynamics under the default training setting \(N=8N=8\) on Qwen3\-1\.7B\-Base\.From left to right: the token\-mean K3 diagnostic of behavior–learner mismatch\([Schulman, 2020](https://arxiv.org/html/2610.20814#bib.bib38)\)\(×10−3\\times 10^\{\-3\}\), policy entropy, MATH\-500 best@32 \(%\), and MATH\-500 mean@32 \(%\), each against optimizer updates \(GRPO orange, AGRO green, GGPO blue\)\. K3 is computed on mixed\-reward\-filtered training minibatches\. In the first two panels, faint lines are the logged series \(one point per eight updates\) and solid lines a nine\-point moving average\. At each reported update, curves summarize seeds 42, 1234, and 2027 by their mean, with shaded bands spanning one sample standard deviation across seeds; MATH\-500 is evaluated every 160 optimizer updates\.
#### Pass@KKand best\-of\-KKcomparison\.

On Qwen3\-1\.7B\-Base, GGPO leads in problem\-weighted five\-benchmark pass@KKfor every evaluatedK∈\{1,4,8,16,32\}K\\in\\\{1,4,8,16,32\\\}with training seed 42 \(Table[3](https://arxiv.org/html/2610.20814#A4.T3)\)\. AtK=32K=32, it reaches73\.90%73\.90\\%, compared with69\.39%69\.39\\%for the strongest baseline and62\.68%62\.68\\%for the base model\. Its seed\-42 pass@1 \(46\.60%46\.60\\%\) exceeds the base model’s pass@4 \(38\.57%38\.57\\%\) and approaches its pass@8 \(47\.43%47\.43\\%\)\. Thus, in this comparison, a single GGPO response succeeds more often than best\-of\-four sampling from the base model, illustrating the comparison in Section[5](https://arxiv.org/html/2610.20814#S5)\. Pass@1 equals mean@32 under our estimator; the46\.60%46\.60\\%value uses seed 42, whereas Table[1](https://arxiv.org/html/2610.20814#S6.T1)reports the three\-seed mean of46\.58%46\.58\\%\. Appendix[D\.2](https://arxiv.org/html/2610.20814#A4.SS2)gives the full pass@KKresults\.

#### Sampling cost\.

GGPO reaches higher final MATH\-500 mean@32 with fewer generated training responses than either baseline in the reported default\-setting comparison \(Figure[2](https://arxiv.org/html/2610.20814#S6.F2)\)\. This complements the comparison at matched optimizer updates and retained responses; token and computation costs also depend on response lengths and execution costs \(Appendix[D\.8](https://arxiv.org/html/2610.20814#A4.SS8)\)\.

Figure 2:Sampling budget under the default training setting on Qwen3\-1\.7B\-Base\.Left: cumulative generated training responses, in millions, against optimizer updates\. Right: MATH\-500 mean@32 accuracy against the same cumulative response count\. Each curve ends at 8,000 optimizer updates \(GRPO orange squares, AGRO green triangles, GGPO blue circles\)\. Counts include responses discarded by mixed\-reward filtering and generated during refill\. GGPO finishes with the highest accuracy and the lowest response count\.

### 6\.3Reference refresh

The default comparison uses a fixed reference for GRPO and refresh everyTref=160T\_\{\\mathrm\{ref\}\}=160updates for AGRO and GGPO\. Table[2](https://arxiv.org/html/2610.20814#S6.T2)compares all three methods across these reference schedules atβ=0\.01\\beta=0\.01\. GGPO’s problem\-weighted average is32\.96%32\.96\\%with a fixed reference \(seed 42\), compared with46\.58%46\.58\\%with refresh every 160 updates \(three\-seed mean\); refresh every 320 updates reaches44\.96%44\.96\\%\(seed 42\)\. AGRO also benefits from refresh, while refreshing GRPO does not close the accuracy gap\. GGPO’s weaker fixed\-reference result shows that reference refresh is an important part of the complete method\. Appendix[D\.5](https://arxiv.org/html/2610.20814#A4.SS5)gives per\-benchmark results, theβ=0\.1\\beta=0\.1conditions, and reference\-drift diagnostics\.

Table 2:Reference refresh on Qwen3\-1\.7B\-Base\.Problem\-weighted five\-benchmark mean@32 \(%\) after 8,000 updates, withβ=0\.01\\beta=0\.01\. A dagger marks default settings reported as means over seeds 42, 1234, and 2027; other cells use seed 42\. Table[6](https://arxiv.org/html/2610.20814#A4.T6)gives the full results\.
### 6\.4Controlled policy staleness

With synchronization delayNN, each frozen behavior collection of32​N32Nprompts is split intoNNdisjoint 32\-prompt minibatches, each used for one optimizer update before the behavior policy is synchronized\. Sample age is therefore0,…,N−10,\\ldots,N\-1optimizer updates\. Training–inference mismatch remains present as a fixed background\. Following controlled asynchronous RLHF evaluations\([Noukhovitch et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib33);[Huang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib19)\), we varyNNover the staleness ladderN∈\{8,16,32,64,128\}N\\in\\\{8,16,32,64,128\\\}with method\-specific hyperparameters fixed; theN=8N=8condition reuses the Qwen3\-1\.7B\-Base default\-setting run\. This varies the age of the sampled responses without changing the method\-specific hyperparameters\. Reference handling is unchanged from the default setting at every delay; Appendix[C\.4](https://arxiv.org/html/2610.20814#A3.SS4)states how a refresh boundary inside a collection is handled\.

Figure[3](https://arxiv.org/html/2610.20814#S6.F3)summarizes performance with the problem\-weighted average of Section[6\.1](https://arxiv.org/html/2610.20814#S6.SS1); Appendix[D\.3](https://arxiv.org/html/2610.20814#A4.SS3)gives the per\-benchmark table and the training\-time diagnostics againstNN\.

Figure 3:Controlled policy staleness on Qwen3\-1\.7B\-Base\.Problem\-weighted final\-checkpoint mean@32 accuracy over the five benchmarks after 8,000 optimizer updates, against synchronization delayNN\(equally spaced positions; values in Table[4](https://arxiv.org/html/2610.20814#A4.T4)\)\. GRPO\+TIS is evaluated atN=128N=128\. Results atN=8,128N=8,128are three\-seed means, with error bars showing±1\\pm 1sample standard deviation;N=16,32,64N=16,32,64use seed 42\.#### Results\.

GGPO maintains stable performance as policy staleness increases fromN=8N=8toN=128N=128, with problem\-weighted accuracy of46\.32%46\.32\\%–46\.64%46\.64\\%\. AGRO loses 4\.2 points fromN=8N=8toN=128N=128, while GRPO is essentially unchanged throughN=64N=64before declining atN=128N=128\. At the largest synchronization delay,N=128N=128, token\-level TIS raises GRPO’s problem\-weighted accuracy from39\.02%39\.02\\%to40\.69%40\.69\\%, while GGPO reaches46\.39%46\.39\\%under the same delay \(Figure[3](https://arxiv.org/html/2610.20814#S6.F3)and Table[4](https://arxiv.org/html/2610.20814#A4.T4)\)\. The training\-time diagnostics show increasing behavior–learner mismatch for GGPO fromN=8N=8toN=128N=128: its K3 rises and normalized effective sample size \(ESS\) falls, while both diagnostics remain nearly unchanged for uncorrected GRPO \(Figure[7](https://arxiv.org/html/2610.20814#A4.F7)\)\.

GRPO’s nearly constant K3 is consistent with the PPO surrogate suppressing updates that would move learner\-to\-behavior ratios farther beyond its clipping interval\. This clipping changes the objective but imposes no hard constraint on the policy ratios\. GGPO uses no learner\-to\-behavior importance weights \(Section[4\.2](https://arxiv.org/html/2610.20814#S4.SS2)\); its rising K3 and stable accuracy are consistent with learning as the learner departs from the collection’s behavior policy\. AGRO also avoids learner\-to\-behavior importance weights, but its accuracy declines at larger delays\. These comparisons still allow the behavior policy to catch up with the learner everyNNupdates\. The offline setting in Section[6\.5](https://arxiv.org/html/2610.20814#S6.SS5)removes this resynchronization, allowing us to examine learning from a single fixed behavior pool\.

### 6\.5Offline training on fixed data

In the offline setting, GRPO, GRPO\+TIS, AGRO, and GGPO perform 400 optimizer updates on the same fixed rollout pool within each seed without generating new responses\. This setting tests learning from fixed data as behavior–learner mismatch accumulates without behavior resynchronization\. Appendix[C\.3](https://arxiv.org/html/2610.20814#A3.SS3)gives the protocol and Table[5](https://arxiv.org/html/2610.20814#A4.T5)the accuracy at every evaluated checkpoint\.

Figure 4:Offline training on Qwen3\-1\.7B\-Base\.Within each seed, all methods use the same fixed rollout pool, batch geometry, prompt order, and 400\-update budget \(GRPO orange, GRPO\+TIS purple dashed, AGRO green, GGPO blue\)\. From left: MATH\-500 mean@32 \(%\), with the reported mean accuracy and bands spanning one reported sample standard deviation \(Table[5](https://arxiv.org/html/2610.20814#A4.T5)\); then, for seed 42, the token\-mean absolute learner\-to\-behavior log ratio \(log scale\), the normalized ESS of the diagnostic sequence\-level learner\-to\-behavior importance weights over each logged block of 2,048 responses \(log scale\), and the actor gradient norm \(Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\)\. The mismatch diagnostics use mixed\-reward\-filtered training responses\.#### Results\.

GGPO improves MATH\-500 accuracy from32\.96%32\.96\\%to59\.79%59\.79\\%within 240 optimizer updates, leading the baselines at every evaluated checkpoint from 160 updates onward \(Figure[4](https://arxiv.org/html/2610.20814#S6.F4); Table[5](https://arxiv.org/html/2610.20814#A4.T5)\)\. GGPO, AGRO, and GRPO\+TIS all peak at 240 updates and decline thereafter, alongside increasing divergence from the fixed behavior policy\.

The diagnostic panels show increasing drift and declining normalized ESS for GRPO\+TIS, AGRO, and GGPO, while uncorrected GRPO’s drift stays near the level set by the initial training–inference mismatch\. For GGPO, normalized ESS is already below10−310^\{\-3\}by 160 updates\. This diagnostic indicates that sequence\-level importance weights on the filtered training responses are highly concentrated at that point\.

### 6\.6Token clipping

The local reference constraint in Section[5](https://arxiv.org/html/2610.20814#S5)motivates limiting individual token log\-ratio contributions in practice\. In the default Qwen3\-1\.7B\-Base setting, the optional clipped variant of Algorithm[1](https://arxiv.org/html/2610.20814#algorithm1), with radiuscclip=0\.05c\_\{\\mathrm\{clip\}\}=0\.05, yields higher early training rewards than the original update, with a smaller gap later \(Figure[10](https://arxiv.org/html/2610.20814#A4.F10), Appendix[D\.9](https://arxiv.org/html/2610.20814#A4.SS9)\)\. This single\-seed comparison measures reward on mixed\-reward\-filtered training responses, so it does not establish improved held\-out accuracy or verify the theorem’s uniform local reference constraint\. All other experiments use the original, unclipped update; Appendix[E](https://arxiv.org/html/2610.20814#A5)gives implementation details\.

## 7Conclusion

GGPO learns from contrasts in KL\-adjusted rewards among behavior\-policy responses without learner\-to\-behavior importance weights\. Under suitable coverage, its finite\-group loss controls the gap between the learner’s single\-response expected reward and the behavior policy’s best\-of\-KKreward\. Reference refresh lets successive rounds build on earlier progress by changing the Gibbs target as the learner improves\.

Our experiments examine this approach on mathematical reasoning, where GGPO outperforms GRPO and AGRO under training–inference mismatch while maintaining performance as policy staleness grows\. AtN=128N=128, GGPO also outperforms GRPO\+TIS under the same synchronization delay\. On fixed offline data, it leads GRPO, GRPO\+TIS, and AGRO at every evaluated checkpoint from 160 updates onward\.

The theory motivates the algorithm by explaining how small group loss and repeated reference updates improve reward, and how tokenwise mismatch can give a more favorable dependence than sequence coverage\. The theorem provides sufficient conditions for convergence and explains how errors in individual updates affect the final reward\. Its parameter settings need not coincide with those that work best in practice\. The experiments complement this analysis by evaluating GGPO under practical training settings and substantial behavior mismatch\.

Our results on mathematical reasoning with binary verifier rewards motivate further study of richer reward signals and multi\-turn tasks\. Extending the analysis to practical training choices and testing GGPO across these settings are natural directions for future work\.

## Acknowledgments

This work was supported in part by NSF grants 2416897, 2505932, and 2601666 and ONR grant N000142512318\. Computing resources were provided by the National Center for Supercomputing Applications \(NCSA\) through Delta \(NSF OAC\-2005572\) and DeltaAI \(NSF OAC\-2320345\), as well as Illinois Computes and the NAIRR Pilot \(NAIRR250157\)\.

Generative AI tools were used under the authors’ direction for assistance with literature searches, coding, conducting and analyzing experiments, deriving theoretical results and proofs, and polishing the manuscript\.

## References

- weq \(2025\)Hugging Face dataset, 2025\.URL[https://huggingface\.co/datasets/weqweasdas/from\_default\_filtered\_openr1\_with\_scores](https://huggingface.co/datasets/weqweasdas/from_default_filtered_openr1_with_scores)\.
- Agarwal et al\. \(2021\)Alekh Agarwal, Sham M\. Kakade, Jason D\. Lee, and Gaurav Mahajan\.On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift\.*Journal of Machine Learning Research*, 22\(98\):1–76, 2021\.
- Arnal et al\. \(2025\)Charles Arnal, Gaëtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Rémi Munos\.Asymmetric REINFORCE for off\-policy reinforcement learning: Balancing positive and negative rewards\.In*Advances in Neural Information Processing Systems*, 2025\.
- Bae et al\. \(2026\)Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, Jeongyeon Nam, and Donghyun Kwak\.Online difficulty filtering for reasoning oriented reinforcement learning\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 700–719, 2026\.
- Bai et al\. \(2026\)Hao Bai, Jiarui Yao, and Tong Zhang\.Enabling token\-wise importance sampling in off\-policy RLVR with proximal KL regularization\.*arXiv preprint arXiv:2610\.18738*, 2026\.
- Bhandari and Russo \(2024\)Jalaj Bhandari and Daniel Russo\.Global optimality guarantees for policy gradient methods\.*Operations Research*, 72\(5\):1906–1927, 2024\.
- Brantley et al\. \(2025\)Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D\. Lee, Wen Sun, Wenhao Zhan, and Xuezhou Zhang\.Accelerating RL for LLM reasoning with optimal advantage regression\.In*Advances in Neural Information Processing Systems*, volume 38, pages 151492–151531, 2025\.
- Chen et al\. \(2024\)Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu\.Noise contrastive alignment of language models with explicit rewards\.In*Advances in Neural Information Processing Systems*, 2024\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- Cheng et al\. \(2025\)Jorge \(Zhoujun\) Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Nilabjo Dey, Yonghao Zhuang, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Taylor W\. Killian, Haonan Li, Mikhail Yurochkin, Eric Xing, Zhengzhong Liu, and Zhiting Hu\.Revisiting reinforcement learning for LLM reasoning from a cross\-domain perspective\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.
- Dong et al\. \(2023\)Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang\.RAFT: Reward ranked finetuning for generative foundation model alignment\.*Transactions on Machine Learning Research*, 2023\.ISSN 2835\-8856\.
- Espeholt et al\. \(2018\)Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu\.IMPALA: Scalable distributed deep\-RL with importance weighted actor\-learner architectures\.In*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pages 1407–1416, 2018\.
- Flet\-Berliac et al\. \(2024\)Yannis Flet\-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist\.Contrastive policy gradient: Aligning LLMs on sequence\-level scores in a supervised\-friendly fashion\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 21353–21370, 2024\.
- Gao et al\. \(2024\)Zhaolin Gao, Jonathan D\. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J\. Andrew Bagnell, Jason D\. Lee, and Wen Sun\.REBEL: Reinforcement learning via regressing relative rewards\.In*Advances in Neural Information Processing Systems*, volume 37, pages 52354–52400, 2024\.
- Geist et al\. \(2019\)Matthieu Geist, Bruno Scherrer, and Olivier Pietquin\.A theory of regularized Markov decision processes\.In*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 2160–2169, 2019\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.*Nature*, 645\(8081\):633–638, 2025\.
- He et al\. \(2024\)Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun\.OlympiadBench: A challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, 2024\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the MATH dataset\.In*Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, volume 1, 2021\.
- Huang et al\. \(2026\)Luke J\. Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, and Song Han\.Stable asynchrony: Variance\-controlled off\-policy RL for LLMs\.In*Proceedings of the 43rd International Conference on Machine Learning*, volume 306 of*Proceedings of Machine Learning Research*, pages 48293–48311\. PMLR, 2026\.
- Kimi Team et al\. \(2025\)Kimi Team et al\.Kimi k1\.5: Scaling reinforcement learning with LLMs\.*arXiv preprint arXiv:2501\.12599*, 2025\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles*, 2023\.
- Kydlíček \(2025\)Hynek Kydlíček\.Math\-Verify: Math verification library\.Software repository, Hugging Face, 2025\.URL[https://github\.com/huggingface/Math\-Verify](https://github.com/huggingface/Math-Verify)\.
- Lewkowycz et al\. \(2022\)Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman\-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur\-Ari, and Vedant Misra\.Solving quantitative reasoning problems with language models\.In*Advances in Neural Information Processing Systems*, volume 35, 2022\.
- Li et al\. \(2026\)Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, and Baoxiang Wang\.Trust region masking for long\-horizon LLM reinforcement learning\.In*Proceedings of the 43rd International Conference on Machine Learning*, volume 306 of*Proceedings of Machine Learning Research*, pages 68742–68754\. PMLR, 2026\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*International Conference on Learning Representations*, 2024\.
- Ling Team et al\. \(2025\)Ling Team, Anqi Shen, Baihui Li, et al\.Every step evolves: Scaling reinforcement learning for trillion\-scale thinking model\.*arXiv preprint arXiv:2510\.18855*, 2025\.
- Liu et al\. \(2025\)Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding R1\-zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- Loshchilov and Hutter \(2019\)Ilya Loshchilov and Frank Hutter\.Decoupled weight decay regularization\.In*International Conference on Learning Representations*, 2019\.
- Marek and Ryabinin \(2026\)Martin Marek and Max Ryabinin\.Score centering stabilizes off\-policy reinforcement learning\.*arXiv preprint arXiv:2609\.20807*, 2026\.
- MiniMax et al\. \(2025\)MiniMax, Aili Chen, Aonian Li, Bangwei Gong, et al\.MiniMax\-M1: Scaling test\-time compute efficiently with lightning attention\.*arXiv preprint arXiv:2506\.13585*, 2025\.
- Nachum et al\. \(2017\)Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans\.Bridging the gap between value and policy based reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Nachum et al\. \(2018\)Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans\.Trust\-PCL: An off\-policy trust region method for continuous control\.In*International Conference on Learning Representations*, 2018\.
- Noukhovitch et al\. \(2025\)Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville\.Asynchronous RLHF: Faster and more efficient off\-policy RL for language models\.In*International Conference on Learning Representations*, 2025\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.
- Richemond and Maginnis \(2017\)Pierre H\. Richemond and Brendan Maginnis\.A short variational proof of equivalence between policy gradients and soft Q learning\.*arXiv preprint arXiv:1712\.08650*, 2017\.
- Richemond et al\. \(2024\)Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot\.Offline regularised reinforcement learning for large language models alignment\.*arXiv preprint arXiv:2405\.19107*, 2024\.
- Ritter et al\. \(2026\)Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang, Kianté Brantley, and Wen Sun\.LLMs can learn to reason via off\-policy RL\.*arXiv preprint arXiv:2602\.19362*, 2026\.
- Schulman \(2020\)John Schulman\.Approximating KL divergence\.Blog post, 2020\.URL[http://joschu\.net/blog/kl\-approx\.html](http://joschu.net/blog/kl-approx.html)\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Sheng et al\. \(2025\)Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu\.HybridFlow: A flexible and efficient RLHF framework\.In*Proceedings of the Twentieth European Conference on Computer Systems*, pages 1279–1297\. ACM, 2025\.
- Tajwar et al\. \(2026\)Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette\.Maximum likelihood reinforcement learning\.In*Proceedings of the 43rd International Conference on Machine Learning*, volume 306 of*Proceedings of Machine Learning Research*, pages 118190–118245\. PMLR, 2026\.
- Tang et al\. \(2026\)Yunhao Tang, Taco Cohen, David W\. Zhang, Michal Valko, and Rémi Munos\.RL\-finetuning LLMs from on\- and off\-policy data with a single algorithm\.In*Proceedings of the 29th International Conference on Artificial Intelligence and Statistics*, volume 300 of*Proceedings of Machine Learning Research*, pages 415–423\. PMLR, 2026\.
- van Erven and Harremoës \(2014\)Tim van Erven and Peter Harremoës\.Rényi divergence and Kullback–Leibler divergence\.*IEEE Transactions on Information Theory*, 60\(7\):3797–3820, 2014\.
- Xiong et al\. \(2025\)Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong\.A minimalist approach to LLM reasoning: from rejection sampling to reinforce\.*arXiv preprint arXiv:2504\.11343*, 2025\.
- Yang et al\. \(2025\)An Yang et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yu et al\. \(2025\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei\-Ying Ma, Ya\-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang\.DAPO: An open\-source LLM reinforcement learning system at scale\.In*Advances in Neural Information Processing Systems*, volume 38, pages 113222–113244, 2025\.
- Yuan et al\. \(2023\)Rui Yuan, Simon S\. Du, Robert M\. Gower, Alessandro Lazaric, and Lin Xiao\.Linear convergence of natural policy gradient methods with log\-linear policies\.In*International Conference on Learning Representations*, 2023\.
- Zhang et al\. \(2026a\)Feng Zhang, Xinhong Ma, Ziqiang Dong, Xi Leng, Jianfei Zhao, Xin Sun, Yang Yang, and Guanjun Jiang\.Revisiting reinforcement learning with verifiable rewards from a contrastive perspective\.*arXiv preprint arXiv:2605\.12969*, 2026a\.
- Zhang et al\. \(2026b\)Yuheng Zhang, Chenlu Ye, Shuowei Jin, Changlong Yu, Wei Xiong, Saurabh Sahu, and Nan Jiang\.Rethinking importance sampling in LLM policy optimization: A cumulative token perspective\.*arXiv preprint arXiv:2605\.07331*, 2026b\.
- Zheng et al\. \(2025a\)Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, and Junyang Lin\.Stabilizing reinforcement learning with LLMs: Formulation and practices\.*arXiv preprint arXiv:2512\.01374*, 2025a\.
- Zheng et al\. \(2025b\)Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin\.Group sequence policy optimization\.*arXiv preprint arXiv:2507\.18071*, 2025b\.
- Zhong et al\. \(2026\)Tianle Zhong, Neiwen Ling, Yifan Pi, Zijun Wei, Tianshu Yu, Geoffrey Fox, Peng Wu, and Xiao Yu\.Diagnosing training inference mismatch in LLM reinforcement learning\.*arXiv preprint arXiv:2605\.14220*, 2026\.

## Appendix AProofs and extensions

The proofs below explain the connections used in the main text, from the population optimality condition to finite\-group reward guarantees\. We work with a finite action space𝒜\\mathcal\{A\}, with policies positive on𝒜\\mathcal\{A\}and finite displayed expectations\. In particular,0<Zβ=𝔼πref​er/β<∞0<Z\_\{\\beta\}=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{ref\}\}\}e^\{r/\\beta\}<\\infty\. We suppress fixed prompts in pointwise derivations and write prompt averages explicitly\. Loss notation suppresses a fixed temperature and reference;Hm​\(πθ,πbeh,x,πref\)H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x;\\pi\_\{\\mathrm\{ref\}\}\)specifies the reference when it varies\.

### A\.1Proof of population duality and coverage

This proves Proposition[1](https://arxiv.org/html/2610.20814#Thmtheorem1), explaining both the loss construction in Section[3](https://arxiv.org/html/2610.20814#S3)and the coverage condition in its regularized\-gap guarantee\. The argument begins with the scalar dual, whose residual provides a measure of departure from the Gibbs optimum\. Profiling this residual gives the contrastive loss, and coverage then connects the behavior\-policy expectation to the learner’s regularized gap\.

###### Lemma 4\(Scalar duality and primal–dual gap\)\.

Fix a promptxx, temperatureβ\>0\\beta\>0, rewardr⁡\(x,a\)r\(x,a\), and positive reference policyπref\(⋅∣x\)\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\. DefineVβ\(π;x\)=𝔼a∼πr\(x,a\)−βDKL\(π∥πref\)V\_\{\\beta\}\(\\pi;x\)=\\mathbb\{E\}\_\{a\\sim\\pi\}r\(x,a\)\-\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi\_\{\\mathrm\{ref\}\}\),Vβ∗​\(x\)=maxπ⁡Vβ​\(π,x\)V\_\{\\beta\}^\{\*\}\(x\)=\\max\_\{\\pi\}V\_\{\\beta\}\(\\pi;x\), andZβ=𝔼a∼πref​er⁡\(x,a\)/βZ\_\{\\beta\}=\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mathrm\{ref\}\}\}e^\{r\(x,a\)/\\beta\}\. For a positive candidateπθ\\pi\_\{\\theta\}, defineRθ​\(x,a\)=r⁡\(x,a\)−β​log⁡\[πθ​\(a∣x\)/πref​\(a∣x\)\]R\_\{\\theta\}\(x,a\)=r\(x,a\)\-\\beta\\log\[\\pi\_\{\\theta\}\(a\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\\mid x\)\],h⁡\(z\)=ez−z−1h\(z\)=e^\{z\}\-z\-1, and

Φx​\(ν\)\\displaystyle\\Phi\_\{x\}\(\\nu\):=ν\+β⁡\(𝔼a∼πref​e\(r⁡\(x,a\)−ν\)/β−1\),\\displaystyle:=\\nu\+\\beta\\bigl\(\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mathrm\{ref\}\}\}e^\{\(r\(x,a\)\-\\nu\)/\\beta\}\-1\\bigr\),\(16\)maxπ⁡Vβ​\(π,x\)⏟primal\\displaystyle\\underbrace\{\\max\_\{\\pi\}V\_\{\\beta\}\(\\pi;x\)\}\_\{\\text\{primal\}\}=β​log⁡Zβ=minν∈ℝ⁡Φx​\(ν\)⏟scalar dual,\\displaystyle=\\beta\\log Z\_\{\\beta\}=\\underbrace\{\\min\_\{\\nu\\in\\mathbb\{R\}\}\\Phi\_\{x\}\(\\nu\)\}\_\{\\text\{scalar dual\}\},Ψx​\(πθ,ν\)\\displaystyle\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\):=Φx​\(ν\)−Vβ​\(πθ,x\)\.\\displaystyle:=\\Phi\_\{x\}\(\\nu\)\-V\_\{\\beta\}\(\\pi\_\{\\theta\};x\)\.ThenΨx​\(πθ,ν\)≥0\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\)\\geq 0,minν⁡Ψx​\(πθ,ν\)=Vβ∗​\(x\)−Vβ​\(πθ,x\)\\min\_\{\\nu\}\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\)=V\_\{\\beta\}^\{\*\}\(x\)\-V\_\{\\beta\}\(\\pi\_\{\\theta\};x\), and

Ψx​\(πθ,ν\)=β​𝔼a∼πθ​h​\(\(Rθ​\(x,a\)−ν\)/β\)\.\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\)=\\beta\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\}\}h\(\(R\_\{\\theta\}\(x,a\)\-\\nu\)/\\beta\)\.\(17\)

###### Proof\.

The Gibbs policyπ∗​\(a∣x\)=πref​\(a∣x\)​er⁡\(x,a\)/β/Zβ\\pi^\{\*\}\(a\\mid x\)=\\pi\_\{\\mathrm\{ref\}\}\(a\\mid x\)e^\{r\(x,a\)/\\beta\}/Z\_\{\\beta\}satisfies

βDKL\(π∥π∗\)=βlogZβ−Vβ\(π;x\)\.\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\\\|\\pi^\{\*\}\)=\\beta\\log Z\_\{\\beta\}\-V\_\{\\beta\}\(\\pi;x\)\.HenceVβ∗​\(x\)=β​log⁡ZβV\_\{\\beta\}^\{\*\}\(x\)=\\beta\\log Z\_\{\\beta\}andπ∗\\pi^\{\*\}is its unique maximizer\. Moreover,Φx′\(ν\)=1−Zβe−ν/β\\Phi\_\{x\}^\{\\prime\}\(\\nu\)=1\-Z\_\{\\beta\}e^\{\-\\nu/\\beta\}andΦx′′\(ν\)=Zβe−ν/β/β\>0\\Phi\_\{x\}^\{\\prime\\prime\}\(\\nu\)=Z\_\{\\beta\}e^\{\-\\nu/\\beta\}/\\beta\>0, so its minimum isΦx​\(β​log⁡Zβ\)=Vβ∗​\(x\)\\Phi\_\{x\}\(\\beta\\log Z\_\{\\beta\}\)=V\_\{\\beta\}^\{\*\}\(x\)\. HenceΦx​\(ν\)\\Phi\_\{x\}\(\\nu\)upper\-bounds the primal optimum for everyν\\nu, soΨx\\Psi\_\{x\}is a nonnegative primal–dual gap, andminν⁡Ψx​\(πθ,ν\)=Vβ∗​\(x\)−Vβ​\(πθ,x\)\\min\_\{\\nu\}\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\)=V\_\{\\beta\}^\{\*\}\(x\)\-V\_\{\\beta\}\(\\pi\_\{\\theta\};x\)\. Finally,𝔼πθ​Rθ=Vβ​\(πθ,x\)\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}R\_\{\\theta\}=V\_\{\\beta\}\(\\pi\_\{\\theta\};x\)andπθ​e\(Rθ−ν\)/β=πref​e\(r−ν\)/β\\pi\_\{\\theta\}e^\{\(R\_\{\\theta\}\-\\nu\)/\\beta\}=\\pi\_\{\\mathrm\{ref\}\}e^\{\(r\-\\nu\)/\\beta\}\. Expandinghhproves Eq\. \([17](https://arxiv.org/html/2610.20814#A1.E17)\)\. ∎

###### Proof of Proposition[1](https://arxiv.org/html/2610.20814#Thmtheorem1)\.

Fixxxand use the quantities defined in Lemma[4](https://arxiv.org/html/2610.20814#Thmtheorem4)\. Replacing the learner expectation byπbeh\\pi\_\{\\mathrm\{beh\}\}and minimizing overν\\nugivesνθ=β​log⁡𝔼πbeh​eRθ/β\\nu\_\{\\theta\}=\\beta\\log\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}e^\{R\_\{\\theta\}/\\beta\}: the first derivative is1−𝔼πbeh​e\(Rθ−ν\)/β1\-\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}e^\{\(R\_\{\\theta\}\-\\nu\)/\\beta\}and the second is strictly positive\. Substitution gives𝒥beh=νθ−𝔼πbeh​Rθ\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\\nu\_\{\\theta\}\-\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}R\_\{\\theta\}, the loss in Eq\. \([3](https://arxiv.org/html/2610.20814#S3.E3)\)\. Define

qθ​\(a∣x\)=πbeh​\(a∣x\)​eRθ​\(x,a\)/β𝔼a′∼πbeh​eRθ​\(x,a′\)/β\.q\_\{\\theta\}\(a\\mid x\)=\\frac\{\\pi\_\{\\mathrm\{beh\}\}\(a\\mid x\)e^\{R\_\{\\theta\}\(x,a\)/\\beta\}\}\{\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\pi\_\{\\mathrm\{beh\}\}\}e^\{R\_\{\\theta\}\(x,a^\{\\prime\}\)/\\beta\}\}\.\(18\)Expanding its KL divergence givesβDKL\(πbeh∥qθ\)=𝒥beh\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{beh\}\}\\\|q\_\{\\theta\}\)=\\mathcal\{J\}\_\{\\mathrm\{beh\}\}, proving Eq\. \([4](https://arxiv.org/html/2610.20814#S3.E4)\)\. Nonnegativity ofhhandπθ/πbeh≤Cx\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{beh\}\}\\leq C\_\{x\}now imply

Vβ∗−Vβ​\(πθ,x\)≤Ψx​\(πθ,νθ\)≤Cx​β​𝔼πbeh​h​\(\(Rθ−νθ\)/β\)=Cx​𝒥beh\.V\_\{\\beta\}^\{\*\}\-V\_\{\\beta\}\(\\pi\_\{\\theta\};x\)\\leq\\Psi\_\{x\}\(\\pi\_\{\\theta\},\\nu\_\{\\theta\}\)\\leq C\_\{x\}\\beta\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}h\(\(R\_\{\\theta\}\-\\nu\_\{\\theta\}\)/\\beta\)=C\_\{x\}\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\.Whenπbeh=πθ\\pi\_\{\\mathrm\{beh\}\}=\\pi\_\{\\theta\}, profiling the same learner residual gives equality with the regularized gap\. Under positive support, zero loss forcesRθR\_\{\\theta\}constant, soπθ∝πref​er/β\\pi\_\{\\theta\}\\propto\\pi\_\{\\mathrm\{ref\}\}e^\{r/\\beta\}and henceπθ=π∗\\pi\_\{\\theta\}=\\pi^\{\*\}\. Conversely, this Gibbs policy has constant adjusted reward and zero loss\. ∎

Averaging the coverage bound over prompts gives𝔼X​\[Vβ∗​\(X\)−Vβ​\(πθ,X\)\]≤𝔼X​\[CX​𝒥beh​\(πθ,X\)\]\\mathbb\{E\}\_\{X\}\[V\_\{\\beta\}^\{\*\}\(X\)\-V\_\{\\beta\}\(\\pi\_\{\\theta\};X\)\]\\leq\\mathbb\{E\}\_\{X\}\[C\_\{X\}\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\(\\pi\_\{\\theta\};X\)\]; moving coverage outside the expectation requires a uniform bound\. Appendix[B\.5](https://arxiv.org/html/2610.20814#A2.SS5.SSS0.Px1)compares sequence coverage with aggregate tokenwise mismatch\.

### A\.2Finite\-group guarantees

We establish the common zero set and population limit used by the population and reference\-refresh results, then state and prove the best\-of\-KKbound of Section[4\.4](https://arxiv.org/html/2610.20814#S4.SS4)\.

###### Proposition 5\(Finite\-group target\)\.

Suppose thatRθR\_\{\\theta\}is bounded on the support ofπbeh\\pi\_\{\\mathrm\{beh\}\}and thata1,…,ama\_\{1\},\\ldots,a\_\{m\}are sampled independently fromπbeh\\pi\_\{\\mathrm\{beh\}\}\. Then, for every integerm≥1m\\geq 1,

0=H1≤Hm≤Hm\+1≤𝒥beh,Hm⟶𝒥beh\.0=H\_\{1\}\\leq H\_\{m\}\\leq H\_\{m\+1\}\\leq\\mathcal\{J\}\_\{\\mathrm\{beh\}\},\\qquad H\_\{m\}\\longrightarrow\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\.\(19\)For everym≥2m\\geq 2,Hm=0H\_\{m\}=0if and only ifRθR\_\{\\theta\}is constant on the support ofπbeh\\pi\_\{\\mathrm\{beh\}\}, equivalently if and only if𝒥beh=0\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=0\. Ifπθ\\pi\_\{\\theta\},πref\\pi\_\{\\mathrm\{ref\}\}, andπbeh\\pi\_\{\\mathrm\{beh\}\}are positive on𝒜\\mathcal\{A\}, this zero condition also impliesπθ=π∗\\pi\_\{\\theta\}=\\pi^\{\*\}\.

###### Proof\.

LetX=eRθ/βX=e^\{R\_\{\\theta\}/\\beta\}andX¯m=m−1​∑i=1mXi\\bar\{X\}\_\{m\}=m^\{\-1\}\\sum\_\{i=1\}^\{m\}X\_\{i\}for i\.i\.d\. copiesXiX\_\{i\}\. Then

Hm=β⁡\(𝔼​log​X¯m−𝔼​log​X\),𝒥beh=β⁡\(log⁡𝔼​X−𝔼​log​X\)\.H\_\{m\}=\\beta\(\\mathbb\{E\}\\log\\bar\{X\}\_\{m\}\-\\mathbb\{E\}\\log X\),\\qquad\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\\beta\(\\log\\mathbb\{E\}X\-\\mathbb\{E\}\\log X\)\.\(20\)Jensen gives0=H1≤Hm≤𝒥beh0=H\_\{1\}\\leq H\_\{m\}\\leq\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\. Givenm\+1m\+1samples, the leave\-one\-out meansAj=m−1​∑i≠jXiA\_\{j\}=m^\{\-1\}\\sum\_\{i\\neq j\}X\_\{i\}average toX¯m\+1\\bar\{X\}\_\{m\+1\}\. Thuslog⁡X¯m\+1≥\(m\+1\)−1​∑jlog⁡Aj\\log\\bar\{X\}\_\{m\+1\}\\geq\(m\+1\)^\{\-1\}\\sum\_\{j\}\\log A\_\{j\}; taking expectations givesHm\+1≥HmH\_\{m\+1\}\\geq H\_\{m\}, since eachAjA\_\{j\}is distributed as the size\-mmsample meanX¯m\\bar\{X\}\_\{m\}\. The bounded i\.i\.d\. sample averageX¯m\\bar\{X\}\_\{m\}converges to𝔼​X\\mathbb\{E\}X, and boundedness ofRθR\_\{\\theta\}boundslog⁡X¯m\\log\\bar\{X\}\_\{m\}, so dominated convergence givesHm→𝒥behH\_\{m\}\\to\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\.

Form≥2m\\geq 2, equality in the group Jensen inequality requires all sampledXiX\_\{i\}to agree\. HenceHm=0H\_\{m\}=0implies two independent copies always agree on the sampled support, soXXandRθR\_\{\\theta\}are constant there\. The converse is immediate; the population Jensen equality gives the same zero set for𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\. Under positivity on𝒜\\mathcal\{A\}, constancy givesπθ∝πref​er/β\\pi\_\{\\theta\}\\propto\\pi\_\{\\mathrm\{ref\}\}e^\{r/\\beta\}, proving the final claim\. ∎

#### Best\-of\-KKreward bound\.

WriteJK​\(π,x\)J\_\{K\}\(\\pi;x\)andJ⁡\(π,x\)=J1​\(π,x\)J\(\\pi;x\)=J\_\{1\}\(\\pi;x\)for the expected best\-of\-KKand single\-response rewards of a policyπ\\piconditional on promptxx, soJK​\(π\)=𝔼X​JK​\(π,X\)J\_\{K\}\(\\pi\)=\\mathbb\{E\}\_\{X\}J\_\{K\}\(\\pi;X\)in Eq\. \([13](https://arxiv.org/html/2610.20814#S4.E13)\), and letLm​\(πθ\)=𝔼X∼ρ​Hm​\(πθ,πbeh,X\)L\_\{m\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{X\\sim\\rho\}H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},X\)\. Forpx,t=PrA∼πbeh\(⋅∣x\)\[r\(x,A\)\>t\]p\_\{x,t\}=\\Pr\_\{A\\sim\\pi\_\{\\mathrm\{beh\}\}\(\\cdot\\mid x\)\}\[r\(x,A\)\>t\], define

κK,m:=max\{1,supx,t:px,t\>01−\(1−px,t\)K1−\(1−px,t\)m−1\}≤max\{1,Km−1\}\.\\kappa\_\{K,m\}:=\\max\\left\\\{1,\\sup\_\{x,t:\\,p\_\{x,t\}\>0\}\\frac\{1\-\(1\-p\_\{x,t\}\)^\{K\}\}\{1\-\(1\-p\_\{x,t\}\)^\{m\-1\}\}\\right\\\}\\leq\\max\\left\\\{1,\\frac\{K\}\{m\-1\}\\right\\\}\.\(21\)The inequality follows because\[1−\(1−p\)k\]/k\[1\-\(1\-p\)^\{k\}\]/kdecreases inkk\. WhenK\>m−1K\>m\-1, the upper bound is approached as a relevant behavior\-policy upper\-tail probability tends to zero, but the actual multiplier can be smaller\.

###### Proposition 6\(Best\-of\-KKand the GGPO minimizer\)\.

Letr∈\[0,1\]r\\in\[0,1\], fix positive behavior and reference policies withΓx≤Γ\\Gamma\_\{x\}\\leq\\Gammaat every prompt, whereΓx\\Gamma\_\{x\}is the oscillation oflog\[πref\(⋅∣x\)/πbeh\(⋅∣x\)\]\\log\[\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)/\\pi\_\{\\mathrm\{beh\}\}\(\\cdot\\mid x\)\], and suppose every policyπθ\\pi\_\{\\theta\}in a positive hypothesis classΠ\\Pisatisfiesπθ/πbeh≤C\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{beh\}\}\\leq Cuniformly\. Form≥2m\\geq 2andK≥1K\\geq 1,

JK​\(πbeh\)−J⁡\(πθ\)≤κK,m​\{β⁡\(log⁡m\+Γ\)\+C​Lm​\(πθ\)\}\.J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\}\)\-J\(\\pi\_\{\\theta\}\)\\leq\\kappa\_\{K,m\}\\\{\\beta\(\\log m\+\\Gamma\)\+CL\_\{m\}\(\\pi\_\{\\theta\}\)\\\}\.\(22\)In particular, withαm=infπθ∈ΠLm​\(πθ\)\\alpha\_\{m\}=\\inf\_\{\\pi\_\{\\theta\}\\in\\Pi\}L\_\{m\}\(\\pi\_\{\\theta\}\), a population minimizer, if one exists, satisfies this bound withLmL\_\{m\}replaced byαm\\alpha\_\{m\}; an approximate minimizer with loss at mostαm\+ζ\\alpha\_\{m\}\+\\zetaincurs an additionalC​κK,m​ζC\\kappa\_\{K,m\}\\zeta\.

###### Proof\.

Fix a prompt and writeCx=maxa⁡πθ​\(a∣x\)/πbeh​\(a∣x\)C\_\{x\}=\\max\_\{a\}\\pi\_\{\\theta\}\(a\\mid x\)/\\pi\_\{\\mathrm\{beh\}\}\(a\\mid x\)\. We first prove the prompt\-level inequality

JK​\(πbeh,x\)−J⁡\(πθ,x\)≤κK,m​\{β⁡\(log⁡m\+Γx\)\+Cx​Hm​\(πθ,πbeh,x\)\}\.J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\};x\)\-J\(\\pi\_\{\\theta\};x\)\\leq\\kappa\_\{K,m\}\\\{\\beta\(\\log m\+\\Gamma\_\{x\}\)\+C\_\{x\}H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x\)\\\}\.\(23\)The multiplier defined in Eq\. \([21](https://arxiv.org/html/2610.20814#A1.E21)\) satisfies the required tail comparison at every prompt\. Suppress the prompt\. Putwi=πθ​\(ai\)/πbeh​\(ai\)w\_\{i\}=\\pi\_\{\\theta\}\(a\_\{i\}\)/\\pi\_\{\\mathrm\{beh\}\}\(a\_\{i\}\),S=m−1​∑iwiS=m^\{\-1\}\\sum\_\{i\}w\_\{i\},W=maxi⁡wiW=\\max\_\{i\}w\_\{i\}, andPi=wi/\(m​S\)P\_\{i\}=w\_\{i\}/\(mS\)\. Letui=ri\+β​log⁡\[πref​\(ai\)/πbeh​\(ai\)\]u\_\{i\}=r\_\{i\}\+\\beta\\log\[\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\)/\\pi\_\{\\mathrm\{beh\}\}\(a\_\{i\}\)\]andzi=ui/β−log⁡wi=Ri/βz\_\{i\}=u\_\{i\}/\\beta\-\\log w\_\{i\}=R\_\{i\}/\\beta\. WithUUuniform on sample indices,c=log⁡𝔼U​ezc=\\log\\mathbb\{E\}\_\{U\}e^\{z\}, andg⁡\(t\)=et−t−1≥0g\(t\)=e^\{t\}\-t\-1\\geq 0, one has𝔼U​g​\(z−c\)=𝒥^m/β\\mathbb\{E\}\_\{U\}g\(z\-c\)=\\widehat\{\\mathcal\{J\}\}\_\{m\}/\\beta\. DefineQi∝eui/βQ\_\{i\}\\propto e^\{u\_\{i\}/\\beta\}andA=𝔼P​ezA=\\mathbb\{E\}\_\{P\}e^\{z\}\. SinceQi=Pi​ezi/AQ\_\{i\}=P\_\{i\}e^\{z\_\{i\}\}/A,

𝔼Pg\(z−c\)−DKL\(P∥Q\)=Ae−c−1−log\(Ae−c\)≥0\.\\mathbb\{E\}\_\{P\}g\(z\-c\)\-D\_\{\\mathrm\{KL\}\}\(P\\\|Q\)=Ae^\{\-c\}\-1\-\\log\(Ae^\{\-c\}\)\\geq 0\.The boundPi/Ui≤W/SP\_\{i\}/U\_\{i\}\\leq W/Stherefore givesβSDKL\(P∥Q\)≤W𝒥^m\\beta SD\_\{\\mathrm\{KL\}\}\(P\\\|Q\)\\leq W\\widehat\{\\mathcal\{J\}\}\_\{m\}\. The finite Gibbs identity yieldsmaxiui−𝔼Pu≤βlogm\+βDKL\(P∥Q\)\\max\_\{i\}u\_\{i\}\-\\mathbb\{E\}\_\{P\}u\\leq\\beta\\log m\+\\beta D\_\{\\mathrm\{KL\}\}\(P\\\|Q\)\. Indeed, expandingDKL\(P∥Q\)D\_\{\\mathrm\{KL\}\}\(P\\\|Q\), using∑iPi​log⁡Pi≥−log⁡m\\sum\_\{i\}P\_\{i\}\\log P\_\{i\}\\geq\-\\log m, and lower\-bounding the log\-partition bymaxi⁡ui/β\\max\_\{i\}u\_\{i\}/\\betagives this inequality\. The oscillation ofu−ru\-ris at mostβ​Γx\\beta\\Gamma\_\{x\}, hencemaxi⁡ri−𝔼P​r≤maxi⁡ui−𝔼P​u\+β​Γx\\max\_\{i\}r\_\{i\}\-\\mathbb\{E\}\_\{P\}r\\leq\\max\_\{i\}u\_\{i\}\-\\mathbb\{E\}\_\{P\}u\+\\beta\\Gamma\_\{x\}\. Multiplying bySSand usingS​𝔼P​r=m−1​∑iwi​riS\\mathbb\{E\}\_\{P\}r=m^\{\-1\}\\sum\_\{i\}w\_\{i\}r\_\{i\}together with the preceding KL bound gives

S​maxi​ri−1m​∑iwi​ri≤β​S​\(log⁡m\+Γx\)\+W​𝒥^m\.S\\max\_\{i\}r\_\{i\}\-\\frac\{1\}\{m\}\\sum\_\{i\}w\_\{i\}r\_\{i\}\\leq\\beta S\(\\log m\+\\Gamma\_\{x\}\)\+W\\widehat\{\\mathcal\{J\}\}\_\{m\}\.\(24\)ForF=maxi⁡riF=\\max\_\{i\}r\_\{i\}, symmetry and importance weighting give𝔼πbehm​\[S​F\]=𝔼πθ⊗πbehm−1​F\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}^\{m\}\}\[SF\]=\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\\otimes\\pi\_\{\\mathrm\{beh\}\}^\{m\-1\}\}F, while𝔼​S=1\\mathbb\{E\}S=1and𝔼⁡\[m−1​∑iwi​ri\]=𝔼πθ​r\\mathbb\{E\}\[m^\{\-1\}\\sum\_\{i\}w\_\{i\}r\_\{i\}\]=\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}r\. At a thresholdt∈\[0,1\]t\\in\[0,1\], writep=πbeh\{r\>t\}p=\\pi\_\{\\mathrm\{beh\}\}\\\{r\>t\\\},v=πθ\{r\>t\}v=\\pi\_\{\\theta\}\\\{r\>t\\\}, anddk=1−\(1−p\)kd\_\{k\}=1\-\(1\-p\)^\{k\}\. By the defining tail comparison,

dK−v≤dK​\(1−v\)≤κK,m​dm−1​\(1−v\)\.d\_\{K\}\-v\\leq d\_\{K\}\(1\-v\)\\leq\\kappa\_\{K,m\}d\_\{m\-1\}\(1\-v\)\.The layer\-cake identities here are

JK​\(πbeh,x\)=∫01dK​𝑑t,J⁡\(πθ,x\)=∫01v​𝑑t,J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\};x\)=\\int\_\{0\}^\{1\}d\_\{K\}\\,dt,\\qquad J\(\\pi\_\{\\theta\};x\)=\\int\_\{0\}^\{1\}v\\,dt,and, because the mixed maximum has tail1−\(1−v\)​\(1−p\)m−1=v\+\(1−v\)​dm−11\-\(1\-v\)\(1\-p\)^\{m\-1\}=v\+\(1\-v\)d\_\{m\-1\},

𝔼⁡\[S​F\]−𝔼πθ​r=∫01\(1−v\)​dm−1​𝑑t\.\\mathbb\{E\}\[SF\]\-\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}r=\\int\_\{0\}^\{1\}\(1\-v\)d\_\{m\-1\}\\,dt\.Integrating the threshold inequality therefore boundsJK​\(πbeh,x\)−J⁡\(πθ,x\)J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\};x\)\-J\(\\pi\_\{\\theta\};x\)byκK,m​\(𝔼⁡\[S​F\]−𝔼πθ​r\)\\kappa\_\{K,m\}\(\\mathbb\{E\}\[SF\]\-\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}r\)\. Take expectations in Eq\. \([24](https://arxiv.org/html/2610.20814#A1.E24)\) and useW≤CxW\\leq C\_\{x\}\. Averaging Eq\. \([23](https://arxiv.org/html/2610.20814#A1.E23)\) over prompts, withCx≤CC\_\{x\}\\leq CandΓx≤Γ\\Gamma\_\{x\}\\leq\\Gamma, proves the proposition\. The minimizer statements follow by substituting the corresponding population losses\. ∎

On policy,πθ=πbeh\\pi\_\{\\theta\}=\\pi\_\{\\mathrm\{beh\}\}andCx=1C\_\{x\}=1at every prompt; withπref=πbeh\\pi\_\{\\mathrm\{ref\}\}=\\pi\_\{\\mathrm\{beh\}\},Γ=0\\Gamma=0\. The class\-wide coverage boundCCcan still exceed one for other candidates\. The minimum loss is zero when the Gibbs policy belongs to the admissible class; a restricted class or coverage cap may prevent this, and the regularization term in the reward bound remains even at zero loss\.

### A\.3Generalization and an oracle inequality

Theorem[7](https://arxiv.org/html/2610.20814#Thmtheorem7)below converts expected group loss into an observed\-loss bound\. The oracle inequality then states what approximate empirical minimization achieves in a restricted hypothesis class\. Fix a prompt distributionρ\\rho, behaviorπbeh\\pi\_\{\\mathrm\{beh\}\}, referenceπref\\pi\_\{\\mathrm\{ref\}\}, temperatureβ\>0\\beta\>0, and hypothesis classΠ\\Pibefore collectingnni\.i\.d\. unfiltered groups\. Forfπθ​\(Z\)=𝒥^m​\(πθ,Z\)f\_\{\\pi\_\{\\theta\}\}\(Z\)=\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};Z\), writeℱ=\{fπθ:πθ∈Π\}\\mathcal\{F\}=\\\{f\_\{\\pi\_\{\\theta\}\}:\\pi\_\{\\theta\}\\in\\Pi\\\}and letLm​\(πθ\)=𝔼​fπθ=𝔼X∼ρ​Hm​\(πθ,πbeh,X\)L\_\{m\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}f\_\{\\pi\_\{\\theta\}\}=\\mathbb\{E\}\_\{X\\sim\\rho\}H\_\{m\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},X\)andL^n​\(πθ\)=n−1​∑jfπθ​\(Zj\)\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)=n^\{\-1\}\\sum\_\{j\}f\_\{\\pi\_\{\\theta\}\}\(Z\_\{j\}\)\. Assume0≤fπθ≤Λ0\\leq f\_\{\\pi\_\{\\theta\}\}\\leq\\Lambdaand that the induced loss family admits a finite uniformη\\eta\-cover in the distance of Eq\. \([41](https://arxiv.org/html/2610.20814#A2.E41)\), with centers in the family: each loss is withinη\\etaof a center on every possible group\. Write

z=log⁡2​𝒩∞​\(η,ℱ\)δ,γn=4​Λ​z/n\+3​η\.z=\\log\\frac\{2\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\mathcal\{F\}\)\}\{\\delta\},\\qquad\\gamma\_\{n\}=4\\Lambda z/n\+3\\eta\.A sufficient envelope isΛ=1\+β​B\\Lambda=1\+\\beta Bwhenosca⁡log⁡\[πθ​\(a∣x\)/πref​\(a∣x\)\]≤B\\operatorname\{osc\}\_\{a\}\\log\[\\pi\_\{\\theta\}\(a\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\\mid x\)\]\\leq Buniformly, since adjusted rewards then have range at mostΛ\\Lambda\. These assumptions apply to all possible groups, not only observed responses\. Coverage remains separate\.

###### Theorem 7\(Empirical GGPO bound\)\.

Under Proposition[6](https://arxiv.org/html/2610.20814#Thmtheorem6)’s assumptions and the sampling and cover conditions above, with probability at least1−δ1\-\\delta, every data\-selectedπθ^∈Π\\pi\_\{\\hat\{\\theta\}\}\\in\\Pisatisfies

JK​\(πbeh\)−J⁡\(πθ^\)≤κK,m​\{β⁡\(log⁡m\+Γ\)\+C⁡\[2​L^n​\(πθ^\)\+γn\]\}\.J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\}\)\-J\(\\pi\_\{\\hat\{\\theta\}\}\)\\leq\\kappa\_\{K,m\}\\\{\\beta\(\\log m\+\\Gamma\)\+C\[2\\widehat\{L\}\_\{n\}\(\\pi\_\{\\hat\{\\theta\}\}\)\+\\gamma\_\{n\}\]\\\}\.\(25\)

The bound applies to the output of GGPO; it requires no convexity or successful empirical minimization\. The reported loss uses unfiltered groups in the original units, rather than the filtered, normalized training log\.

###### Proposition 8\(Raw\-loss oracle inequality and reward consequence\)\.

Fix the behavior and reference policies and candidate class before drawingnnindependent unfiltered groups, with raw losses in\[0,Λ\]\[0,\\Lambda\]and the uniform loss cover specified above\. Suppose the returned policy satisfies

L^n​\(πθ^\)≤infπθ∈ΠL^n​\(πθ\)\+ξ\.\\widehat\{L\}\_\{n\}\(\\pi\_\{\\hat\{\\theta\}\}\)\\leq\\inf\_\{\\pi\_\{\\theta\}\\in\\Pi\}\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)\+\\xi\.Writeαm=infπθ∈ΠLm​\(πθ\)\\alpha\_\{m\}=\\inf\_\{\\pi\_\{\\theta\}\\in\\Pi\}L\_\{m\}\(\\pi\_\{\\theta\}\)for the best population raw loss in the class\. With probability at least1−δ1\-\\delta,

Lm​\(πθ^\)≤4​αm\+2​ξ\+3​γn\.L\_\{m\}\(\\pi\_\{\\hat\{\\theta\}\}\)\\leq 4\\alpha\_\{m\}\+2\\xi\+3\\gamma\_\{n\}\.\(26\)If the reward\-comparison assumptions of Proposition[6](https://arxiv.org/html/2610.20814#Thmtheorem6)also hold, then on the same event,

JK​\(πbeh\)−J⁡\(πθ^\)≤κK,m​\{β⁡\(log⁡m\+Γ\)\+C⁡\[4​αm\+2​ξ\+3​γn\]\}\.J\_\{K\}\(\\pi\_\{\\mathrm\{beh\}\}\)\-J\(\\pi\_\{\\hat\{\\theta\}\}\)\\leq\\kappa\_\{K,m\}\\\{\\beta\(\\log m\+\\Gamma\)\+C\[4\\alpha\_\{m\}\+2\\xi\+3\\gamma\_\{n\}\]\\\}\.\(27\)

###### Proof\.

We first compare population and empirical raw losses uniformly over the candidate policies\. A cover center is itself the raw loss of a candidate policy, sayπ′\\pi^\{\\prime\}\. Its loss on one random groupZZis nonnegative and bounded byΛ\\Lambda, so its variance satisfies

Var⁡\(fπ′​\(Z\)\)≤𝔼⁡\[fπ′​\(Z\)2\]≤Λ​Lm​\(π′\)\.\\operatorname\{Var\}\(f\_\{\\pi^\{\\prime\}\}\(Z\)\)\\leq\\mathbb\{E\}\[f\_\{\\pi^\{\\prime\}\}\(Z\)^\{2\}\]\\leq\\Lambda L\_\{m\}\(\\pi^\{\\prime\}\)\.Thus policies with small population loss also have small loss variance\. Bernstein’s inequality gives, with failure probability at most2​e−z2e^\{\-z\},

\|L^n​\(π′\)−Lm​\(π′\)\|≤2​Λ​Lm​\(π′\)​z/n\+2​Λ​z3​n≤Lm​\(π′\)2\+5​Λ​z3​n\.\|\\widehat\{L\}\_\{n\}\(\\pi^\{\\prime\}\)\-L\_\{m\}\(\\pi^\{\\prime\}\)\|\\leq\\sqrt\{2\\Lambda L\_\{m\}\(\\pi^\{\\prime\}\)z/n\}\+\\frac\{2\\Lambda z\}\{3n\}\\leq\\frac\{L\_\{m\}\(\\pi^\{\\prime\}\)\}\{2\}\+\\frac\{5\\Lambda z\}\{3n\}\.The last inequality absorbs the square\-root term into half of the population loss and a term of order1/n1/n\. Rearranging and rounding10/310/3up to44gives both

Lm​\(π′\)≤2​L^n​\(π′\)\+4​Λ​z/n,L^n​\(π′\)≤2​Lm​\(π′\)\+4​Λ​z/n\.L\_\{m\}\(\\pi^\{\\prime\}\)\\leq 2\\widehat\{L\}\_\{n\}\(\\pi^\{\\prime\}\)\+4\\Lambda z/n,\\qquad\\widehat\{L\}\_\{n\}\(\\pi^\{\\prime\}\)\\leq 2L\_\{m\}\(\\pi^\{\\prime\}\)\+4\\Lambda z/n\.The definitionz=log⁡\[2​𝒩∞​\(η,ℱ\)/δ\]z=\\log\[2\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\mathcal\{F\}\)/\\delta\]assigns failure probabilityδ/𝒩∞​\(η,ℱ\)\\delta/\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\mathcal\{F\}\)to each center\. A union bound therefore makes these inequalities hold for every center with probability at least1−δ1\-\\delta\.

To extend them to a candidateπθ\\pi\_\{\\theta\}, choose a centerπ′\\pi^\{\\prime\}whose raw loss differs by at mostη\\etaon every possible group\. Both their population and empirical losses then differ by at mostη\\eta\. Consequently,

Lm​\(πθ\)≤Lm​\(π′\)\+η≤2​L^n​\(π′\)\+4​Λ​z/n\+η≤2​L^n​\(πθ\)\+4​Λ​z/n\+3​η\.L\_\{m\}\(\\pi\_\{\\theta\}\)\\leq L\_\{m\}\(\\pi^\{\\prime\}\)\+\\eta\\leq 2\\widehat\{L\}\_\{n\}\(\\pi^\{\\prime\}\)\+4\\Lambda z/n\+\\eta\\leq 2\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)\+4\\Lambda z/n\+3\\eta\.The reverse direction has the same approximation cost\. Recallingγn=4​Λ​z/n\+3​η\\gamma\_\{n\}=4\\Lambda z/n\+3\\eta, we obtain, simultaneously for allπθ∈Π\\pi\_\{\\theta\}\\in\\Pi,

Lm​\(πθ\)≤2​L^n​\(πθ\)\+γn,L^n​\(πθ\)≤2​Lm​\(πθ\)\+γn\.L\_\{m\}\(\\pi\_\{\\theta\}\)\\leq 2\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)\+\\gamma\_\{n\},\\qquad\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)\\leq 2L\_\{m\}\(\\pi\_\{\\theta\}\)\+\\gamma\_\{n\}\.\(28\)This event depends only on the sampled groups and holds regardless of which policy the solver returns\.

We now compare the returned policy with any candidateπθ\\pi\_\{\\theta\}\. The first inequality bounds the returned policy’s population loss by its empirical loss\. The assumed solver accuracy compares that empirical loss with the candidate’s, and the second inequality returns to population loss:

Lm​\(πθ^\)\\displaystyle L\_\{m\}\(\\pi\_\{\\hat\{\\theta\}\}\)≤2​L^n​\(πθ^\)\+γn\\displaystyle\\leq 2\\widehat\{L\}\_\{n\}\(\\pi\_\{\\hat\{\\theta\}\}\)\+\\gamma\_\{n\}≤2​L^n​\(πθ\)\+2​ξ\+γn\\displaystyle\\leq 2\\widehat\{L\}\_\{n\}\(\\pi\_\{\\theta\}\)\+2\\xi\+\\gamma\_\{n\}≤4​Lm​\(πθ\)\+2​ξ\+3​γn\.\\displaystyle\\leq 4L\_\{m\}\(\\pi\_\{\\theta\}\)\+2\\xi\+3\\gamma\_\{n\}\.Taking the infimum over candidates proves Eq\. \([26](https://arxiv.org/html/2610.20814#A1.E26)\); no attained minimizer is needed\. Substituting this raw\-loss bound into Proposition[6](https://arxiv.org/html/2610.20814#Thmtheorem6)gives the reward consequence in Eq\. \([27](https://arxiv.org/html/2610.20814#A1.E27)\)\. ∎

###### Proof of Theorem[7](https://arxiv.org/html/2610.20814#Thmtheorem7)\.

Use the uniform concentration bound in Eq\. \([28](https://arxiv.org/html/2610.20814#A1.E28)\), proved above before applying any solver\-accuracy assumption\. Substitute its first inequality into Proposition[6](https://arxiv.org/html/2610.20814#Thmtheorem6)\. Uniformity permits any data\-selected output in the fixed class\. ∎

The raw\-loss conclusion of Proposition[8](https://arxiv.org/html/2610.20814#Thmtheorem8)also applies within each refresh round, conditional on the preceding rounds\. Lemma[18](https://arxiv.org/html/2610.20814#Thmtheorem18)uses this conclusion with the local loss boundΛ=β\\Lambda=\\betaand makes it simultaneous across rounds\.

#### Prompt count and optimization setting\.

The oracle terms separate approximation, empirical optimization, and statistics\. These errors play different roles: optimizing the empirical loss accurately cannot compensate for a class that cannot fit the target\. In particular, smallξ\\xialone is insufficient if the minimum lossαm\\alpha\_\{m\}is large\. For anε\\varepsilonstatistical contribution in the observed\-loss bound, chooseη=ε/\(6​C​κK,m\)\\eta=\\varepsilon/\(6C\\kappa\_\{K,m\}\)andn≥8​Λ​C​κK,m​log⁡\[2​𝒩∞​\(η,ℱ\)/δ\]/εn\\geq 8\\Lambda C\\kappa\_\{K,m\}\\log\[2\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\mathcal\{F\}\)/\\delta\]/\\varepsilon\. The oracle remainder has the same rate up to constants\. These are conditional small\-loss rates; entropy is evaluated at the chosen accuracy, and coverage or boundedness may depend on response length\. The analysis covers a fixed offline collection or a stage with fixed behavior and reference policies, not arbitrary behavior and reference changes over a full run\. Additional response groups on a fixed set of sampled prompts do not supply additional independent prompt draws fromρ\\rho\. Independent prompt draws count towardnneven when their values repeat\.

#### Group size\.

Withm=K≥2m=K\\geq 2, the multiplier is at most two; withm=K\+1m=K\+1, it is one\. Thus groups of orderKKdirectly support the best\-of\-KKtarget and can reduce the required number of fresh prompt groups\. The loss, its complexity, and the temperature bias can also change withmm, so monotone improvement is not guaranteed\. Total responses aren​mnm\. Appendix[D\.7](https://arxiv.org/html/2610.20814#A4.SS7)studies group size empirically\.

### A\.4GGPO gradient, normalization, and filtering

#### Off\-policy tilt and on\-policy limit\.

By Eq\. \([2](https://arxiv.org/html/2610.20814#S3.E2)\),eRi/βe^\{R\_\{i\}/\\beta\}is proportional toπ∗​\(ai∣x\)/πθ​\(ai∣x\)\\pi^\{\*\}\(a\_\{i\}\\mid x\)/\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)\. Because responses are sampled fromπbeh\\pi\_\{\\mathrm\{beh\}\}, their normalized weighted empirical distribution converges to

qθ​\(a∣x\)=πbeh​\(a∣x\)​π∗​\(a∣x\)/πθ​\(a∣x\)𝔼a′∼πbeh​\[π∗​\(a′∣x\)/πθ​\(a′∣x\)\],q\_\{\\theta\}\(a\\mid x\)=\\frac\{\\pi\_\{\\mathrm\{beh\}\}\(a\\mid x\)\\pi^\{\*\}\(a\\mid x\)/\\pi\_\{\\theta\}\(a\\mid x\)\}\{\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\pi\_\{\\mathrm\{beh\}\}\}\[\\pi^\{\*\}\(a^\{\\prime\}\\mid x\)/\\pi\_\{\\theta\}\(a^\{\\prime\}\\mid x\)\]\},the induced tilt in Eq\. \([18](https://arxiv.org/html/2610.20814#A1.E18)\)\. Holdingπbeh\\pi\_\{\\mathrm\{beh\}\}fixed gives

∇θ𝒥beh=−β⁡\(𝔼qθ​∇θ​log⁡πθ−𝔼πbeh​∇θ​log⁡πθ\)\.\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\-\\beta\\left\(\\mathbb\{E\}\_\{q\_\{\\theta\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\-\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\right\)\.Whenπbeh=πθ\\pi\_\{\\mathrm\{beh\}\}=\\pi\_\{\\theta\},qθ=π∗q\_\{\\theta\}=\\pi^\{\*\}and the second expectation vanishes\. The objective value then equals the reverse\-KL regularized gap, while the frozen\-πbeh\\pi\_\{\\mathrm\{beh\}\}derivative is the forward\-KL/cross\-entropy gradient towardπ∗\\pi^\{\*\}\. Weight concentration is summarized byESSq=1/∑iqi2∈\[1,m\]\\mathrm\{ESS\}\_\{q\}=1/\\sum\_\{i\}q\_\{i\}^\{2\}\\in\[1,m\]\.

Let𝒥^mg:=𝒥^m​\(πθ,xg,Ag\)\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}:=\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};x\_\{g\},A\_\{g\}\), withAg=ag1:mA\_\{g\}=a^\{g\}\_\{1:m\}, be the loss of groupgg, and letℓ¯\\bar\{\\ell\}be the mean response length in the learner minibatch\. The implemented loss is given in Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\)\. Ifsis\_\{i\}is the sequence log likelihood of responseii, then∂𝒥^mg/∂si=−β⁡\(qi−1/m\)\\partial\\widehat\{\\mathcal\{J\}\}\_\{m\}^\{\\,g\}/\\partial s\_\{i\}=\-\\beta\(q\_\{i\}\-1/m\)\. After applying the normalization above, its derivative is−\(qi−1/m\)/\(Gmbℓ¯m−1\)\-\(q\_\{i\}\-1/m\)/\(G\_\{\\mathrm\{mb\}\}\\bar\{\\ell\}\\sqrt\{m\-1\}\)\. Writing the usual token\-mean policy\-gradient form as−ci/\(Gmbmℓ¯\)\-c\_\{i\}/\(G\_\{\\mathrm\{mb\}\}m\\bar\{\\ell\}\)therefore gives the group coefficient

ci=m⁡\(qi−1/m\)m−1\.c\_\{i\}=\\frac\{m\(q\_\{i\}\-1/m\)\}\{\\sqrt\{m\-1\}\}\.\(29\)The factors1/β1/\\betaand1/ℓ¯1/\\bar\{\\ell\}remove the rawβ\\betaprefactor and convert the group average to a token mean; the factor1/m−11/\\sqrt\{m\-1\}keeps the coefficient scale similar to GRPO’s standardized advantages\. Thusβ\\betaaffects the implemented update throughqq, not through a remaining global multiplier\.

#### Binary\-reward limit and weight concentration\.

Let𝒮\\mathcal\{S\}be thekkresponses of a mixed\-reward group with verifier reward one, and writeΔi=log⁡\[πθ​\(ai∣x\)/πref​\(ai∣x\)\]\\Delta\_\{i\}=\\log\[\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\\mid x\)\]\. For fixed finiteΔi\\Delta\_\{i\}, asβ→0\\beta\\to 0the inter\-class logit gap in Eq\. \([10](https://arxiv.org/html/2610.20814#S4.E10)\) dominates, soqqconcentrates on𝒮\\mathcal\{S\}withqi→softmaxi∈𝒮⁡\(−Δi\)q\_\{i\}\\to\\softmax\_\{i\\in\\mathcal\{S\}\}\(\-\\Delta\_\{i\}\)andqi→0q\_\{i\}\\to 0otherwise\. By Eq\. \([29](https://arxiv.org/html/2610.20814#A1.E29)\), every response outside𝒮\\mathcal\{S\}then hasci→−1/m−1c\_\{i\}\\to\-1/\\sqrt\{m\-1\}, and the signed sum over correct responses is

∑i∈𝒮ci⟶m−km−1\.\\sum\_\{i\\in\\mathcal\{S\}\}c\_\{i\}\\longrightarrow\\frac\{m\-k\}\{\\sqrt\{m\-1\}\}\.For GRPO’s reward advantages standardized using the sample standard deviation with variance denominatorm−1m\-1, and omitting the10−610^\{\-6\}stabilizer, the corresponding advantage sum isk​\(m−k\)​\(m−1\)/m\\sqrt\{k\(m\-k\)\(m\-1\)/m\}\. For AGRO, equal learner\-to\-reference log ratios within the group cancel in leave\-one\-out centering, giving an advantage sum ofk⁡\(m−k\)/\(m−1\)k\(m\-k\)/\(m\-1\)\. Normalizing each curve to its value atk=7k=7gives Figure[6](https://arxiv.org/html/2610.20814#A4.F6)\.

Because allocation inside𝒮\\mathcal\{S\}followsqqrather than being uniform, a correct response can receiveci<0c\_\{i\}<0\. In this limit that happens exactly when\(m−1\)​e−Δi<∑j∈𝒮∖\{i\}e−Δj\(m\-1\)e^\{\-\\Delta\_\{i\}\}<\\sum\_\{j\\in\\mathcal\{S\}\\setminus\\\{i\\\}\}e^\{\-\\Delta\_\{j\}\}; equivalently, writingΔ¯−i=−log⁡\[\(k−1\)−1​∑j∈𝒮∖\{i\}e−Δj\]\\bar\{\\Delta\}\_\{\-i\}=\-\\log\[\(k\-1\)^\{\-1\}\\sum\_\{j\\in\\mathcal\{S\}\\setminus\\\{i\\\}\}e^\{\-\\Delta\_\{j\}\}\], whenΔi−Δ¯−i\>log⁡\[\(m−1\)/\(k−1\)\]\\Delta\_\{i\}\-\\bar\{\\Delta\}\_\{\-i\}\>\\log\[\(m\-1\)/\(k\-1\)\]\. A lone correct response \(k=1k=1\) can never be penalized this way\.

If the learner\-to\-reference log ratiosΔi\\Delta\_\{i\}are equal across the successful responses𝒮\\mathcal\{S\}, the binary\-reward limit above yields

ci⟶1m−1​\(m​rik−1\)\.c\_\{i\}\\longrightarrow\\frac\{1\}\{\\sqrt\{m\-1\}\}\\left\(\\frac\{mr\_\{i\}\}\{k\}\-1\\right\)\.The expression in parentheses is the centered success\-normalized advantage of MaxRL\([Tajwar et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib42)\)\. MaxRL derives this update from an on\-policy maximum\-likelihood formulation, whereas GGPO derives its log\-mean\-exp loss from KL\-regularized duality and permits behavior–learner mismatch without learner\-to\-behavior importance weights\. For equally rewarded responses,

qiqj=exp⁡\[−\(Δi−Δj\)\]\\frac\{q\_\{i\}\}\{q\_\{j\}\}=\\exp\\\!\\left\[\-\(\\Delta\_\{i\}\-\\Delta\_\{j\}\)\\right\]at everyβ\>0\\beta\>0\. Thus, reference\-dependent weighting persists even at small temperatures when these log ratios differ\.

#### Reference anchoring and refresh\.

The log\-ratio differenceΔi−Δj\\Delta\_\{i\}\-\\Delta\_\{j\}controls the relative weights of equally rewarded responses without attenuation byβ\\beta\. At an exact policy\-copy boundary,πref=πθ\\pi\_\{\\mathrm\{ref\}\}=\\pi\_\{\\theta\}, allΔi\\Delta\_\{i\}vanish, so these responses receive equal weights\. Subsequent learner updates change the log ratios and favor responses with smallerΔi\\Delta\_\{i\}\. This explains how reference refresh resets the within\-class weighting\.

The reward\-only ablation setsqi=softmaxi⁡\(ri/β\)q\_\{i\}=\\softmax\_\{i\}\(r\_\{i\}/\\beta\)in Eq\. \([29](https://arxiv.org/html/2610.20814#A1.E29)\), retaining the score\-gradient update and normalization\. For mixed binary\-reward groups, its coefficients approach the MaxRL form above asβ→0\\beta\\to 0, without requiring equalΔi\\Delta\_\{i\}\. This removes the reference\-dependent weighting that persists in full GGPO at small temperatures\.

#### Mixed\-group filtering and normalization\.

The batch\-dependent normalization in Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\) changes the expected training loss, but its positive multiplier preserves zeros\. For a fixed promptxxwith0<Pra∼πbeh\(⋅∣x\)\[r\(x,a\)=1\]<10<\\Pr\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\(\\cdot\\mid x\)\}\[r\(x,a\)=1\]<1, defineEmix=\{mini⁡r⁡\(x,ai\)=0,maxi⁡r⁡\(x,ai\)=1\}E\_\{\\mathrm\{mix\}\}=\\\{\\min\_\{i\}r\(x,a\_\{i\}\)=0,\\ \\max\_\{i\}r\(x,a\_\{i\}\)=1\\\}and

Hmmix\(πθ;πbeh,x\):=𝔼a1:m∼πbehm\[𝒥^m\(πθ;x,a1:m\)∣Emix\]\.H\_\{m\}^\{\\mathrm\{mix\}\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x\):=\\mathbb\{E\}\_\{a\_\{1:m\}\\sim\\pi\_\{\\mathrm\{beh\}\}^\{m\}\}\[\\widehat\{\\mathcal\{J\}\}\_\{m\}\(\\pi\_\{\\theta\};x,a\_\{1:m\}\)\\mid E\_\{\\mathrm\{mix\}\}\]\.\(30\)
###### Proposition 9\(Zero set under mixed\-group filtering\)\.

Supposem≥2m\\geq 2,𝒜\\mathcal\{A\}is finite, the verifier reward is deterministic and binary, and0<Pra∼πbeh\[r\(x,a\)=1\]<10<\\Pr\_\{a\\sim\\pi\_\{\\mathrm\{beh\}\}\}\[r\(x,a\)=1\]<1\. ThenHmmix​\(πθ,πbeh,x\)=0H\_\{m\}^\{\\mathrm\{mix\}\}\(\\pi\_\{\\theta\};\\pi\_\{\\mathrm\{beh\}\},x\)=0if and only ifRθ​\(x,a\)R\_\{\\theta\}\(x,a\)is constant on the support ofπbeh\\pi\_\{\\mathrm\{beh\}\}\. Hence, for each such prompt, the filtered loss has the same zero set on behavior support as the unfiltered targetHmH\_\{m\}and the population objective𝒥beh\\mathcal\{J\}\_\{\\mathrm\{beh\}\}\. Positive batch\-dependent normalization preserves this zero set over the retained prompt distribution\.

###### Proof\.

The empirical loss is nonnegative and is zero exactly when all adjusted rewards in the group agree\. Any behavior\-supported reward\-zero action and reward\-one action can co\-occur in a mixed group with positive probability, so a zero conditional expectation forces their adjusted rewards to agree\. Two supported actions with the same reward can each be connected through any supported action of the opposite reward\. Hence all supported actions have the same adjusted reward\. The converse is immediate\. Finally, a positive batch\-dependent multiplier cannot create or remove zeros of a nonnegative minibatch loss\. ∎

### A\.5Reference refresh: exact and imperfect updates

We first show how exact Gibbs targets compose across reference updates\. We then prove Proposition[11](https://arxiv.org/html/2610.20814#Thmtheorem11)by accounting for the residual error in each round and summing the resulting reward comparisons\.

#### Fixed\-reference bias and exact composition\.

For binary reward with reference success probabilitys∈\(0,1\)s\\in\(0,1\), the zero\-loss Gibbs policy succeeds with probabilitys​e1/β/\(1−s\+s​e1/β\)<1se^\{1/\\beta\}/\(1\-s\+se^\{1/\\beta\}\)<1\. Best\-of\-KKfrom the same reference succeeds with probability1−\(1−s\)K1\-\(1\-s\)^\{K\}, which can be strictly larger\. This gap remains at zero loss for every group size, including the infinite\-group limit\. Reference refresh changes the target itself\.

###### Lemma 10\(Exact reference composition\)\.

Suppose the learner, reference, and behavior policies are positive on𝒜\\mathcal\{A\}\. In every roundtt, supposeπt\\pi\_\{t\}has zero population finite\-group loss formt≥2m\_\{t\}\\geq 2independent responses fromπbeh,t\\pi\_\{\\mathrm\{beh\},t\}, relative to referenceπt−1\\pi\_\{t\-1\}\. Then temperaturesβt\>0\\beta\_\{t\}\>0compose asπT​\(a∣x\)∝π0​\(a∣x\)​exp⁡\{r⁡\(x,a\)​∑t=1Tβt−1\}\\pi\_\{T\}\(a\\mid x\)\\propto\\pi\_\{0\}\(a\\mid x\)\\exp\\\{r\(x,a\)\\sum\_\{t=1\}^\{T\}\\beta\_\{t\}^\{\-1\}\\\}\. In particular, for constantβ\\betaand the best\-of\-KKselection policyuKu\_\{K\}fromπ0\\pi\_\{0\},

JK​\(π0\)−J⁡\(πT\)≤β​log⁡KT\.J\_\{K\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{T\}\)\\leq\\frac\{\\beta\\log K\}\{T\}\.

###### Proof\.

Under positivity on𝒜\\mathcal\{A\}, Proposition[5](https://arxiv.org/html/2610.20814#Thmtheorem5)identifies zero population finite\-group loss with the Gibbs policy\. With referenceπt−1\\pi\_\{t\-1\}in roundtt, Eq\. \([2](https://arxiv.org/html/2610.20814#S3.E2)\) therefore givesπt​\(a∣x\)∝πt−1​\(a∣x\)​er⁡\(x,a\)/βt\\pi\_\{t\}\(a\\mid x\)\\propto\\pi\_\{t\-1\}\(a\\mid x\)e^\{r\(x,a\)/\\beta\_\{t\}\}\. For constant temperature, iterating yields Eq\. \([15](https://arxiv.org/html/2610.20814#S4.E15)\)\. More generally, temperaturesβt\>0\\beta\_\{t\}\>0give effective temperatureβeff=\(∑t=1Tβt−1\)−1\\beta\_\{\\rm eff\}=\(\\sum\_\{t=1\}^\{T\}\\beta\_\{t\}^\{\-1\}\)^\{\-1\}\. Write𝒦\(u,v\)=𝔼XDKL\(u\(⋅∣X\)∥v\(⋅∣X\)\)\\mathcal\{K\}\(u,v\)=\\mathbb\{E\}\_\{X\}D\_\{\\mathrm\{KL\}\}\(u\(\\cdot\\mid X\)\\\|v\(\\cdot\\mid X\)\)\. For constantβ\\beta, the final policy maximizesJ⁡\(π\)−\(β/T\)​𝒦​\(π,π0\)J\(\\pi\)\-\(\\beta/T\)\\mathcal\{K\}\(\\pi,\\pi\_\{0\}\), so for every comparatoruu,

J⁡\(u\)−J⁡\(πT\)≤\(β/T\)​𝒦​\(u,π0\)\.J\(u\)\-J\(\\pi\_\{T\}\)\\leq\(\\beta/T\)\\mathcal\{K\}\(u,\\pi\_\{0\}\)\.LetuKu\_\{K\}be the distribution of the selected response amongKKindependent draws fromπ0\\pi\_\{0\}, using a fixed tie\-breaking rule\. Selection of actionaarequires at least one draw ofaa, henceuK​\(a∣x\)≤K​π0​\(a∣x\)u\_\{K\}\(a\\mid x\)\\leq K\\pi\_\{0\}\(a\\mid x\)\. ConsequentlyJ⁡\(uK\)=JK​\(π0\)J\(u\_\{K\}\)=J\_\{K\}\(\\pi\_\{0\}\)and𝒦⁡\(uK,π0\)≤log⁡K\\mathcal\{K\}\(u\_\{K\},\\pi\_\{0\}\)\\leq\\log K, which proves the reward bound\. ∎

#### Imperfect reference updates\.

###### Proposition 11\(Reference refresh with imperfect rounds\)\.

Letr∈\[0,1\]r\\in\[0,1\],m≥2m\\geq 2, anduKu\_\{K\}be the best\-of\-KKselection policy from a positive initial policyπ0\\pi\_\{0\}\. For each positive round outputπt\\pi\_\{t\}and positive behaviorπbeh,t\\pi\_\{\\mathrm\{beh\},t\}, assume uniformly over prompts and actions that

πt​\(a∣x\)πbeh,t​\(a∣x\)≤C,uK​\(a∣x\)πbeh,t​\(a∣x\)≤C,osca⁡\{r⁡\(x,a\)β−log⁡πt​\(a∣x\)πt−1​\(a∣x\)\}≤D\.\\frac\{\\pi\_\{t\}\(a\\mid x\)\}\{\\pi\_\{\\mathrm\{beh\},t\}\(a\\mid x\)\}\\leq C,\\qquad\\frac\{u\_\{K\}\(a\\mid x\)\}\{\\pi\_\{\\mathrm\{beh\},t\}\(a\\mid x\)\}\\leq C,\\qquad\\operatorname\{osc\}\_\{a\}\\\!\\left\\\{\\frac\{r\(x,a\)\}\{\\beta\}\-\\log\\frac\{\\pi\_\{t\}\(a\\mid x\)\}\{\\pi\_\{t\-1\}\(a\\mid x\)\}\\right\\\}\\leq D\.\(31\)Hereosca\\operatorname\{osc\}\_\{a\}denotes maximum minus minimum over responsesaaat a fixed prompt\. Define the average output policy and average population loss by

π¯T=1T​∑t=1Tπt,L¯T=1T​∑t=1T𝔼X​Hm​\(πt,πbeh,t,X,πt−1\)\.\\bar\{\\pi\}\_\{T\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\pi\_\{t\},\\qquad\\bar\{L\}\_\{T\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\_\{X\}H\_\{m\}\(\\pi\_\{t\};\\pi\_\{\\mathrm\{beh\},t\},X;\\pi\_\{t\-1\}\)\.Then

JK​\(π0\)−J⁡\(π¯T\)≤β​log⁡KT\+4​β​C​eD1−1/m​L¯T\.J\_\{K\}\(\\pi\_\{0\}\)\-J\(\\bar\{\\pi\}\_\{T\}\)\\leq\\frac\{\\beta\\log K\}\{T\}\+\\sqrt\{\\frac\{4\\beta Ce^\{D\}\}\{1\-1/m\}\\,\\bar\{L\}\_\{T\}\}\.\(32\)

For roundtt, letRt=r−β​log⁡\(πt/πt−1\)R\_\{t\}=r\-\\beta\\log\(\\pi\_\{t\}/\\pi\_\{t\-1\}\)andLt​\(πt\)=𝔼X​Hm​\(πt,πbeh,t,X,πt−1\)L\_\{t\}\(\\pi\_\{t\}\)=\\mathbb\{E\}\_\{X\}H\_\{m\}\(\\pi\_\{t\};\\pi\_\{\\mathrm\{beh\},t\},X;\\pi\_\{t\-1\}\)\. Use the common coverage, residual\-range, and group\-size boundsCC,DD, andmmfrom the proposition\.

###### Proof of Proposition[11](https://arxiv.org/html/2610.20814#Thmtheorem11)\.

First, an adjusted rewardRRwithosc⁡\(R/β\)≤D\\operatorname\{osc\}\(R/\\beta\)\\leq Dobeys the finite\-group variance bound

Hm≥\(1−1/m\)​e−D2​β​Varπbeh⁡\(R\)\.H\_\{m\}\\geq\\frac\{\(1\-1/m\)e^\{\-D\}\}\{2\\beta\}\\operatorname\{Var\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\(R\)\.\(33\)For a realized group, setzi=Ri/βz\_\{i\}=R\_\{i\}/\\betaandg⁡\(s\)=log⁡\(m−1​∑ies​zi\)g\(s\)=\\log\(m^\{\-1\}\\sum\_\{i\}e^\{sz\_\{i\}\}\)\. Fors∈\[0,1\]s\\in\[0,1\], the normalized weightes​zi/∑jes​zje^\{sz\_\{i\}\}/\\sum\_\{j\}e^\{sz\_\{j\}\}of every group member is at leaste−D/me^\{\-D\}/m\. The minimum\-mean\-square characterization of variance therefore givesg′′​\(s\)=Vars⁡\(z\)≥e−D​Vargroup⁡\(z\)g^\{\\prime\\prime\}\(s\)=\\operatorname\{Var\}\_\{s\}\(z\)\\geq e^\{\-D\}\\operatorname\{Var\}\_\{\\rm group\}\(z\)\. Integrating with weight1−s1\-sbounds the group loss below byβ​e−D​Vargroup⁡\(z\)/2\\beta e^\{\-D\}\\operatorname\{Var\}\_\{\\rm group\}\(z\)/2\. Its expectation is Eq\. \([33](https://arxiv.org/html/2610.20814#A1.E33)\), since empirical variance with denominatormmhas expectation\(1−1/m\)​Varπbeh⁡\(R/β\)\(1\-1/m\)\\operatorname\{Var\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\(R/\\beta\)\.

LetR¯t=𝔼πbeh,t​Rt\\bar\{R\}\_\{t\}=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\},t\}\}R\_\{t\}\. Since both density ratios haveπbeh,t\\pi\_\{\\mathrm\{beh\},t\}\-mean one,

𝔼uK​Rt−𝔼πt​Rt=𝔼πbeh,t​\[\(uKπbeh,t−πtπbeh,t\)​\(Rt−R¯t\)\]\.\\mathbb\{E\}\_\{u\_\{K\}\}R\_\{t\}\-\\mathbb\{E\}\_\{\\pi\_\{t\}\}R\_\{t\}=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\},t\}\}\\\!\\left\[\\left\(\\frac\{u\_\{K\}\}\{\\pi\_\{\\mathrm\{beh\},t\}\}\-\\frac\{\\pi\_\{t\}\}\{\\pi\_\{\\mathrm\{beh\},t\}\}\\right\)\(R\_\{t\}\-\\bar\{R\}\_\{t\}\)\\right\]\.Cauchy–Schwarz and the coverage assumptions in Proposition[11](https://arxiv.org/html/2610.20814#Thmtheorem11)then give pointwise

𝔼uK​Rt−𝔼πt​Rt≤2​C​Varπbeh,t⁡\(Rt\)\.\\mathbb\{E\}\_\{u\_\{K\}\}R\_\{t\}\-\\mathbb\{E\}\_\{\\pi\_\{t\}\}R\_\{t\}\\leq\\sqrt\{2C\\operatorname\{Var\}\_\{\\pi\_\{\\mathrm\{beh\},t\}\}\(R\_\{t\}\)\}\.Indeed,𝔼πbeh,t​\[\(uK/πbeh,t−πt/πbeh,t\)2\]≤2​C\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\},t\}\}\[\(u\_\{K\}/\\pi\_\{\\mathrm\{beh\},t\}\-\\pi\_\{t\}/\\pi\_\{\\mathrm\{beh\},t\}\)^\{2\}\]\\leq 2C, because both densities integrate to one and their cross product is nonnegative\. Using Eq\. \([33](https://arxiv.org/html/2610.20814#A1.E33)\) and Jensen over prompts bounds this residual difference by\{4​β​C​eD​Lt​\(πt\)/\(1−1/m\)\}1/2\\\{4\\beta Ce^\{D\}L\_\{t\}\(\\pi\_\{t\}\)/\(1\-1/m\)\\\}^\{1/2\}\. The exact identity

J⁡\(uK\)−J⁡\(πt\)=\\displaystyle J\(u\_\{K\}\)\-J\(\\pi\_\{t\}\)=\{\}β⁡\[𝒦⁡\(uK,πt−1\)−𝒦⁡\(uK,πt\)−𝒦⁡\(πt,πt−1\)\]\\displaystyle\\beta\[\\mathcal\{K\}\(u\_\{K\},\\pi\_\{t\-1\}\)\-\\mathcal\{K\}\(u\_\{K\},\\pi\_\{t\}\)\-\\mathcal\{K\}\(\\pi\_\{t\},\\pi\_\{t\-1\}\)\]\+𝔼X​\[𝔼uK​Rt−𝔼πt​Rt\]\\displaystyle\+\\mathbb\{E\}\_\{X\}\[\\mathbb\{E\}\_\{u\_\{K\}\}R\_\{t\}\-\\mathbb\{E\}\_\{\\pi\_\{t\}\}R\_\{t\}\]then telescopes over rounds\. Dropping nonnegative terminal and step KL terms and using𝒦⁡\(uK,π0\)≤log⁡K\\mathcal\{K\}\(u\_\{K\},\\pi\_\{0\}\)\\leq\\log Kyields the population inequality

JK​\(π0\)−J⁡\(π¯T\)≤β​log⁡KT\+1T​∑t=1T4​β​C​eD1−1/m​Lt​\(πt\)\.J\_\{K\}\(\\pi\_\{0\}\)\-J\(\\bar\{\\pi\}\_\{T\}\)\\leq\\frac\{\\beta\\log K\}\{T\}\+\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\sqrt\{\\frac\{4\\beta Ce^\{D\}\}\{1\-1/m\}L\_\{t\}\(\\pi\_\{t\}\)\}\.Finally,T−1​∑t=1TLt​\(πt\)≤L¯TT^\{\-1\}\\sum\_\{t=1\}^\{T\}\\sqrt\{L\_\{t\}\(\\pi\_\{t\}\)\}\\leq\\sqrt\{\\bar\{L\}\_\{T\}\}gives Eq\. \([32](https://arxiv.org/html/2610.20814#A1.E32)\)\.

∎

### A\.6Relation to AGRO: approximation and weighting

This calculation supports the comparison in Section[2](https://arxiv.org/html/2610.20814#S2): AGRO is the local quadratic approximation to the population contrastive loss when adjusted rewards have small spread\. The finite\-group comparison below also explains the role of exponential weighting and the slack in a uniform variance upper bound\. Neither comparison globally orders the unscaled losses\.

For the local relation to AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\), letR¯θ=𝔼πbeh​Rθ\\bar\{R\}\_\{\\theta\}=\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}R\_\{\\theta\}andR~=Rθ−R¯θ\\tilde\{R\}=R\_\{\\theta\}\-\\bar\{R\}\_\{\\theta\}\. The population contrastive loss can be written as the centered log moment

𝒥beh=β​log⁡𝔼πbeh​eR~/β\.\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\\beta\\log\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}e^\{\\tilde\{R\}/\\beta\}\.\(34\)Because𝔼πbeh​R~=0\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\\tilde\{R\}=0, Taylor expansion as∥R~∥∞/β→0\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}/\\beta\\to 0gives

𝒥beh=12​β​𝔼πbeh​R~2\+O⁡\(∥R~∥∞3β2\)=12​β​Varπbeh​\[Rθ\]\+O⁡\(∥R~∥∞3β2\)\.\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\\frac\{1\}\{2\\beta\}\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\\tilde\{R\}^\{2\}\+O\\\!\\left\(\\frac\{\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}^\{3\}\}\{\\beta^\{2\}\}\\right\)=\\frac\{1\}\{2\\beta\}\\operatorname\{Var\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\[R\_\{\\theta\}\]\+O\\\!\\left\(\\frac\{\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}^\{3\}\}\{\\beta^\{2\}\}\\right\)\.\(35\)AGRO’s off\-policy population loss isℒAGRO=12​Varπbeh​\[Rθ\]\\mathcal\{L\}\_\{\\mathrm\{AGRO\}\}=\\tfrac\{1\}\{2\}\\operatorname\{Var\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}\[R\_\{\\theta\}\], so locally𝒥beh=ℒAGRO/β\\mathcal\{J\}\_\{\\mathrm\{beh\}\}=\\mathcal\{L\}\_\{\\mathrm\{AGRO\}\}/\\betaup to higher\-order terms\. Equation \(4\) of[Tang et al\. \(2026\)](https://arxiv.org/html/2610.20814#bib.bib43)gives the leave\-one\-out finite\-group stochastic gradient of this variance objective\. Our implementation divides out its exteriorβ\\betafactor and averages over response tokens \(Eq\. \([62](https://arxiv.org/html/2610.20814#A3.E62)\)\)\. The corresponding coefficient relation follows by defining the exponentially tilted distribution

d​qθd​πbeh=eRθ/β𝔼πbeh​eRθ/β=1\+R~β\+O⁡\(∥R~∥∞2β2\)\.\\frac\{dq\_\{\\theta\}\}\{d\\pi\_\{\\mathrm\{beh\}\}\}=\\frac\{e^\{R\_\{\\theta\}/\\beta\}\}\{\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\}\}\}e^\{R\_\{\\theta\}/\\beta\}\}=1\+\\frac\{\\tilde\{R\}\}\{\\beta\}\+O\\\!\\left\(\\frac\{\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}^\{2\}\}\{\\beta^\{2\}\}\\right\)\.\(36\)Thus the raw GGPO coefficient multiplying the policy score is

−β⁡\(d​qθd​πbeh−1\)=−R~\+O⁡\(∥R~∥∞2β\)\.\-\\beta\\left\(\\frac\{dq\_\{\\theta\}\}\{d\\pi\_\{\\mathrm\{beh\}\}\}\-1\\right\)=\-\\tilde\{R\}\+O\\\!\\left\(\\frac\{\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}^\{2\}\}\{\\beta\}\\right\)\.\(37\)The leading term−R~\-\\tilde\{R\}is exactly the score\-function coefficient obtained by differentiating the normalized AGRO lossℒAGRO/β\\mathcal\{L\}\_\{\\mathrm\{AGRO\}\}/\\beta, holding the behavior distribution fixed\. Hence the scaled AGRO loss is the local quadratic approximation to the raw population GGPO loss, and its centered linear coefficient is the first\-order approximation to the raw GGPO coefficient when∥R~∥∞/β→0\\lVert\\tilde\{R\}\\rVert\_\{\\infty\}/\\beta\\to 0\. Our default training setting uses a smaller temperature relative to verifier\-reward differences, so we treat AGRO as a distinct objective rather than an approximation along the observed training trajectory\.

## Appendix BFormal final\-policy guarantee and proof

### B\.1Formal setup and final\-policy guarantee

The population and finite\-group bounds in Eqs\. \([5](https://arxiv.org/html/2610.20814#S3.E5)\) and \([14](https://arxiv.org/html/2610.20814#S4.E14)\) use sequence coverage, which can grow rapidly with response length\. We relate behavior and learner distributions through moments of their response likelihood ratio\. Conditional token log moments provide a sufficient bound on these coverage moments\. Together with local reference control, this yields a finite\-group loss calibration and, after repeated reference updates, a finite\-data guarantee for the final policy\.

Proposition[11](https://arxiv.org/html/2610.20814#Thmtheorem11)controls an average output using population losses and behavior coverage of the learner and best\-of\-KKcomparator\. Here we assume binary rewards and initial correct\-response coverage\. Each exact Gibbs update multiplies correct\-response odds bye1/βe^\{1/\\beta\}; the theorem quantifies how errors in fitting these updates affect the final reward\.

#### Reference\-update rounds\.

Responses have at mostHHtokens \(pad with a deterministic terminal token when necessary\)\. Writeyyfor a complete response andaafor a token\. For a policyπ\\pi,πh​\(a∣s\)\\pi\_\{h\}\(a\\mid s\)is its token conditional at positionhh, where a legal historys=\(x,y<h\)s=\(x,y\_\{<h\}\)consists of a prompt and an allowed response prefix\. Each round holds the reference fixed while fitting a new policy\. In roundtt, set the reference toπt−1\\pi\_\{t\-1\}and freeze behaviorπbeh,t\\pi\_\{\\mathrm\{beh\},t\}and a candidate classΠt\\Pi\_\{t\}before drawing data\. Drawnnindependent prompt groups fromρ\\rho, each withm≥2m\\geq 2independent responses underπbeh,t\\pi\_\{\\mathrm\{beh\},t\}\. Minimize the raw loss in Eq\. \([6](https://arxiv.org/html/2610.20814#S4.E6)\), with referenceπt−1\\pi\_\{t\-1\}, overΠt\\Pi\_\{t\}to the empirical accuracy specified below\. Denote the output byπt\\pi\_\{t\}and use it as the next reference\. Groups are unfiltered and fresh across rounds; data may be reused within a round\.

###### Assumption 1\(Local reference control\)\.

Rewards satisfyr⁡\(x,y\)∈\{0,1\}r\(x,y\)\\in\\\{0,1\\\}, and all policies are positive on the same finite response support\. The temperature satisfiesβ≥2​H\+1\\beta\\geq 2H\+1, and every candidate inΠt\\Pi\_\{t\}satisfies

\|log⁡πh​\(a∣s\)πt−1,h​\(a∣s\)\|≤1β\(h=1,…,H\)\\left\|\\log\\frac\{\\pi\_\{h\}\(a\\mid s\)\}\{\\pi\_\{t\-1,h\}\(a\\mid s\)\}\\right\|\\leq\\frac\{1\}\{\\beta\}\\qquad\(h=1,\\ldots,H\)\(40\)at every legal historyssand tokenaa, in every roundtt\.

The local bound limits variation in adjusted rewards\. Together withβ≥2​H\+1\\beta\\geq 2H\+1, it keeps the group loss in\[0,β\]\[0,\\beta\], allowing us to estimate its expectation from finitely many groups\.

The exact KL\-regularized optimum for referenceμ=πt−1\\mu=\\pi\_\{t\-1\}ispμ∗​\(y∣x\)∝μ⁡\(y∣x\)​er⁡\(x,y\)/βp\_\{\\mu\}^\{\*\}\(y\\mid x\)\\propto\\mu\(y\\mid x\)e^\{r\(x,y\)/\\beta\}\. Its conditional normalizerZ⁡\(s\)=𝔼μ​\[er/β∣s\]Z\(s\)=\\mathbb\{E\}\_\{\\mu\}\[e^\{r/\\beta\}\\mid s\]lies in\[1,e1/β\]\[1,e^\{1/\\beta\}\]because rewards are binary\. Hence

pμ,h∗​\(a∣s\)μh​\(a∣s\)=Z⁡\(s​a\)Z⁡\(s\)∈\[e−1/β,e1/β\],\|logpμ,h∗​\(a∣s\)μh​\(a∣s\)\|≤1β\.\\frac\{p\_\{\\mu,h\}^\{\*\}\(a\\mid s\)\}\{\\mu\_\{h\}\(a\\mid s\)\}=\\frac\{Z\(sa\)\}\{Z\(s\)\}\\in\[e^\{\-1/\\beta\},e^\{1/\\beta\}\],\\qquad\\left\|\\log\\frac\{p\_\{\\mu,h\}^\{\*\}\(a\\mid s\)\}\{\\mu\_\{h\}\(a\\mid s\)\}\\right\|\\leq\\frac\{1\}\{\\beta\}\.Thus each round’s optimum satisfies the constraint; the final comparatorπ⋆\\pi\_\{\\star\}need not, since repeated updates can move far from the initial policy\. The errors below allow approximate fitting within the constrained class\. We can also relax the token bound to\(1\+γ\)/β\(1\+\\gamma\)/\\betaforγ≥0\\gamma\\geq 0ifβ≥1\+2​H​\(1\+γ\)\\beta\\geq 1\+2H\(1\+\\gamma\); Appendix[B\.6](https://arxiv.org/html/2610.20814#A2.SS6)gives the details, including the revised behavior\-lag bound\.

The local constraint also motivates the practical token clipping in Eq\. \([12](https://arxiv.org/html/2610.20814#S4.E12)\)\. That variant limits token log\-ratio contributions to the response coefficients, with some bias, but leaves the policy class unconstrained\. The theorem therefore analyzes the raw loss under Eq\. \([40](https://arxiv.org/html/2610.20814#A2.E40)\), whereas Section[6\.6](https://arxiv.org/html/2610.20814#S6.SS6)evaluates clipping and Appendix[E](https://arxiv.org/html/2610.20814#A5)specifies its implementation\.

###### Definition 1\(Round losses\)\.

For referenceπt−1\\pi\_\{t\-1\}and a groupg=\(x,y1:m\)g=\(x,y\_\{1:m\}\), letℓt​\(π,g\)\\ell\_\{t\}\(\\pi;g\)be the raw group loss in Eq\. \([6](https://arxiv.org/html/2610.20814#S4.E6)\)\. Define its population and empirical averages by

Lt​\(π\):=𝔼X∼ρ​Hm​\(π,πbeh,t,X,πt−1\),L^t​\(π\):=1n​∑j=1nℓt​\(π,gj\),L\_\{t\}\(\\pi\):=\\mathbb\{E\}\_\{X\\sim\\rho\}H\_\{m\}\(\\pi;\\pi\_\{\\mathrm\{beh\},t\},X;\\pi\_\{t\-1\}\),\\qquad\\widehat\{L\}\_\{t\}\(\\pi\):=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\ell\_\{t\}\(\\pi;g\_\{j\}\),whereg1,…,gng\_\{1\},\\ldots,g\_\{n\}are the sampled groups in roundtt\.

Estimating the loss of a single fixed policy is not enough, because the returned policy is chosen using the same data\. The finite\-data guarantees therefore depend on the complexity of the group\-loss functions\. We measure this complexity using the following uniform covering number\.

###### Definition 2\(Uniform loss covering number\)\.

For a familyℱ\\mathcal\{F\}of real\-valued functions on prompt\-response groups, define

‖f−f′‖∞:=supg\|f⁡\(g\)−f′​\(g\)\|\.\\\|f\-f^\{\\prime\}\\\|\_\{\\infty\}:=\\sup\_\{g\}\|f\(g\)\-f^\{\\prime\}\(g\)\|\.\(41\)The uniformL∞L\_\{\\infty\}covering number𝒩∞​\(η,ℱ\)\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\mathcal\{F\}\)is the smallest cardinality of a subsetℱη⊆ℱ\\mathcal\{F\}\_\{\\eta\}\\subseteq\\mathcal\{F\}such that, for everyf∈ℱf\\in\\mathcal\{F\}, somef′∈ℱηf^\{\\prime\}\\in\\mathcal\{F\}\_\{\\eta\}satisfies‖f−f′‖∞≤η\\\|f\-f^\{\\prime\}\\\|\_\{\\infty\}\\leq\\eta\. Set𝒩∞=∞\\mathcal\{N\}\_\{\\infty\}=\\inftyif no finite cover exists\. The supremum ranges over all possible groups\.

A uniform cover therefore controls approximation even on groups absent from the sampled data, which allows us to bound finite\-data error\.

###### Assumption 2\(Representation, optimization, and loss complexity\)\.

Fix nonnegative boundsα\\alpha,ξ\\xi, andη\\eta, and an integerM≥1M\\geq 1\. In every round,

infπ∈ΠtLt​\(π\)\\displaystyle\\inf\_\{\\pi\\in\\Pi\_\{t\}\}L\_\{t\}\(\\pi\)≤α\\displaystyle\\leq\\alpha\(representation\),\\displaystyle\\text\{\(representation\)\},\(42\)L^t​\(πt\)−infπ∈ΠtL^t​\(π\)\\displaystyle\\widehat\{L\}\_\{t\}\(\\pi\_\{t\}\)\-\\inf\_\{\\pi\\in\\Pi\_\{t\}\}\\widehat\{L\}\_\{t\}\(\\pi\)≤ξ\\displaystyle\\leq\\xi\(optimization\),\\displaystyle\\text\{\(optimization\)\},𝒩∞​\(η,\{ℓt​\(π,⋅\):π∈Πt\}\)\\displaystyle\\mathcal\{N\}\_\{\\infty\}\(\\eta,\\\{\\ell\_\{t\}\(\\pi;\\cdot\):\\pi\\in\\Pi\_\{t\}\\\}\)≤M\\displaystyle\\leq M\(loss cover\)\.\\displaystyle\\text\{\(loss cover\)\}\.The candidate class and cover are chosen before sampling in each round\.

#### Relation to approximation assumptions in prior work\.

Bounds on approximation and estimation errors also appear in compatible function\-approximation analyses of policy gradients\([Agarwal et al\., 2021](https://arxiv.org/html/2610.20814#bib.bib2), Section 6\)and inexact policy mirror descent\([Yuan et al\., 2023](https://arxiv.org/html/2610.20814#bib.bib48), Section 3\)\. Closure under policy improvement is another related structural condition\([Bhandari and Russo, 2024](https://arxiv.org/html/2610.20814#bib.bib6)\)\. Here we bound population raw\-loss representation error and achieved empirical optimization error; these are distinct from the loss\-cover condition used for generalization\. The Gibbs target satisfying the token constraint does not by itself place it in a chosen model class\. For a tabular class containing that target,r−β​log⁡\(pμ∗/μ\)=β​log⁡Zμ​\(x\)r\-\\beta\\log\(p\_\{\\mu\}^\{\*\}/\\mu\)=\\beta\\log Z\_\{\\mu\}\(x\)is constant over responses, so every group loss is zero and the representation bound can useα=0\\alpha=0\. Loss complexity and achieved empirical accuracy remain separate requirements\.[Agarwal et al\. \(2021, Remark 24\)](https://arxiv.org/html/2610.20814#bib.bib2)give linear\-MDP examples of zero transfer bias for their compatible approximation error; that statement alone does not establish zero raw\-loss representation error here\.

###### Definition 3\(Combined approximation error\)\.

Define

εA:=4​α\+2​ξ\+9​η\.\\varepsilon\_\{A\}:=4\\alpha\+2\\xi\+9\\eta\.\(43\)

The three components describe different sources of error\. Hereα\\alphadescribes how well the candidate class can fit the Gibbs target in population loss, andξ\\xidescribes the accuracy of empirical minimization\. The cover errorη\\etaallows the finite\-data argument to apply to an infinite class\. For a finite class, use all its loss functions as centers, setη=0\\eta=0, and letMMbound its cardinality\.

Increasing the sample size alone does not remove these errors\. The caseεA=0\\varepsilon\_\{A\}=0requires arbitrarily small attainable population loss, exact empirical minimization, and no loss\-cover error\.

#### Behavior mismatch\.

The loss is measured under behavior, whereas reward is evaluated under the returned learner\. Define the full\-response ratioWt​\(y∣x\)=πt​\(y∣x\)/πbeh,t​\(y∣x\)W\_\{t\}\(y\\mid x\)=\\pi\_\{t\}\(y\\mid x\)/\\pi\_\{\\mathrm\{beh\},t\}\(y\\mid x\)and the uniform coefficients

𝒞p\\displaystyle\\mathcal\{C\}\_\{p\}=ess​supt,x,training\(𝔼πbeh,t\(⋅∣x\)Wtp\)1/\(p−1\),\\displaystyle=\\operatorname\*\{ess\\,sup\}\_\{t,x,\\,\\mathrm\{training\}\}\\left\(\\mathbb\{E\}\_\{\\pi\_\{\\mathrm\{beh\},t\}\(\\cdot\\mid x\)\}W\_\{t\}^\{p\}\\right\)^\{1/\(p\-1\)\},2≤p<∞,\\displaystyle 2\\leq p<\\infty,\(44\)𝒞∞\\displaystyle\\mathcal\{C\}\_\{\\infty\}=ess​supt,x,y,training⁡Wt​\(y∣x\)\.\\displaystyle=\\operatorname\*\{ess\\,sup\}\_\{t,x,y,\\,\\mathrm\{training\}\}W\_\{t\}\(y\\mid x\)\.The supremum includes all training randomness and the data\-dependent returned policy\. A uniform bound over the candidate class before sampling is sufficient but stronger than necessary\. These coefficients describe distributional coverage, separately from the loss\-cover sizeMM;log⁡𝒞p\\log\\mathcal\{C\}\_\{p\}is the uniform order\-ppRényi divergence\([van Erven and Harremoës, 2014](https://arxiv.org/html/2610.20814#bib.bib44)\)\.

A tokenwise sufficient condition usesbt,h=πbeh,t,h\(⋅∣sh\)b\_\{t,h\}=\\pi\_\{\\mathrm\{beh\},t,h\}\(\\cdot\\mid s\_\{h\}\)andwt,h​\(a\)=πt,h​\(a∣sh\)/bt,h​\(a\)w\_\{t,h\}\(a\)=\\pi\_\{t,h\}\(a\\mid s\_\{h\}\)/b\_\{t,h\}\(a\), wheresh=\(x,y<h\)s\_\{h\}=\(x,y\_\{<h\}\)\. LetVlog<∞V\_\{\\log\}<\\inftybe a deterministic bound such that

ess​supt,training,x,y∑h=1H𝔼a∼πt,h\(⋅∣sh\)\[\(logwt,h\(a\)\)2\]≤Vlog\.\\operatorname\*\{ess\\,sup\}\_\{t,\\,\\mathrm\{training\},\\,x,y\}\\sum\_\{h=1\}^\{H\}\\mathbb\{E\}\_\{a\\sim\\pi\_\{t,h\}\(\\cdot\\mid s\_\{h\}\)\}\[\(\\log w\_\{t,h\}\(a\)\)^\{2\}\]\\leq V\_\{\\log\}\.\(45\)The sum follows one supported response path before taking the worst case over paths, prompts, rounds, and training randomness\. For a chosen0<ν≤10<\\nu\\leq 1, the additional bound

\|log⁡wt,h​\(a\)\|≤min⁡\{log⁡2,3​ν/2\}\|\\log w\_\{t,h\}\(a\)\|\\leq\\min\\\{\\log 2,3\\nu/2\\\}\(46\)uniformly over rounds, legal histories, actions, and training randomness implies, by Lemma[15](https://arxiv.org/html/2610.20814#Thmtheorem15),

𝒞1\+1/ν≤cν,cν:=exp⁡\(\(1\+ν−1\)​Vlog\)\.\\mathcal\{C\}\_\{1\+1/\\nu\}\\leq c\_\{\\nu\},\\qquad c\_\{\\nu\}:=\\exp\(\(1\+\\nu^\{\-1\}\)V\_\{\\log\}\)\.\(47\)This bound holds for any finiteVlogV\_\{\\log\}, with its size entering throughcνc\_\{\\nu\}\. The learner\-to\-behavior token condition is separate from the learner\-to\-reference constraint in Assumption[1](https://arxiv.org/html/2610.20814#Thmassumption1)\. Both conditions are used in the analysis; the loss contains no learner\-to\-behavior importance weights\.

Finally, the initial policy needs to assign enough probability to correct responses for repeated Gibbs updates to reach the desired reward\. Fix a comparator policyπ⋆\\pi\_\{\\star\}and use the following coverage condition\.

###### Assumption 3\(Initial correct\-response coverage\)\.

There is a constantC⋆≥1C\_\{\\star\}\\geq 1such that

Pry∼π⋆\(⋅∣x\)\[r\(x,y\)=1\]≤C⋆Pry∼π0\(⋅∣x\)\[r\(x,y\)=1\]for every promptx\.\\Pr\_\{y\\sim\\pi\_\{\\star\}\(\\cdot\\mid x\)\}\[r\(x,y\)=1\]\\leq C\_\{\\star\}\\Pr\_\{y\\sim\\pi\_\{0\}\(\\cdot\\mid x\)\}\[r\(x,y\)=1\]\\qquad\\text\{for every prompt $x$\}\.\(48\)

ThusC⋆C\_\{\\star\}bounds how much more often the comparator can succeed than the initialization\. Best\-of\-KKsampling fromπ0\\pi\_\{0\}satisfies this condition withC⋆=KC\_\{\\star\}=K: the probability of at least one correct response amongKKindependent draws is at mostKKtimes the single\-response success probability\.

###### Theorem 13\(Final reward and finite\-data convergence\)\.

Suppose Assumptions[1](https://arxiv.org/html/2610.20814#Thmassumption1),[2](https://arxiv.org/html/2610.20814#Thmassumption2), and[3](https://arxiv.org/html/2610.20814#Thmassumption3)hold, and use the sampling procedure above\. Fixε,δ∈\(0,1\)\\varepsilon,\\delta\\in\(0,1\)and set

T=⌈β​log⁡6​C⋆ε⌉,z=log⁡\(2​T​M/δ\),X=C⋆​T2/ε\.T=\\left\\lceil\\beta\\log\\frac\{6C\_\{\\star\}\}\{\\varepsilon\}\\right\\rceil,\\qquad z=\\log\(2TM/\\delta\),\\qquad X=C\_\{\\star\}T^\{2\}/\\varepsilon\.\(49\)For a fixed real2≤p<∞2\\leq p<\\inftywith𝒞p<∞\\mathcal\{C\}\_\{p\}<\\infty, letν=1/\(p−1\)\\nu=1/\(p\-1\)and choose

n≥⌈max⁡\{1,768​π2​z​X,24​z​\(32​π2​X\)1\+ν​𝒞p\}⌉\.n\\geq\\left\\lceil\\max\\left\\\{1,768\\pi^\{2\}zX,24z\(32\\pi^\{2\}X\)^\{1\+\\nu\}\\mathcal\{C\}\_\{p\}\\right\\\}\\right\\rceil\.\(50\)With probability at least1−δ1\-\\deltaover the sampled training groups,

J⁡\(π⋆\)−J⁡\(πT\)≤ε\+2​π2​C⋆​T2β​\[4​εA\+4​β1/p​\(𝒞pp−1−1\)1/p​εA1−1/p\]\.J\(\\pi\_\{\\star\}\)\-J\(\\pi\_\{T\}\)\\leq\\varepsilon\+\\frac\{2\\pi^\{2\}C\_\{\\star\}T^\{2\}\}\{\\beta\}\\left\[4\\varepsilon\_\{A\}\+4\\beta^\{1/p\}\(\\mathcal\{C\}\_\{p\}^\{p\-1\}\-1\)^\{1/p\}\\varepsilon\_\{A\}^\{1\-1/p\}\\right\]\.\(51\)HereεA=4​α\+2​ξ\+9​η\\varepsilon\_\{A\}=4\\alpha\+2\\xi\+9\\etais defined in Eq\. \([43](https://arxiv.org/html/2610.20814#A2.E43)\), andJJis expected single\-response reward\. Under Eqs\. \([45](https://arxiv.org/html/2610.20814#A2.E45)\) and \([46](https://arxiv.org/html/2610.20814#A2.E46)\), one may replace𝒞p\\mathcal\{C\}\_\{p\}bycνc\_\{\\nu\}in both the budget and fitting residual\. Choosing the smallest permitted integer gives

n=O⁡\(z​X\)\+O⁡\(z​cν​X1\+ν\),0<ν≤1,n=O\(zX\)\+O\(zc\_\{\\nu\}X^\{1\+\\nu\}\),\\qquad 0<\\nu\\leq 1,with absolute constants uniformly inν\\nu: the two terms can use768​π2768\\pi^\{2\}and24​\(32​π2\)224\(32\\pi^\{2\}\)^\{2\}, respectively\.

Each run usesT​nTnfresh prompt groups andm​T​nmTnscored responses\. The scalarπ\\piin the constants is the circle constant\.

#### Interpretation\.

The budgets control statistical reward error byε\\varepsilon\. Prompt draws may repeat, but additional groups on a fixed prompt pool do not supply fresh independent draws fromρ\\rho\. The fitting residual retains representation, achieved empirical optimization, and loss\-cover errors; increasingnnalone does not remove them\. WhenεA=0\\varepsilon\_\{A\}=0, it vanishes\. Applying the result to a chosen model requires these error bounds as well as the actual policy constraints\. The proof in Appendix[B\.5](https://arxiv.org/html/2610.20814#A2.SS5)combines raw\-loss calibration, finite\-data concentration, and relative angular transfer\.

We first relate group loss to the error in one Gibbs update\. We then track how errors in these updates affect the final success probability\. For a reference policyμ\\mu, defineFμ​\(π\)=𝔼X​Vβ​\(π,X\)F\_\{\\mu\}\(\\pi\)=\\mathbb\{E\}\_\{X\}V\_\{\\beta\}\(\\pi;X\)usingμ\\muin place ofπref\\pi\_\{\\mathrm\{ref\}\}in Eq\. \([1](https://arxiv.org/html/2610.20814#S3.E1)\)\. Letpμ∗p\_\{\\mu\}^\{\*\}be its Gibbs maximizer and writeFμ∗=Fμ​\(pμ∗\)F\_\{\\mu\}^\{\*\}=F\_\{\\mu\}\(p\_\{\\mu\}^\{\*\}\)\. Write𝒦\(π,pμ∗\)=𝔼XDKL\(π\(⋅∣X\)∥pμ∗\(⋅∣X\)\)\\mathcal\{K\}\(\\pi,p\_\{\\mu\}^\{\*\}\)=\\mathbb\{E\}\_\{X\}D\_\{\\mathrm\{KL\}\}\(\\pi\(\\cdot\\mid X\)\\\|p\_\{\\mu\}^\{\*\}\(\\cdot\\mid X\)\)\. The Gibbs identity givesΔt=Fπt−1∗−Fπt−1​\(πt\)=β​𝒦​\(πt,pπt−1∗\)\\Delta\_\{t\}=F\_\{\\pi\_\{t\-1\}\}^\{\*\}\-F\_\{\\pi\_\{t\-1\}\}\(\\pi\_\{t\}\)=\\beta\\mathcal\{K\}\(\\pi\_\{t\},p\_\{\\pi\_\{t\-1\}\}^\{\*\}\)\. Throughout,osc\\operatorname\{osc\}means maximum minus minimum over responses\.

### B\.2Probability transfer and raw\-loss calibration

The next two lemmas restate the probability results of[Bai et al\. \(2026, Lemmas F\.1 and F\.2\)](https://arxiv.org/html/2610.20814#bib.bib5)in our notation\. Their proofs are given there\.

###### Lemma 15\(Sequential log moments\)\.

Fix a prompt and realized positive learner and behavior policies on a common finite autoregressive support of at mostHHtokens\. At each history, letwh​\(a\)=πh​\(a∣sh\)/πbeh,h​\(a∣sh\)w\_\{h\}\(a\)=\\pi\_\{h\}\(a\\mid s\_\{h\}\)/\\pi\_\{\\mathrm\{beh\},h\}\(a\\mid s\_\{h\}\)\. Pad terminated paths with ratio one\. WriteYh=log⁡whY\_\{h\}=\\log w\_\{h\}andW=∏h=1HwhW=\\prod\_\{h=1\}^\{H\}w\_\{h\}\. Suppose the sum of𝔼πh​Yh2\\mathbb\{E\}\_\{\\pi\_\{h\}\}Y\_\{h\}^\{2\}along each supported path is at mostVlogV\_\{\\log\},wh≥1/2w\_\{h\}\\geq 1/2, andYh−𝔼πh​Yh≤bY\_\{h\}\-\\mathbb\{E\}\_\{\\pi\_\{h\}\}Y\_\{h\}\\leq bat every history for a deterministicb\>0b\>0\. Then for everyλ≥0\\lambda\\geq 0,

log⁡𝔼beh​W1\+λ≤Vlog​\[λ\+eb​λ−1−b​λb2\]\.\\log\\mathbb\{E\}\_\{\\mathrm\{beh\}\}W^\{1\+\\lambda\}\\leq V\_\{\\log\}\\left\[\\lambda\+\\frac\{e^\{b\\lambda\}\-1\-b\\lambda\}\{b^\{2\}\}\\right\]\.\(54\)Forb​λ<3b\\lambda<3, the right side is at mostVlog​\[λ\+λ2/\(2​\(1−b​λ/3\)\)\]V\_\{\\log\}\[\\lambda\+\\lambda^\{2\}/\(2\(1\-b\\lambda/3\)\)\]\. The same bounds hold for every prefix ratio\. Under Eq\. \([46](https://arxiv.org/html/2610.20814#A2.E46)\), takeλ=1/ν\\lambda=1/\\nuandb=3​ν/2b=3\\nu/2to obtain Eq\. \([47](https://arxiv.org/html/2610.20814#A2.E47)\)\.

The conditional mean𝔼πh​Yh\\mathbb\{E\}\_\{\\pi\_\{h\}\}Y\_\{h\}is a nonnegative KL divergence, so an upper bound onYhY\_\{h\}also bounds its centered value\. These statements are conditional on the realized policies; the deterministic uniform budget in Eq\. \([45](https://arxiv.org/html/2610.20814#A2.E45)\) covers prompts, rounds, and training randomness\. See[Bai et al\. \(2026, Lemma F\.1, proof\)](https://arxiv.org/html/2610.20814#bib.bib5)\.

###### Lemma 16\(Bounded\-function moment transfer\)\.

For distributionsP,QP,Qon a common finite support, letW=d​P/d​QW=dP/dQ,0≤f≤Bf0\\leq f\\leq B\_\{f\}, anda=𝔼Q​fa=\\mathbb\{E\}\_\{Q\}f\. For every realp≥2p\\geq 2,

𝔼P​f≤a\+\(𝔼Q​Wp−1\)1/p​Bf1/p​a1−1/p\.\\mathbb\{E\}\_\{P\}f\\leq a\+\(\\mathbb\{E\}\_\{Q\}W^\{p\}\-1\)^\{1/p\}B\_\{f\}^\{1/p\}a^\{1\-1/p\}\.IfW≤𝒞∞W\\leq\\mathcal\{C\}\_\{\\infty\}almost surely, then𝔼P​f≤𝒞∞​a\\mathbb\{E\}\_\{P\}f\\leq\\mathcal\{C\}\_\{\\infty\}a\.

See[Bai et al\. \(2026, Lemma F\.2, proof\)](https://arxiv.org/html/2610.20814#bib.bib5)for both inequalities\.

###### Lemma 17\(Raw\-loss calibration\)\.

Let learnerπ\\piand behaviorbbbe positive on a common support, and let the response score satisfyoscy⁡\(R/β\)≤1\\operatorname\{osc\}\_\{y\}\(R/\\beta\)\\leq 1at every prompt\. Form≥2m\\geq 2independent responses frombb, letLLbe the expected raw group loss, averaged over prompts, and define

Gπ\(R\)=𝔼X∼ρ\[βlog𝔼y∼π\(⋅∣X\)eR⁡\(X,y\)/β−𝔼y∼π\(⋅∣X\)R\(X,y\)\]\.G\_\{\\pi\}\(R\)=\\mathbb\{E\}\_\{X\\sim\\rho\}\\left\[\\beta\\log\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid X\)\}e^\{R\(X,y\)/\\beta\}\-\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid X\)\}R\(X,y\)\\right\]\.Ifsupx𝔼b​\(π/b\)p≤𝒞pp−1\\sup\_\{x\}\\mathbb\{E\}\_\{b\}\(\\pi/b\)^\{p\}\\leq\\mathcal\{C\}\_\{p\}^\{p\-1\}for2≤p<∞2\\leq p<\\infty, then

Gπ​\(R\)≤4​L\+4​β1/p​\(𝒞pp−1−1\)1/p​L1−1/p\.G\_\{\\pi\}\(R\)\\leq 4L\+4\\beta^\{1/p\}\(\\mathcal\{C\}\_\{p\}^\{p\-1\}\-1\)^\{1/p\}L^\{1\-1/p\}\.\(55\)If insteadπ/b≤𝒞∞\\pi/b\\leq\\mathcal\{C\}\_\{\\infty\}uniformly, thenGπ​\(R\)≤4​𝒞∞​LG\_\{\\pi\}\(R\)\\leq 4\\mathcal\{C\}\_\{\\infty\}L\. ForR=r−β​log⁡\(π/πref\)R=r\-\\beta\\log\(\\pi/\\pi\_\{\\mathrm\{ref\}\}\),Gπ​\(R\)=Fπref∗−Fπref​\(π\)G\_\{\\pi\}\(R\)=F\_\{\\pi\_\{\\mathrm\{ref\}\}\}^\{\*\}\-F\_\{\\pi\_\{\\mathrm\{ref\}\}\}\(\\pi\)\.

###### Proof\.

Fix a prompt and any response distributionPP\. Forw=R/βw=R/\\betaandg⁡\(τ\)=log⁡𝔼P​eτ​wg\(\\tau\)=\\log\\mathbb\{E\}\_\{P\}e^\{\\tau w\}, tilting changes densities by factors betweene−τe^\{\-\\tau\}andeτe^\{\\tau\}\. The minimum\-over\-centers formula for variance givese−τ​VarP​w≤g′′​\(τ\)≤eτ​VarP​we^\{\-\\tau\}\\mathrm\{Var\}\_\{P\}w\\leq g^\{\\prime\\prime\}\(\\tau\)\\leq e^\{\\tau\}\\mathrm\{Var\}\_\{P\}w\. Integrating with weight1−τ1\-\\tauyields

VarP​Re​β≤β​log⁡𝔼P​eR/β−𝔼P​R≤e−2β​VarP​R\.\\frac\{\\mathrm\{Var\}\_\{P\}R\}\{e\\beta\}\\leq\\beta\\log\\mathbb\{E\}\_\{P\}e^\{R/\\beta\}\-\\mathbb\{E\}\_\{P\}R\\leq\\frac\{e\-2\}\{\\beta\}\\mathrm\{Var\}\_\{P\}R\.\(56\)For the uniform distribution on a sampled group, the expected empirical variance is\(1−1/m\)​Varb​R\(1\-1/m\)\\mathrm\{Var\}\_\{b\}R\. Hence the conditional lossLxL\_\{x\}satisfiesv:=Varb​R≤2​e​β​Lxv:=\\mathrm\{Var\}\_\{b\}R\\leq 2e\\beta L\_\{x\}\. Putu=R−𝔼b​Ru=R\-\\mathbb\{E\}\_\{b\}R, so\|u\|≤β\|u\|\\leq\\beta\. Apply Lemma[16](https://arxiv.org/html/2610.20814#Thmtheorem16)withP=πP=\\pi,Q=bQ=b,f=u2f=u^\{2\},Bf=β2B\_\{f\}=\\beta^\{2\}, andDp=𝒞pp−1−1D\_\{p\}=\\mathcal\{C\}\_\{p\}^\{p\-1\}\-1:

Varπ​R≤𝔼π​u2≤v\+Dp1/p​β2/p​v1−1/p\.\\mathrm\{Var\}\_\{\\pi\}R\\leq\\mathbb\{E\}\_\{\\pi\}u^\{2\}\\leq v\+D\_\{p\}^\{1/p\}\\beta^\{2/p\}v^\{1\-1/p\}\.The upper curvature bound now gives

Gπ,x​\(R\)≤2​e​\(e−2\)​Lx\+\(e−2\)​\(2​e\)1−1/p​β1/p​Dp1/p​Lx1−1/p\.G\_\{\\pi,x\}\(R\)\\leq 2e\(e\-2\)L\_\{x\}\+\(e\-2\)\(2e\)^\{1\-1/p\}\\beta^\{1/p\}D\_\{p\}^\{1/p\}L\_\{x\}^\{1\-1/p\}\.Both coefficients are below44\. Average over prompts and use concavity ofLx1−1/pL\_\{x\}^\{1\-1/p\}\. Under bounded coverage the same local transfer lemma givesVarπ​R≤𝒞∞​v\\mathrm\{Var\}\_\{\\pi\}R\\leq\\mathcal\{C\}\_\{\\infty\}v, proving the second conclusion\. ∎

### B\.3Oracle inequality for each refresh round

The next lemma compares the population raw loss of each returned policy with the best population loss in that round’s candidate class\. Its probability statement holds simultaneously over all refresh rounds, so it applies to the entire sequence used in the final\-reward analysis\.

###### Lemma 18\(Per\-refresh oracle inequality\)\.

Suppose Assumptions[1](https://arxiv.org/html/2610.20814#Thmassumption1)and[2](https://arxiv.org/html/2610.20814#Thmassumption2)hold\. In each ofTTrounds, freeze the reference, behavior policy, candidate class, and loss cover before samplingnnfresh independent prompt groups as in Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1)\. Each group containsm≥2m\\geq 2independent behavior\-policy responses and is unfiltered\. Letπt∈Πt\\pi\_\{t\}\\in\\Pi\_\{t\}be the returned policy with empirical suboptimality at mostξ\\xi, as specified in Eq\. \([42](https://arxiv.org/html/2610.20814#A2.E42)\)\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setz=log⁡\(2​T​M/δ\)z=\\log\(2TM/\\delta\)\. With probability at least1−δ1\-\\delta, simultaneously for everyt=1,…,Tt=1,\\ldots,T,

Lt​\(πt\)≤4​infπ∈ΠtLt​\(π\)\+2​ξ\+24​β​zn\+9​η\.L\_\{t\}\(\\pi\_\{t\}\)\\leq 4\\inf\_\{\\pi\\in\\Pi\_\{t\}\}L\_\{t\}\(\\pi\)\+2\\xi\+\\frac\{24\\beta z\}\{n\}\+9\\eta\.\(57\)In particular, withεA=4​α\+2​ξ\+9​η\\varepsilon\_\{A\}=4\\alpha\+2\\xi\+9\\eta,

Lt​\(πt\)≤εA\+24​β​zn\.L\_\{t\}\(\\pi\_\{t\}\)\\leq\\varepsilon\_\{A\}\+\\frac\{24\\beta z\}\{n\}\.\(58\)

###### Proof\.

Condition on all data and policies from the preceding rounds\. The current reference, behavior, candidate class, and loss cover are then fixed, and the new groups are independent\. Assumption[1](https://arxiv.org/html/2610.20814#Thmassumption1)bounds the adjusted\-reward range by1\+2​H≤β1\+2H\\leq\\beta, so every candidate’s raw group loss lies in\[0,β\]\[0,\\beta\]\.

Apply the raw\-loss oracle inequality in Proposition[8](https://arxiv.org/html/2610.20814#Thmtheorem8), Eq\. \([26](https://arxiv.org/html/2610.20814#A1.E26)\), withΛ=β\\Lambda=\\beta, candidate classΠt\\Pi\_\{t\}, and failure probabilityδ/T\\delta/T\. Its population and empirical losses are nowLtL\_\{t\}andL^t\\widehat\{L\}\_\{t\}\. The round’s loss\-cover size is at mostMM, so the logarithmic term in that proposition is at most

log⁡2​Mδ/T=log⁡2​M​Tδ=z\.\\log\\frac\{2M\}\{\\delta/T\}=\\log\\frac\{2MT\}\{\\delta\}=z\.The proposition therefore gives, with conditional probability at least1−δ/T1\-\\delta/T,

Lt​\(πt\)≤4​infπ∈ΠtLt​\(π\)\+2​ξ\+12​β​zn\+9​η\.L\_\{t\}\(\\pi\_\{t\}\)\\leq 4\\inf\_\{\\pi\\in\\Pi\_\{t\}\}L\_\{t\}\(\\pi\)\+2\\xi\+\\frac\{12\\beta z\}\{n\}\+9\\eta\.Here the statistical term is3​γn3\\gamma\_\{n\}withγn≤4​β​z/n\+3​η\\gamma\_\{n\}\\leq 4\\beta z/n\+3\\eta\. This application uses only the raw\-loss conclusion of the proposition; its best\-of\-KKreward assumptions are not needed\.

The conditional failure bound holds for every preceding history\. Averaging over that history gives unconditional failure probability at mostδ/T\\delta/Tfor each round\. A union bound over theTTrounds then gives one event of probability at least1−δ1\-\\deltaon which all the round\-wise oracle inequalities hold\. Loosening1212to2424gives Eq\. \([57](https://arxiv.org/html/2610.20814#A2.E57)\)\. Finally, the representation boundinfπ∈ΠtLt​\(π\)≤α\\inf\_\{\\pi\\in\\Pi\_\{t\}\}L\_\{t\}\(\\pi\)\\leq\\alphaand the definitionεA=4​α\+2​ξ\+9​η\\varepsilon\_\{A\}=4\\alpha\+2\\xi\+9\\etagive Eq\. \([58](https://arxiv.org/html/2610.20814#A2.E58)\)\. ∎

### B\.4Reference refresh and final reward

We use the relative angular transfer bound of[Bai et al\. \(2026, Lemma F\.9\)](https://arxiv.org/html/2610.20814#bib.bib5), restated below in our notation\. It converts the per\-round Gibbs gaps into a final\-policy reward bound\.

###### Lemma 19\(Polynomial angular propagation\)\.

Suppose rewards are binary, the KL coefficient in \([1](https://arxiv.org/html/2610.20814#S3.E1)\) isβ\>0\\beta\>0, and the comparator has coverage constantC⋆C\_\{\\star\}as defined in Appendix[B\.1](https://arxiv.org/html/2610.20814#A2.SS1)\. For a positive reference policyμ\\mu, recall the prompt\-averaged regularized objective and its optimal value:

Fμ\(π\):=𝔼X∼ρ\[𝔼Y∼π\(⋅∣X\)r\(X,Y\)−βDKL\(π\(⋅∣X\)∥μ\(⋅∣X\)\)\],Fμ∗:=maxπFμ\(π\),F\_\{\\mu\}\(\\pi\):=\\mathbb\{E\}\_\{X\\sim\\rho\}\\\!\\left\[\\mathbb\{E\}\_\{Y\\sim\\pi\(\\cdot\\mid X\)\}r\(X,Y\)\-\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\(\\cdot\\mid X\)\\\|\\mu\(\\cdot\\mid X\)\)\\right\],\\qquad F\_\{\\mu\}^\{\*\}:=\\max\_\{\\pi\}F\_\{\\mu\}\(\\pi\),where the maximum is over all response policies on the common support\. Fixs∈\(0,1\)s\\in\(0,1\)andT≥⌈β​log⁡\(3​C⋆/s\)⌉T\\geq\\lceil\\beta\\log\(3C\_\{\\star\}/s\)\\rceil\. For any positive policy sequenceπ0,…,πT\\pi\_\{0\},\\ldots,\\pi\_\{T\}, setΔt=Fπt−1∗−Fπt−1​\(πt\)\\Delta\_\{t\}=F\_\{\\pi\_\{t\-1\}\}^\{\*\}\-F\_\{\\pi\_\{t\-1\}\}\(\\pi\_\{t\}\)\. Then

J⁡\(π⋆\)−J⁡\(πT\)≤s\+2​π2​C⋆β​\(∑t=1TΔt\)2\.J\(\\pi\_\{\\star\}\)\-J\(\\pi\_\{T\}\)\\leq s\+\\frac\{2\\pi^\{2\}C\_\{\\star\}\}\{\\beta\}\\left\(\\sum\_\{t=1\}^\{T\}\\sqrt\{\\Delta\_\{t\}\}\\right\)^\{2\}\.\(59\)In particular,

J⁡\(π⋆\)−J⁡\(πT\)≤s\+2​π2​C⋆​Tβ​∑t=1TΔt≤s\+2​π2​C⋆​T2β​maxt​Δt\.J\(\\pi\_\{\\star\}\)\-J\(\\pi\_\{T\}\)\\leq s\+\\frac\{2\\pi^\{2\}C\_\{\\star\}T\}\{\\beta\}\\sum\_\{t=1\}^\{T\}\\Delta\_\{t\}\\leq s\+\\frac\{2\\pi^\{2\}C\_\{\\star\}T^\{2\}\}\{\\beta\}\\max\_\{t\}\\Delta\_\{t\}\.\(60\)HereC⋆C\_\{\\star\}controls initial correct\-solution coverage and the scalarπ\\piis the circle constant\. The error coefficient grows polynomially withTT\.

The cited result assumes binary rewards, positive policies, and the initial correct\-response coverage in Assumption[3](https://arxiv.org/html/2610.20814#Thmassumption3)\. Its round countKKis ourTT, and its proximal gapDt−1=β𝔼XDKL\(πt∥pπt−1∗\)D\_\{t\-1\}=\\beta\\mathbb\{E\}\_\{X\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{t\}\\\|p\_\{\\pi\_\{t\-1\}\}^\{\*\}\)equalsΔt\\Delta\_\{t\}by the Gibbs identity above\. This is a deterministic result for the realized policy sequence; its proof is given in the cited lemma\. Substituting gives Eq\. \([59](https://arxiv.org/html/2610.20814#A2.E59)\); Cauchy–Schwarz gives Eq\. \([60](https://arxiv.org/html/2610.20814#A2.E60)\)\.

### B\.5Proof of Theorem[13](https://arxiv.org/html/2610.20814#Thmtheorem13)

###### Proof of Theorem[13](https://arxiv.org/html/2610.20814#Thmtheorem13)and Remark[14](https://arxiv.org/html/2610.20814#Thmtheorem14)\.

Step 1: Bound the population raw loss in every round\.Apply Lemma[18](https://arxiv.org/html/2610.20814#Thmtheorem18)withz=log⁡\(2​T​M/δ\)z=\\log\(2TM/\\delta\)as in Eq\. \([49](https://arxiv.org/html/2610.20814#A2.E49)\)\. With probability at least1−δ1\-\\delta, the population\-loss bound in Eq\. \([58](https://arxiv.org/html/2610.20814#A2.E58)\) holds for every round\. Work on this simultaneous event for the remainder of the proof\. The moment bound is uniform over training randomness and therefore applies to every realized returned policy\.

Step 2: Convert raw loss into a per\-refresh optimality gap\.Apply Lemma[17](https://arxiv.org/html/2610.20814#Thmtheorem17)toR=r−β​log⁡\(πt/πt−1\)R=r\-\\beta\\log\(\\pi\_\{t\}/\\pi\_\{t\-1\}\), whose score range is bounded by the local reference and temperature conditions\. For finitepp, putq=1−1/pq=1\-1/pandDp=𝒞pp−1−1D\_\{p\}=\\mathcal\{C\}\_\{p\}^\{p\-1\}\-1\. Lemma[17](https://arxiv.org/html/2610.20814#Thmtheorem17)and\(a\+b\)q≤aq\+bq\(a\+b\)^\{q\}\\leq a^\{q\}\+b^\{q\}give

Δt≤4​εA\+4​β1/p​Dp1/p​εAq\+96​β​zn\+4​β​Dp1/p​\(24​z/n\)q\.\\Delta\_\{t\}\\leq 4\\varepsilon\_\{A\}\+4\\beta^\{1/p\}D\_\{p\}^\{1/p\}\\varepsilon\_\{A\}^\{q\}\+\\frac\{96\\beta z\}\{n\}\+4\\beta D\_\{p\}^\{1/p\}\(24z/n\)^\{q\}\.Setζ=β/\(4​π2​X\)\\zeta=\\beta/\(4\\pi^\{2\}X\)\. The first statistical term is at mostζ/2\\zeta/2whenn≥768​π2​z​Xn\\geq 768\\pi^\{2\}zX\. The second is at mostζ/2\\zeta/2whenn≥24​z​\(32​π2​X\)1\+ν​Dpνn\\geq 24z\(32\\pi^\{2\}X\)^\{1\+\\nu\}D\_\{p\}^\{\\nu\}; useDpν≤𝒞pD\_\{p\}^\{\\nu\}\\leq\\mathcal\{C\}\_\{p\}to obtain Eq\. \([50](https://arxiv.org/html/2610.20814#A2.E50)\)\. Under bounded coverage, Lemma[17](https://arxiv.org/html/2610.20814#Thmtheorem17)givesΔt≤4​𝒞∞​Lt​\(πt\)\\Delta\_\{t\}\\leq 4\\mathcal\{C\}\_\{\\infty\}L\_\{t\}\(\\pi\_\{t\}\)\. Its statistical term96​𝒞∞​β​z/n96\\mathcal\{C\}\_\{\\infty\}\\beta z/nis at mostζ\\zetaunder Eq\. \([52](https://arxiv.org/html/2610.20814#A2.E52)\)\.

Step 3: Propagate the gaps to the final reward\.Apply Lemma[19](https://arxiv.org/html/2610.20814#Thmtheorem19), using Eq\. \([60](https://arxiv.org/html/2610.20814#A2.E60)\) withs=ε/2s=\\varepsilon/2, on the simultaneous event from Step 1\. The finite\-moment gap bound gives Eq\. \([51](https://arxiv.org/html/2610.20814#A2.E51)\), and the bounded\-coverage gap bound gives Eq\. \([53](https://arxiv.org/html/2610.20814#A2.E53)\)\. This proves the bounded\-coverage case of Remark[14](https://arxiv.org/html/2610.20814#Thmtheorem14); its fourth\-moment case follows by settingp=4p=4in the finite\-moment bound\. Finally, Lemma[15](https://arxiv.org/html/2610.20814#Thmtheorem15)supplies𝒞p≤cν\\mathcal\{C\}\_\{p\}\\leq c\_\{\\nu\}under the stated token conditions, and\(32​π2\)1\+ν≤\(32​π2\)2\(32\\pi^\{2\}\)^\{1\+\\nu\}\\leq\(32\\pi^\{2\}\)^\{2\}for0<ν≤10<\\nu\\leq 1gives the absolute order constants\. Only groups count as independent prompt observations\. ∎

#### Token moments and sequence coverage\.

The coefficient𝒞∞\\mathcal\{C\}\_\{\\infty\}controls a full\-response ratio, whereasVlogV\_\{\\log\}aggregates conditional token second moments along a path\. For example,\|log⁡wh\|≤a/H\|\\log w\_\{h\}\|\\leq a/\\sqrt\{H\}allowsVlog=a2V\_\{\\log\}=a^\{2\}and the sequence\-ratio envelopeexp⁡\(a​H\)\\exp\(a\\sqrt\{H\}\)\. When the token cap also holds, the Bernstein coefficient is at mostexp⁡\(\(1\+ν−1\)​a2\)\\exp\(\(1\+\\nu^\{\-1\}\)a^\{2\}\)\. The representation and achieved optimization errors remain separate conditions on the model class and solver\. Uniform coverage is the infinite\-order branch because𝔼b​Wp≤𝒞∞p−1​𝔼b​W=𝒞∞p−1\\mathbb\{E\}\_\{b\}W^\{p\}\\leq\\mathcal\{C\}\_\{\\infty\}^\{p\-1\}\\mathbb\{E\}\_\{b\}W=\\mathcal\{C\}\_\{\\infty\}^\{p\-1\}\. If a returned learner equals behavior, its response ratio is one and the direct calibration reduces toΔt≤4​Lt​\(πt\)\\Delta\_\{t\}\\leq 4L\_\{t\}\(\\pi\_\{t\}\)\. In the analyzed procedure behavior is frozen before fitting, so sampling from the current reference generally still produces learner–behavior mismatch\.

### B\.6The Gibbs target and behavior lag

The conditional\-normalizer calculation after Assumption[1](https://arxiv.org/html/2610.20814#Thmassumption1)shows that the exact Gibbs target satisfies the local constraint\. Under its relaxed version\|log⁡\(πh/πt−1,h\)\|≤\(1\+γ\)/β\|\\log\(\\pi\_\{h\}/\\pi\_\{t\-1,h\}\)\|\\leq\(1\+\\gamma\)/\\betaforγ≥0\\gamma\\geq 0, the score range is at most1\+2​H​\(1\+γ\)1\+2H\(1\+\\gamma\)\. Thusβ≥1\+2​H​\(1\+γ\)\\beta\\geq 1\+2H\(1\+\\gamma\)preserves the calibration and concentration arguments, provided representation and loss complexity hold for the relaxed class\.

For stored\-policy behaviorπbeh,t=πt−dt\\pi\_\{\\mathrm\{beh\},t\}=\\pi\_\{t\-d\_\{t\}\}with1≤dt≤min⁡\(d,t\)1\\leq d\_\{t\}\\leq\\min\(d,t\), telescoping gives\|log⁡wt,h\|≤d⁡\(1\+γ\)/β\|\\log w\_\{t,h\}\|\\leq d\(1\+\\gamma\)/\\betaand one may takeVlog=H​d2​\(1\+γ\)2/β2V\_\{\\log\}=Hd^\{2\}\(1\+\\gamma\)^\{2\}/\\beta^\{2\}\. This supplies the Bernstein token condition whend⁡\(1\+γ\)/β≤min⁡\{log⁡2,3​ν/2\}d\(1\+\\gamma\)/\\beta\\leq\\min\\\{\\log 2,3\\nu/2\\\}\. Independently of this cap, multiplying token ratios gives𝒞∞≤exp⁡\(H​d​\(1\+γ\)/β\)\\mathcal\{C\}\_\{\\infty\}\\leq\\exp\(Hd\(1\+\\gamma\)/\\beta\), so the direct branch has a sufficient budgetn=O⁡\(z​X​exp⁡\(H​d​\(1\+γ\)/β\)\)n=O\(zX\\exp\(Hd\(1\+\\gamma\)/\\beta\)\)\. For behavior equal to the current reference,d=1d=1; the relaxed temperature condition makes this coverage coefficient smaller thane\\sqrt\{e\}\. The moment branch instead usescν=exp⁡\(\(1\+ν−1\)​H​d2​\(1\+γ\)2/β2\)c\_\{\\nu\}=\\exp\(\(1\+\\nu^\{\-1\}\)Hd^\{2\}\(1\+\\gamma\)^\{2\}/\\beta^\{2\}\)with anX1\+νX^\{1\+\\nu\}term\. Which budget is smaller depends on both these coefficients and the target accuracy\. The lagddcounts reference rounds, separately from experimental synchronization delay\. Additional training–inference mismatch requires bounds on the actual learner\-to\-behavior ratios in Eqs\. \([45](https://arxiv.org/html/2610.20814#A2.E45)\) and \([46](https://arxiv.org/html/2610.20814#A2.E46)\)\.

#### Alternative bound from token spans\.

Letbhspanb\_\{h\}^\{\\rm span\}be deterministic bounds, uniform over histories, prompts, rounds, and training randomness, on the span across actions oflog⁡wh\\log w\_\{h\}\. Putr=∑h\(bhspan\)2/8r=\\sum\_\{h\}\(b\_\{h\}^\{\\rm span\}\)^\{2\}/8\. At each history writeYh=log⁡whY\_\{h\}=\\log w\_\{h\}\. Normalization𝔼πh​e−Yh=1\\mathbb\{E\}\_\{\\pi\_\{h\}\}e^\{\-Y\_\{h\}\}=1and Hoeffding’s lemma give𝔼πh​Yh≤\(bhspan\)2/8\\mathbb\{E\}\_\{\\pi\_\{h\}\}Y\_\{h\}\\leq\(b\_\{h\}^\{\\rm span\}\)^\{2\}/8andlog⁡𝔼bh​ep​Yh≤p⁡\(p−1\)​\(bhspan\)2/8\\log\\mathbb\{E\}\_\{b\_\{h\}\}e^\{pY\_\{h\}\}\\leq p\(p\-1\)\(b\_\{h\}^\{\\rm span\}\)^\{2\}/8\. Successive conditioning gives𝒞p≤er​p\\mathcal\{C\}\_\{p\}\\leq e^\{rp\}\. This bound applies without the lower\-ratio condition of Lemma[15](https://arxiv.org/html/2610.20814#Thmtheorem15)\. For stored\-policy lag,r≤H​d2​\(1\+γ\)2/\(2​β2\)r\\leq Hd^\{2\}\(1\+\\gamma\)^\{2\}/\(2\\beta^\{2\}\)\. Useer​pe^\{rp\}as an upper bound on𝒞p\\mathcal\{C\}\_\{p\}in Theorem[13](https://arxiv.org/html/2610.20814#Thmtheorem13)\.

## Appendix CExperimental details

### C\.1Data and evaluation

#### Training data\.

We describe the data construction and evaluation protocol here so that the reported comparisons can be reproduced\. ALP\-Math combines the mathematics split of[Guru\-RL](https://huggingface.co/datasets/LLM360/guru-RL-92k)\([Cheng et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib10)\)with an OpenR1\-derived dataset\([2025](https://arxiv.org/html/2610.20814#bib.bib1)\)\.222Dataset revisions: Guru\-RL,2e2790a962a3c099bfb5ea61389cbf98a5ea439b; OpenR1,da327b36b4dc077240aa8efd7f25c4017cd74ef4\.We use the 54,404 problems intrain/math\_\_combined\_54\.4k\.parquetfromLLM360/guru\-RL\-92k\. Fromweqweasdas/from\_default\_filtered\_openr1\_with\_scores, we retain the 23,036 examples whose upstream mean verifier score lies strictly between0\.20\.2and0\.80\.8\. We merge the two sources and deduplicate by exact user\-content string after applying the common prompt format, within and across sources\. For each duplicated prompt, we keep the first occurrence, processing Guru\-RL before OpenR1\. This reduces 77,440 records to 73,293 unique prompts\. Length filtering retains 73,290 prompts: each input has at most 2,048 tokens after applying the chat template, including system and user instructions and the generation prompt\. We independently checked the training data against all five evaluation benchmarks and found no overlapping problems\. The Qwen3\-1\.7B\-Base default\-setting dataloaders shuffle with seeds 42, 1234, and 2027\. TheN=128N=128condition also uses three training seeds; other settings with periodic behavior synchronization use seed 42\.

Training\-time evaluation uses[MATH\-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)from PRM800K\. Final evaluation additionally uses[AIME 2024](https://huggingface.co/datasets/math-ai/aime24),[AIME 2025](https://huggingface.co/datasets/math-ai/aime25),[OlympiadBench](https://huggingface.co/datasets/math-ai/olympiadbench)and[Minerva Math](https://huggingface.co/datasets/math-ai/minervamath)\([Lewkowycz et al\., 2022](https://arxiv.org/html/2610.20814#bib.bib23)\)\. MATH\-500 is the PRM800K subset of MATH\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.20814#bib.bib18);[Lightman et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib25)\), and OlympiadBench is the text\-only mathematics split\([He et al\., 2024](https://arxiv.org/html/2610.20814#bib.bib17)\)\. Ground truths are their boxed answers\. The evaluation sets contain 500 MATH\-500 problems, 30 AIME 2024 problems, 30 AIME 2025 problems, 674 OlympiadBench problems, and 272 Minerva Math problems, for 1,506 problems in total\. Every problem\-weighted average we report uses these dataset sizes as weights\. The accompanying preprocessing code pins these evaluation datasets to fixed revisions\.333MATH\-500:[6e4ed1a](https://huggingface.co/datasets/HuggingFaceH4/MATH-500/tree/6e4ed1a2a79af7d8630a6b768ec859cb5af4d3be); AIME 2024:[83a7f38](https://huggingface.co/datasets/math-ai/aime24/tree/83a7f387baaa524a8bda0022eac0541582297103); AIME 2025:[563bb84](https://huggingface.co/datasets/math-ai/aime25/tree/563bb8404243c5f09de6ec262f2db674fe5bce9b); OlympiadBench:[4faaf1e](https://huggingface.co/datasets/math-ai/olympiadbench/tree/4faaf1e6ec17d11a4218a9bf4c049ecaf954dd84); Minerva Math:[ee46ddc](https://huggingface.co/datasets/math-ai/minervamath/tree/ee46ddc498933b1977577953250ca5c66be64f96)\.

#### Models and prompts\.

We use[Qwen3\-1\.7B\-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base)and[Qwen3\-4B\-Base](https://huggingface.co/Qwen/Qwen3-4B-Base)with their corresponding tokenizers\. The exact system prompt is “Please reason step by step, and put your final answer within\\boxed\{\}\.” Each user problem is followed by “Let’s think step by step and output the final answer within\\boxed\{\}\.”

#### Reward and decoding\.

The binary reward uses Math\-Verify\([Kydlíček, 2025](https://arxiv.org/html/2610.20814#bib.bib22)\)to verify answers against the ground truth\. We use no separate format bonus or length penalty\.

Training rollouts use vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.20814#bib.bib21)\)at sampling temperature 1\.0, top\-p=1\.0p=1\.0, top\-kkdisabled, 2,048\-token prompt and response maxima, andm=8m=8responses per prompt\. Training\-time and final evaluation use the same sampling temperature, top\-pp, top\-kk, and length settings\. Each checkpoint is evaluated once, drawing 32 responses per problem\. The end\-of\-sequence \(EOS\) token is included in the response probability, and the response mask includes generated tokens through the first EOS token while excluding subsequent padding\. If generation reaches the 2,048\-token response maximum without EOS, all generated tokens define the maximum\-length\-truncated prefix; no unobserved EOS factor is included\.

#### Trainer and evaluation policies\.

The bounds useπθ\\pi\_\{\\theta\}, the policy defined by the trainer likelihoods\. Evaluation samples fromπθeval\\pi\_\{\\theta\}^\{\\mathrm\{eval\}\}, the rollout implementation of the same checkpoint; mean@32 estimatesJ⁡\(πθeval\)J\(\\pi\_\{\\theta\}^\{\\mathrm\{eval\}\}\)\. For a common prompt distributionρ\\rhoand reward functionr∈\[0,1\]r\\in\[0,1\],

\|J\(πθeval\)−J\(πθ\)\|≤𝔼X∼ρTV\(πθeval\(⋅∣X\),πθ\(⋅∣X\)\),\\bigl\|J\(\\pi\_\{\\theta\}^\{\\mathrm\{eval\}\}\)\-J\(\\pi\_\{\\theta\}\)\\bigr\|\\leq\\mathbb\{E\}\_\{X\\sim\\rho\}\\operatorname\{TV\}\\\!\\left\(\\pi\_\{\\theta\}^\{\\mathrm\{eval\}\}\(\\cdot\\mid X\),\\pi\_\{\\theta\}\(\\cdot\\mid X\)\\right\),whereTV⁡\(p,q\)=12​∑a\|p⁡\(a\)−q⁡\(a\)\|\\operatorname\{TV\}\(p,q\)=\\tfrac\{1\}\{2\}\\sum\_\{a\}\|p\(a\)\-q\(a\)\|compares distributions over complete responses\. Subtracting this term transfers a learner reward lower bound to the evaluation policy\.

#### Best\-of\-KKevaluation\.

For binary rewards, best@KKand pass@KKboth estimateJK​\(πθeval\)J\_\{K\}\(\\pi\_\{\\theta\}^\{\\mathrm\{eval\}\}\)in Eq\. \([13](https://arxiv.org/html/2610.20814#S4.E13)\)\. Givencccorrect responses out of 32, the unbiased estimator is1−\(32−cK\)/\(32K\)1\-\\binom\{32\-c\}\{K\}/\\binom\{32\}\{K\}for1≤K≤321\\leq K\\leq 32\([Chen et al\., 2021](https://arxiv.org/html/2610.20814#bib.bib9)\), with\(32−cK\)=0\\binom\{32\-c\}\{K\}=0whenK\>32−cK\>32\-c\. It averages the maximum reward over all size\-KKsubsets: atK=1K=1, it equalsc/32c/32\(mean@32\); atK=32K=32, it equals𝟏\{c\>0\}\\mathbf\{1\}\\\{c\>0\\\}\. Reported scores average these values over problems\.

#### Mixed\-reward filtering and refill\.

In the default setting \(N=8N=8\), each behavior collection draws the next 256 prompts from the shuffled dataloader and generatesmmresponses per prompt under one frozen pre\-update policy\. We retain groups containing at least one correct and one incorrect response and continue sampling until the collection is full\. Each collection allows at most 32 generation batches, including the initial batch; training stops with an error if the target batch cannot be filled\. For a prompt with success probabilitypp, the probability that a sampled group is retained is1−pm−\(1−p\)m1\-p^\{m\}\-\(1\-p\)^\{m\}\. The procedure therefore changes the training distribution by favoring prompts that produce mixed\-reward groups\. Groups with equal rewards carry no verifier signal for GRPO, AGRO, or GGPO, although reference regularization can still contribute an update\. Within each matched comparison, methods use the same filtering rule, refill budget, batch geometry, and prompt order\.

#### Mismatch diagnostics\.

K3 and ESS are computed on learner minibatches retained after mixed\-reward filtering\. The behavior–learner mismatch curve reports token\-mean K3\([Schulman, 2020](https://arxiv.org/html/2610.20814#bib.bib38)\),exp⁡\(λτ\)−λτ−1\\exp\(\\lambda\_\{\\tau\}\)\-\\lambda\_\{\\tau\}\-1, forλτ=log⁡πθ​\(aτ∣a<τ\)−log⁡πbeh​\(aτ∣a<τ\)\\lambda\_\{\\tau\}=\\log\\pi\_\{\\theta\}\(a^\{\\tau\}\\mid a^\{<\\tau\}\)\-\\log\\pi\_\{\\mathrm\{beh\}\}\(a^\{\\tau\}\\mid a^\{<\\tau\}\), averaged over the logged response tokens\. For theBrespB\_\{\\mathrm\{resp\}\}responses in a logged batch, the learner\-to\-behavior sequence importance weight iswi=exp⁡\(∑τMi​τ​λi​τ\)w\_\{i\}=\\exp\(\\sum\_\{\\tau\}M\_\{i\\tau\}\\lambda\_\{i\\tau\}\), and the reported normalized effective sample size isESSIS=\(∑iwi\)2/\(Bresp​∑iwi2\)∈\[1/Bresp,1\]\\mathrm\{ESS\}\_\{\\mathrm\{IS\}\}=\(\\sum\_\{i\}w\_\{i\}\)^\{2\}/\(B\_\{\\mathrm\{resp\}\}\\sum\_\{i\}w\_\{i\}^\{2\}\)\\in\[1/B\_\{\\mathrm\{resp\}\},1\]\. These sequence\-level weights provide a mismatch diagnostic for all methods\. For Figure[4](https://arxiv.org/html/2610.20814#S6.F4), drift is the token mean of\|λτ\|\|\\lambda\_\{\\tau\}\|over response tokens selected by the maskMM\. Its ESS uses theBresp=2,048B\_\{\\mathrm\{resp\}\}=2\{,\}048responses of each logged block of eight learner minibatches, so the minimum normalized ESS is1/2,0481/2\{,\}048\. Gradient norms use each method’s native actor\-loss normalization and are therefore interpreted only within a method over training\.

### C\.2Training setup

Training uses AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.20814#bib.bib28)\): learning rate10−610^\{\-6\}, betas\(0\.9,0\.999\)\(0\.9,0\.999\), weight decay0\.010\.01, constant schedule, no warmup, and gradient clip 1\. We use one actor epoch per collection and no entropy bonus\. In the default setting \(N=8N=8\), each 256\-prompt collection containsm=8m=8responses per prompt and is divided into 32\-prompt minibatches, givingN=8N=8optimizer updates\. Unless stated otherwise, every method usesβ=0\.01\\beta=0\.01, and neither AGRO nor GGPO applies a token\-level current\-to\-pre\-update ratio, clipping, learner\-to\-behavior importance weights, or an additive KL term\.

#### Objectives and reference policies\.

GGPO optimizes the loss in Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\)\. In every main comparison, AGRO and GGPO refreshπref←πθ\\pi\_\{\\mathrm\{ref\}\}\\leftarrow\\pi\_\{\\theta\}everyTref=160T\_\{\\mathrm\{ref\}\}=160optimizer updates, because their regularized targets depend on the active reference, whereas our GRPO implementation retains its initial reference\. Appendix[D\.5](https://arxiv.org/html/2610.20814#A4.SS5)compares a fixed reference,Tref=160T\_\{\\mathrm\{ref\}\}=160, andTref=320T\_\{\\mathrm\{ref\}\}=320for all three methods\.

#### Default GRPO\.

GRPO standardizes rewards within each group ofm=8m=8responses by subtracting the group mean and dividing by the sample standard deviation plus10−610^\{\-6\}; the resulting advantages are held fixed during differentiation\. The token\-level PPO ratio\([Schulman et al\., 2017](https://arxiv.org/html/2610.20814#bib.bib39)\)divides the current learner probability by the behavior probability recorded during rollout\. The surrogate clips this ratio to\[0\.8,1\.2\]\[0\.8,1\.2\], applies dual clipping withc=10c=10for negative advantages, and averages over response tokens selected by the mask\. It uses the recorded rollout denominator directly, with no old\-trainer recomputation or additional off\-policy correction\. A K3 KL penalty \(low\_var\_kl\) with coefficient0\.010\.01regularizes the learner toward the fixed initial reference; the entropy coefficient is zero\.

#### GRPO with token\-level truncated importance sampling\.

GRPO\+TIS uses a token\-level importance\-weight cap of 2 in theN=128N=128staleness comparison and offline training\. Letμ\\mube the behavior policy that generated the data andπold\\pi\_\{\\mathrm\{old\}\}the trainer policy at the start of an update block\. AtN=128N=128, each update block consists of 8 optimizer updates\. Offline training uses blocks of 8 optimizer updates, refreshingπold\\pi\_\{\\mathrm\{old\}\}at each block boundary while keepingμ\\muand its stored token log probabilities fixed throughout training\. We compute each block’s old\-trainer token log probabilities once under itsπold\\pi\_\{\\mathrm\{old\}\}\. For tokenai​τa\_\{i\\tau\}with prompt\-and\-response\-prefix contexthi​τh\_\{i\\tau\}, the TIS weight and PPO ratio are

wi​τ\\displaystyle w\_\{i\\tau\}=min⁡\{πold​\(ai​τ∣hi​τ\)μ⁡\(ai​τ∣hi​τ\),2\},\\displaystyle=\\min\\\!\\left\\\{\\frac\{\\pi\_\{\\mathrm\{old\}\}\(a\_\{i\\tau\}\\mid h\_\{i\\tau\}\)\}\{\\mu\(a\_\{i\\tau\}\\mid h\_\{i\\tau\}\)\},\\,2\\right\\\},\(61\)ri​τ​\(θ\)\\displaystyle r\_\{i\\tau\}\(\\theta\)=πθ​\(ai​τ∣hi​τ\)πold​\(ai​τ∣hi​τ\)\.\\displaystyle=\\frac\{\\pi\_\{\\theta\}\(a\_\{i\\tau\}\\mid h\_\{i\\tau\}\)\}\{\\pi\_\{\\mathrm\{old\}\}\(a\_\{i\\tau\}\\mid h\_\{i\\tau\}\)\}\.The TIS weight multiplies the token\-level PPO clipped surrogate\. The old\-trainer log probabilities, TIS weights, and PPO denominator remain fixed within each block; the current\-policy numerator changes with each learner update\. Weights above 2 are capped at 2, without batch normalization or rejection sampling\. All other settings match the GRPO baseline in the corresponding setting\.

#### Default AGRO\.

AGRO\([Tang et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib43)\)uses leave\-one\-out centered adjusted rewards\. For responseiiin groupgg, let

sg,i​\(θ\)=∑τ∈responselog⁡πθ​\(ag,iτ∣xg,ag,i<τ\),zg,i=rg,i−β⁡\(sg,i​\(θ\)−sg,iref\)\.s\_\{g,i\}\(\\theta\)=\\sum\_\{\\tau\\in\\mathrm\{response\}\}\\log\\pi\_\{\\theta\}\(a\_\{g,i\}^\{\\tau\}\\mid x\_\{g\},a\_\{g,i\}^\{<\\tau\}\),\\qquad z\_\{g,i\}=r\_\{g,i\}\-\\beta\\bigl\(s\_\{g,i\}\(\\theta\)\-s\_\{g,i\}^\{\\mathrm\{ref\}\}\\bigr\)\.Withz¯g=m−1​∑izg,i\\bar\{z\}\_\{g\}=m^\{\-1\}\\sum\_\{i\}z\_\{g,i\}, the leave\-one\-out advantage is

ag,iLOO=zg,i−1m−1​∑j≠izg,j=mm−1​\(zg,i−z¯g\)\.a\_\{g,i\}^\{\\mathrm\{LOO\}\}=z\_\{g,i\}\-\\frac\{1\}\{m\-1\}\\sum\_\{j\\neq i\}z\_\{g,j\}=\\frac\{m\}\{m\-1\}\(z\_\{g,i\}\-\\bar\{z\}\_\{g\}\)\.We remove the exteriorβ\\betafactor from AGRO’s gradient and use token\-mean aggregation to reuse GRPO’s training hyperparameters\. For a global learner minibatch containingGGgroups with mean response lengthℓ¯\\bar\{\\ell\}, the optimized surrogate is

LAGRO=−1G​m​ℓ¯∑g=1G∑i=1msg\(ag,iLOO\)sg,i\(θ\),L\_\{\\mathrm\{AGRO\}\}=\-\\frac\{1\}\{Gm\\bar\{\\ell\}\}\\sum\_\{g=1\}^\{G\}\\sum\_\{i=1\}^\{m\}\\operatorname\{sg\}\\\!\\left\(a\_\{g,i\}^\{\\mathrm\{LOO\}\}\\right\)s\_\{g,i\}\(\\theta\),\(62\)whereβ=0\.01\\beta=0\.01remains in the adjusted reward andG​m​ℓ¯Gm\\bar\{\\ell\}is the total number of valid response tokens selected by the mask\. The operatorsg\\operatorname\{sg\}stops gradients through the entire advantage, including its current\-policy log ratio; only the finalsg,i​\(θ\)s\_\{g,i\}\(\\theta\)factor receives gradients\. Rewards, reference log probabilities, and response masks are held fixed\. The surrogate uses no PPO ratio, clipping, additional importance correction, additive KL loss, or entropy bonus\. The reference is refreshed every 160 optimizer updates\.

#### Numerical precision\.

The default GRPO, AGRO, and GGPO runs, as well as theN=128N=128GRPO\+TIS runs, store actor parameters in FP32, execute the forward pass with BF16 mixed precision, and reduce gradients in FP32\. Rollout uses BF16; quantization and quantization\-aware training are disabled\.

#### Implementation and seeds\.

All three objectives are implemented in one fork of verl\([Sheng et al\., 2025](https://arxiv.org/html/2610.20814#bib.bib41)\)\(version 0\.9\.0\.dev\) with FSDP actor sharding and vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.20814#bib.bib21)\)rollout, so they share the data pipeline, verifier, refill logic, and evaluation code; their objectives and reference\-policy handling follow the complete configurations above\. Seeds and checkpoint reporting follow Section[6\.1](https://arxiv.org/html/2610.20814#S6.SS1)\.

### C\.3Experimental protocols

#### Default training and policy staleness\.

The default and controlled policy staleness runs use 8,000 optimizer updates\. Each behavior collection suppliesNNdistinct learner minibatches, with one update per minibatch and behavior synchronization after the collection; Section[6\.4](https://arxiv.org/html/2610.20814#S6.SS4)states the minibatch partition and sample ages\. ForN=128N=128, the final collection is generated in full, and 64 of its minibatches are used to complete the 8,000\-update budget\.

Evaluation and checkpointing occur every 160 optimizer updates in every run, independently ofTrefT\_\{\\mathrm\{ref\}\}\.

#### Offline setting\.

The fixed rollout pool contains only mixed\-reward groups generated once by Qwen3\-1\.7B\-Base, which is also the initial reference policy\. At generation time, we store each response’s behavior token log probabilities and reuse them throughout training\. Current learner token log probabilities are evaluated on the stored responses for the K3, drift, and ESS measurements; GRPO’s learner\-to\-behavior ratios are formed against the stored behavior values\. Methods share the same groups and order without new generation\. For each seed, we prepare 400 fixed learner minibatches and perform one optimizer update on each, with evaluation every 80 updates and no response revisited\. GRPO\+TIS refreshes its old\-trainer policy every 8 updates as specified in Appendix[C\.2](https://arxiv.org/html/2610.20814#A3.SS2.SSS0.Px3)\. AGRO and GGPO useTref=160T\_\{\\mathrm\{ref\}\}=160, while GRPO and GRPO\+TIS keep the initial reference fixed\.

### C\.4GGPO implementation

#### Differentiation semantics\.

For responseii, the learner log likelihood issi=∑τMi​τ​log⁡πθ​\(aiτ∣ai<τ\)s\_\{i\}=\\sum\_\{\\tau\}M\_\{i\\tau\}\\log\\pi\_\{\\theta\}\(a\_\{i\}^\{\\tau\}\\mid a\_\{i\}^\{<\\tau\}\), with the response maskMMselecting the valid generated tokens; the reference log likelihood is formed over the same masked sequence\. Sampled tokens, verifier rewards, reference log likelihoods, and masks are held fixed during the actor update\. We form the adjusted rewards and directly differentiate the log\-mean\-exp loss in Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\) with respect to the learner log probabilities\. A detached copy ofqqis used only to report the resulting coefficients and diagnostics, not as a separately optimized surrogate\. Autograd therefore produces the gradient in Eq\. \([9](https://arxiv.org/html/2610.20814#S4.E9)\) exactly, followed by the external normalization in Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\)\.

#### End\-to\-end GGPO update\.

The update follows Algorithm[1](https://arxiv.org/html/2610.20814#algorithm1)\. Refill continues until there are enough mixed\-reward groups forNNdistinct 32\-prompt learner minibatches; the loss is evaluated with a stable log\-sum\-exp, and each minibatch receives one AdamW update\.

WhenNNdoes not divideTrefT\_\{\\mathrm\{ref\}\}, as atN=64N=64andN=128N=128, a refresh boundary can fall between two minibatches from the same behavior collection\. Algorithm[1](https://arxiv.org/html/2610.20814#algorithm1)refreshes the reference at that exact boundary; the already sampled behavior data remain unchanged, and subsequent learner updates evaluate their minibatches under the active reference\.

#### Weight\-concentration diagnostics\.

With𝒮\\mathcal\{S\}andkkas in Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4), over theN∈\{8,16,32,64\}N\\in\\\{8,16,32,64\\\}diagnostic subset of the staleness ladder, the logged within\-class normalized effective sample size—theESS\\mathrm\{ESS\}ofqqrestricted to𝒮\\mathcal\{S\}, divided bykk, over groups withk≥2k\\geq 2—has medians of0\.770\.77–0\.800\.80, while the whole\-groupESSq=1/∑iqi2\\mathrm\{ESS\}\_\{q\}=1/\\sum\_\{i\}q\_\{i\}^\{2\}has medians of1\.961\.96–2\.282\.28out ofm=8m=8\. The whole\-group concentration reflects separation between reward classes\.

## Appendix DAdditional experiments

### D\.1Learning by problem difficulty

To examine where learning occurs, Figure[5](https://arxiv.org/html/2610.20814#A4.F5)splits MATH\-500 by the base model’s empirical success ratep^0\\hat\{p\}\_\{0\}over 32 responses per problem, into the 410 problems with0<p^0≤0\.750<\\hat\{p\}\_\{0\}\\leq 0\.75and the 63 with no correct response \(p^0=0\\hat\{p\}\_\{0\}=0\), and tracks the three default\-setting runs on each subset\. The remaining 27 problems reach above 95% mean@32 for every method within 640 optimizer updates\. This division concerns the difficulty of the problems under the base model\. Figure[6](https://arxiv.org/html/2610.20814#A4.F6)separately compares the methods’ analytic per\-group coefficients\.

Figure 5:MATH\-500 learning curves by base\-model difficulty on Qwen3\-1\.7B\-Base\.Left: mean@32 on the 410 problems the base model solves part of the time \(0<p^0≤0\.750<\\hat\{p\}\_\{0\}\\leq 0\.75\)\. Right: fraction of the 63 initially unsolved problems \(no correct response among the base model’s 32 samples,p^0=0\\hat\{p\}\_\{0\}=0\) with at least one correct response among 32 samples\. Both panels plot GRPO, AGRO, and GGPO in the default setting with seed 42 against optimizer updates\.Figure 6:Analytic coefficient sum for correct responses in a mixed\-reward group\.Form=8m=8, curves show the signed sumWcorrect=∑i:ri=1ciW\_\{\\mathrm\{correct\}\}=\\sum\_\{i:r\_\{i\}=1\}c\_\{i\}, normalized to its value atk=7k=7for each method\. GRPO uses standardized reward advantages,Wcorrect∝k⁡\(m−k\)W\_\{\\mathrm\{correct\}\}\\propto\\sqrt\{k\(m\-k\)\}; AGRO uses leave\-one\-out advantages with equal learner\-to\-reference log ratios within the group,Wcorrect∝k⁡\(m−k\)/\(m−1\)W\_\{\\mathrm\{correct\}\}\\propto k\(m\-k\)/\(m\-1\); GGPO uses Eq\. \([29](https://arxiv.org/html/2610.20814#A1.E29)\) in the binary\-reward limitβ→0\\beta\\to 0at fixed finite log ratios,Wcorrect=\(m−k\)/m−1W\_\{\\mathrm\{correct\}\}=\(m\-k\)/\\sqrt\{m\-1\}\.
### D\.2Pass@KKevaluation

For each problem, pass@KKis the probability that at least one ofKKindependently sampled responses is correct, estimated per problem from the 32 evaluation samples using the unbiased estimator in Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\. For binary rewards, pass@KKand best@KKcoincide; the estimated pass@1 equals mean@32\. The training curves report MATH\-500 best@32, while Table[3](https://arxiv.org/html/2610.20814#A4.T3)reports problem\-weighted pass@KKover all five benchmarks at each method’s final 8,000\-update checkpoint\.

Table 3:Pass@KKacross five benchmarks on Qwen3\-1\.7B\-Base\.Problem\-weighted pass@KK\(%\) at the final 8,000\-update checkpoint for training seed 42\. The base model is evaluated under the same protocol\. Bold marks the best method at eachKK\.#### Results\.

All three methods raise pass@KKabove the base model at everyKK, and GGPO has the highest value at everyKK\(Table[3](https://arxiv.org/html/2610.20814#A4.T3)\)\. Its margin over the strongest baseline grows from 1\.5 points atK=1K=1to about 4\.5 points atK≥16K\\geq 16, while AGRO’s advantage over GRPO disappears at larger sampling budgets\. GGPO’s advantage also appears in the MATH\-500 best@32 curves and the fraction of initially unsolved problems it solves within 32 samples \(Figures[1](https://arxiv.org/html/2610.20814#S6.F1)and[5](https://arxiv.org/html/2610.20814#A4.F5)\)\. For a rarely solved problem with success probabilityppandK​p≪1Kp\\ll 1, the contribution to pass@KKis approximatelyK​pKp, compared withppto pass@1\. Improvements on such problems can therefore produce larger gains at higher sampling budgets, consistent with the difficulty analysis\.

### D\.3Full results under controlled policy staleness

Table[4](https://arxiv.org/html/2610.20814#A4.T4)gives the per\-benchmark final\-checkpoint results across synchronization delays\.

Table 4:Full evaluation across controlled policy staleness on Qwen3\-1\.7B\-Base\.Final\-checkpoint mean@32 accuracy after 8,000 optimizer updates on five benchmarks \(%\), by synchronization delayNN\. Results atN=8,128N=8,128are three\-seed means;N=8N=8reuses Table[1](https://arxiv.org/html/2610.20814#S6.T1), andN=16,32,64N=16,32,64use seed 42\. The final column gives the problem\-weighted average\. Sample standard deviations across seeds are reported for all columns atN=8,128N=8,128\.Figure 7:Training\-time diagnostics across the staleness ladder on Qwen3\-1\.7B\-Base\.Each panel plots one metric against synchronization delayNN\(equally spaced positions\), averaged over the last 800 optimizer updates of each run: the token\-mean K3 diagnostic of behavior–learner mismatch \(×10−3\\times 10^\{\-3\}\), the normalized ESS of the diagnostic sequence\-level learner\-to\-behavior importance weights \(Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\), policy entropy, and mean response length in tokens \(GRPO orange squares, AGRO green triangles, GGPO blue circles\)\. All diagnostics use seed 42; K3 and ESS are computed on mixed\-reward\-filtered training responses\. K3 and ESS use logarithmic vertical axes\. Corresponding accuracies are in Table[4](https://arxiv.org/html/2610.20814#A4.T4)\.#### Results\.

AtN=128N=128, GRPO\+TIS improves on GRPO across all five benchmarks, while GGPO achieves higher accuracy than GRPO\+TIS on each benchmark \(Table[4](https://arxiv.org/html/2610.20814#A4.T4)\)\. GGPO also retains higher entropy than GRPO at every delay, as in the default setting \(Figure[7](https://arxiv.org/html/2610.20814#A4.F7)\)\.

### D\.4Offline training details

Appendix[C\.3](https://arxiv.org/html/2610.20814#A3.SS3)gives the generation, logging, and refresh protocol\. Table[5](https://arxiv.org/html/2610.20814#A4.T5)reports the mean and sample standard deviation of MATH\-500 accuracy across the three seeds; Figure[4](https://arxiv.org/html/2610.20814#S6.F4)provides the corresponding curves and seed\-42 diagnostics\.

Table 5:Offline MATH\-500 accuracy across three seeds on Qwen3\-1\.7B\-Base\.Mean@32 accuracy \(%\) at each evaluated optimizer update, reported as the mean±\\pmsample standard deviation across training seeds 42, 1234, and 2027\. Update 0 is the base model\.
### D\.5Reference\-refresh cadence and regularization

All three methods regularize toward a reference policy; the main comparisons use the settings of Section[6\.1](https://arxiv.org/html/2610.20814#S6.SS1)\. We compare each method under a fixed reference,Tref=160T\_\{\\mathrm\{ref\}\}=160, andTref=320T\_\{\\mathrm\{ref\}\}=320, and evaluate GGPO atβ=0\.1\\beta=0\.1with a fixed reference andTref=160T\_\{\\mathrm\{ref\}\}=160\. Table[6](https://arxiv.org/html/2610.20814#A4.T6)reports the eleven\-condition grid across all five benchmarks at the final checkpoint\.

Table 6:Reference\-refresh cadence and regularization on Qwen3\-1\.7B\-Base\.Final\-checkpoint mean@32 accuracy after 8,000 optimizer updates on five benchmarks \(%\)\. “Fixed” keeps the initial reference throughout training;TrefT\_\{\\mathrm\{ref\}\}is the refresh cadence in optimizer updates\. Theβ=0\.01\\beta=0\.01rows at each method’s default cadence reuse the three\-seed means from Table[1](https://arxiv.org/html/2610.20814#S6.T1); the remaining conditions use seed 42\. The final column uses the problem\-weighted average of Table[1](https://arxiv.org/html/2610.20814#S6.T1)\. The compact main\-text summary in Table[2](https://arxiv.org/html/2610.20814#S6.T2)retains this seed convention\.Table 7:Learner\-to\-reference log\-ratio and entropy diagnostics on Qwen3\-1\.7B\-Base\.Mean sequence log ratiolog⁡\[πθ/πref\]\\log\[\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{ref\}\}\]over the responses in the training batch, and policy entropy, each averaged over the last 800 optimizer updates of the seed\-42 run\.Figure 8:GGPO training under different reference\-refresh cadences and regularization strengths on Qwen3\-1\.7B\-Base\.MATH\-500 mean@32 accuracy against optimizer updates forβ∈\{0\.01,0\.1\}\\beta\\in\\\{0\.01,0\.1\\\}with a fixed reference or refresh every 160 updates, and forβ=0\.01\\beta=0\.01with refresh every 320 updates\. All curves use seed 42\. Final results across all five benchmarks are reported in Table[6](https://arxiv.org/html/2610.20814#A4.T6)\.#### Results\.

The ablation shows that periodic reference refresh is an empirically important part of GGPO, rather than only a theoretical extension\. Atβ=0\.01\\beta=0\.01, the problem\-weighted average rises from32\.9632\.96with a fixed reference to46\.5846\.58with the defaultTref=160T\_\{\\mathrm\{ref\}\}=160schedule;Tref=320T\_\{\\mathrm\{ref\}\}=320reaches44\.9644\.96\(Table[6](https://arxiv.org/html/2610.20814#A4.T6)\)\. Thus a fixed reference costs GGPO 13\.6 points and leaves it below fixed\-reference GRPO\. AGRO also performs best withTref=160T\_\{\\mathrm\{ref\}\}=160, but its fixed\-reference penalty is smaller at 3\.6 points, while GRPO is essentially unchanged across the three conditions\.

Figure[8](https://arxiv.org/html/2610.20814#A4.F8)shows that fixed\-reference GGPO levels off after its early gains, with a lower plateau atβ=0\.1\\beta=0\.1\. With reference refresh, accuracy continues to improve through 8,000 optimizer updates at both temperatures\.

The reference shapes AGRO’s and GGPO’s training targets, whereas GRPO uses it only in an additive KL penalty \(Appendix[C](https://arxiv.org/html/2610.20814#A3)\)\. This distinction is consistent with GRPO’s small accuracy changes across refresh cadences\. GGPO shows a larger accuracy improvement from reference refresh than AGRO and GRPO in this comparison\. The following diagnostics examine the associated reference drift and changes in response weights\.

#### Reference\-drift diagnostics\.

Table[7](https://arxiv.org/html/2610.20814#A4.T7)provides complementary empirical evidence\. With the common fixed reference, GGPO’s mean sequence log ratio is6\.06\.0nats, compared with AGRO’s19\.219\.2\. The same ordering holds withTref=160T\_\{\\mathrm\{ref\}\}=160\(0\.60\.6versus3\.83\.8\) andTref=320T\_\{\\mathrm\{ref\}\}=320\(0\.70\.7versus6\.66\.6\)\. GGPO’s consistently lower log ratios on the retained training responses are consistent with stronger effective reference anchoring under the implemented updates\. The fixed\-reference comparison is especially informative because both methods use the same initial anchor\. These means are computed on each method’s behavior\-policy samples after mixed\-reward filtering\.

#### Weight concentration among successful responses\.

Figure[9](https://arxiv.org/html/2610.20814#A4.F9)compares the distribution of GGPO’s weights among successful responses with a fixed reference and withTref=160T\_\{\\mathrm\{ref\}\}=160atβ=0\.1\\beta=0\.1\. For groups withk≥2k\\geq 2successful responses, letq~i=qi/∑j∈𝒮qj\\widetilde\{q\}\_\{i\}=q\_\{i\}/\\sum\_\{j\\in\\mathcal\{S\}\}q\_\{j\}fori∈𝒮i\\in\\mathcal\{S\}, where𝒮\\mathcal\{S\}is the successful subset\. The within\-success normalized ESS is\[k​∑i∈𝒮q~i2\]−1∈\[1/k,1\]\[k\\sum\_\{i\\in\\mathcal\{S\}\}\\widetilde\{q\}\_\{i\}^\{2\}\]^\{\-1\}\\in\[1/k,1\]; one denotes uniform weights, and smaller values indicate greater concentration\. Because these responses have equal rewards, Eq\. \([10](https://arxiv.org/html/2610.20814#S4.E10)\) givesq~i∝e−Δi\\widetilde\{q\}\_\{i\}\\propto e^\{\-\\Delta\_\{i\}\}, whereΔi=log⁡\[πθ​\(ai∣x\)/πref​\(ai∣x\)\]\\Delta\_\{i\}=\\log\[\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\\mid x\)\]\. Thus this diagnostic isolates how learner\-to\-reference log\-ratio differences redistribute weight within the successful subset\.

Figure 9:Within\-success weight concentration for GGPO on Qwen3\-1\.7B\-Base\.Normalized ESS of the weights restricted to successful responses, for groups with at least two successes, atβ=0\.1\\beta=0\.1with seed 42\. Blue uses a fixed reference; orange refreshes every 160 optimizer updates\. Higher values indicate more uniform weights among successful responses\.After the initial decline, the refreshed\-reference curve remains above the fixed\-reference curve, although both decrease over training\. The higher ESS with refresh is consistent with limiting accumulated log\-ratio differences among successful responses, as described in Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4.SSS0.Px3)\. This provides a weight\-level view of the reference dependence accompanying the accuracy trends in Figure[8](https://arxiv.org/html/2610.20814#A4.F8)\.

Averaging over the final five recorded refresh windows and the matching fixed\-reference interval gives normalized ESS values of0\.5820\.582with a fixed reference and0\.6900\.690with refresh; the corresponding fractions of groups withk≥2k\\geq 2are0\.7580\.758and0\.8390\.839\. The raw group loss remains nonzero, averaging0\.6190\.619and0\.5310\.531, respectively\. These diagnostics establish the presence of reference\-dependent reweighting during finite\-loss training but do not isolate its contribution to the accuracy gain\.

### D\.6Ablations on GGPO’s group weights

GGPO’s weights areqi=softmaxi⁡\(ri/β−Δi\)q\_\{i\}=\\softmax\_\{i\}\(r\_\{i\}/\\beta\-\\Delta\_\{i\}\), whereΔi=log⁡\[πθ​\(ai∣x\)/πref​\(ai∣x\)\]\\Delta\_\{i\}=\\log\[\\pi\_\{\\theta\}\(a\_\{i\}\\mid x\)/\\pi\_\{\\mathrm\{ref\}\}\(a\_\{i\}\\mid x\)\]\. We vary the reward temperatureβ\\betaand remove the reference\-dependent term−Δi\-\\Delta\_\{i\}under the default Qwen3\-1\.7B\-Base configuration withN=8N=8\.

#### Temperature choice\.

Reducingβ\\betafrom0\.010\.01to0\.0010\.001changes GGPO’s problem\-weighted average from46\.5846\.58to46\.7846\.78, while AGRO decreases from45\.0745\.07to44\.3744\.37\. We useβ=0\.01\\beta=0\.01in the main comparisons\.

#### Removing the reference\-dependent term\.

Under the defaultN=8N=8setting, we isolate the contribution of reference\-dependent weighting by replacingqi=softmaxi⁡\(ri/β−Δi\)q\_\{i\}=\\softmax\_\{i\}\(r\_\{i\}/\\beta\-\\Delta\_\{i\}\)withqi=softmaxi⁡\(ri/β\)q\_\{i\}=\\softmax\_\{i\}\(r\_\{i\}/\\beta\)atβ=0\.01\\beta=0\.01, while retaining the remaining GGPO update\. The reward\-only variant scores45\.1245\.12with seed 42, compared with the full method’s three\-seed mean of46\.5846\.58\. For mixed binary\-reward groups, the resulting update coefficients approach the success\-normalized form of MaxRL\([Tajwar et al\., 2026](https://arxiv.org/html/2610.20814#bib.bib42)\)at smallβ\\beta, up to GGPO’s normalization \(Appendix[A\.4](https://arxiv.org/html/2610.20814#A1.SS4)\)\. This component ablation supports reference\-dependent weighting in the defaultN=8N=8setting\.

### D\.7Finite group size

We study the effect of group size at a fixed optimizer\-update budget by varying the number of responses per prompt overm∈\{4,8,16\}m\\in\\\{4,8,16\\\}for Qwen3\-1\.7B\-Base under the default GGPO configuration\. Every run uses 32 groups per learner minibatch and 8,000 optimizer updates; them=8m=8condition reuses the default\-setting run\. Each update uses32​m32mretained responses, and raw generation cost also depends on mixed\-group acceptance and refill\.

Table 8:Evaluation across GGPO group sizes on Qwen3\-1\.7B\-Base\.Problem\-weighted best@KK\(%\) over the five benchmarks at the final 8,000\-update checkpoint for training seed 42\. For binary rewards, best@KKequals pass@KK\. Them=8m=8row reuses the GGPO results in Table[3](https://arxiv.org/html/2610.20814#A4.T3)\. All runs use 32 groups per learner minibatch\.#### Results\.

Increasingmmfrom 4 to 16 improves best@KKat every reportedKK, with gains growing from 0\.37 points atK=1K=1to 1\.93 points atK=32K=32\(Table[8](https://arxiv.org/html/2610.20814#A4.T8)\)\. Larger groups are more likely to contain a correct response on difficult prompts, providing a mixed\-reward training signal that smaller groups may miss\. The larger gains at higher sampling budgets are consistent with improvements on problems that the learner solves infrequently\.

### D\.8Sampling budget

Figure[2](https://arxiv.org/html/2610.20814#S6.F2)complements the update\-based comparison in Section[6\.2](https://arxiv.org/html/2610.20814#S6.SS2)with cumulative generated training responses for Qwen3\-1\.7B\-Base under the default setting \(N=8N=8,m=8m=8\)\. Counts include all generated candidate responses, including those discarded by mixed\-reward filtering and those generated during refill \(Appendix[C\.1](https://arxiv.org/html/2610.20814#A3.SS1)\)\.

### D\.9Token\-clipping training rewards

Figure[10](https://arxiv.org/html/2610.20814#A4.F10)gives the single\-seed training\-reward comparison discussed in Section[6\.6](https://arxiv.org/html/2610.20814#S6.SS6)\. The rewards are measured on mixed\-reward\-filtered training responses; Appendix[E](https://arxiv.org/html/2610.20814#A5)gives the implementation\.

Figure 10:Token clipping improves early training progress\.Mean training reward \(higher is better\) on Qwen3\-1\.7B\-Base against optimizer updates, withN=8N=8,m=8m=8,β=0\.01\\beta=0\.01,Tref=160T\_\{\\mathrm\{ref\}\}=160, and seed 42\. Blue shows the original GGPO update; orange shows the detached\-coefficient variant with token log\-ratio clipping at±0\.05\\pm 0\.05\. Rewards are measured on mixed\-reward\-filtered training data\. Faint lines show values aggregated over eight updates; bold lines show trailing 160\-update means\. Clipping raises early training reward, with a smaller difference later in training\.

## Appendix EToken\-clipping implementation

This appendix explains how the clipped adjusted rewards determine response coefficients and how those coefficients enter the detached\-gradient surrogate used in Section[6\.6](https://arxiv.org/html/2610.20814#S6.SS6)\. The original GGPO algorithm uses no token log\-ratio clipping\.

#### Clipped reference correction\.

For responseaiga\_\{i\}^\{g\}in groupgg, lethg,i,t=\(xg,ai,<tg\)h\_\{g,i,t\}=\(x\_\{g\},a^\{g\}\_\{i,<t\}\)and letMg,i,tM\_\{g,i,t\}be the valid\-token mask\. We clip each learner\-to\-reference token log ratio before summing it:

dg,i,t​\(θ\)\\displaystyle d\_\{g,i,t\}\(\\theta\)=log⁡πθ​\(ai,tg∣hg,i,t\)πref​\(ai,tg∣hg,i,t\),\\displaystyle=\\log\\frac\{\\pi\_\{\\theta\}\(a^\{g\}\_\{i,t\}\\mid h\_\{g,i,t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(a^\{g\}\_\{i,t\}\\mid h\_\{g,i,t\}\)\},R¯g,i​\(θ\)\\displaystyle\\bar\{R\}\_\{g,i\}\(\\theta\)=r⁡\(xg,aig\)−β​∑tMg,i,t​clip⁡\(dg,i,t​\(θ\),−cclip,cclip\),\\displaystyle=r\(x\_\{g\},a\_\{i\}^\{g\}\)\-\\beta\\sum\_\{t\}M\_\{g,i,t\}\\operatorname\{clip\}\(d\_\{g,i,t\}\(\\theta\),\-c\_\{\\mathrm\{clip\}\},c\_\{\\mathrm\{clip\}\}\),\(63\)q¯g,i​\(θ\)\\displaystyle\\bar\{q\}\_\{g,i\}\(\\theta\)=exp⁡\(R¯g,i​\(θ\)/β\)∑jexp⁡\(R¯g,j​\(θ\)/β\),c¯g,i​\(θ\)=m​q¯g,i​\(θ\)−1m−1\.\\displaystyle=\\frac\{\\exp\(\\bar\{R\}\_\{g,i\}\(\\theta\)/\\beta\)\}\{\\sum\_\{j\}\\exp\(\\bar\{R\}\_\{g,j\}\(\\theta\)/\\beta\)\},\\qquad\\bar\{c\}\_\{g,i\}\(\\theta\)=\\frac\{m\\bar\{q\}\_\{g,i\}\(\\theta\)\-1\}\{\\sqrt\{m\-1\}\}\.\(64\)We usecclip=0\.05c\_\{\\mathrm\{clip\}\}=0\.05, equivalent to clipping the token probability ratio to\[e−0\.05,e0\.05\]\[e^\{\-0\.05\},e^\{0\.05\}\]\. The denominator is the reference policy; this is not a learner\-to\-behavior importance weight\.

#### Objective and stop\-gradient\.

LetTtok=∑g,i,tMg,i,tT\_\{\\mathrm\{tok\}\}=\\sum\_\{g,i,t\}M\_\{g,i,t\}be the total valid\-token count in the learner minibatch, and letθ0\\theta\_\{0\}denote the parameters at which the response coefficients are computed for the current update\. We minimize the local surrogate

Lclip\(θ;θ0\)=−1Ttok∑g,isg\[c¯g,i\(θ0\)\]∑tMg,i,tlogπθ\(ai,tg∣hg,i,t\),L\_\{\\mathrm\{clip\}\}\(\\theta;\\theta\_\{0\}\)=\-\\frac\{1\}\{T\_\{\\mathrm\{tok\}\}\}\\sum\_\{g,i\}\\operatorname\{sg\}\[\\bar\{c\}\_\{g,i\}\(\\theta\_\{0\}\)\]\\sum\_\{t\}M\_\{g,i,t\}\\log\\pi\_\{\\theta\}\(a^\{g\}\_\{i,t\}\\mid h\_\{g,i,t\}\),\(65\)wheresg\\operatorname\{sg\}holds its entire argument constant during differentiation\. Clipping changes the coefficients, while differentiation uses the original, unclipped token log probabilities\. Tokens outside the clipping interval still contribute their score gradients, and all valid tokens remain inTtokT\_\{\\mathrm\{tok\}\}\. Coefficients are recomputed for each update; rewards, sampled tokens, reference probabilities, and masks are held fixed\. The normalization matches Eq\. \([11](https://arxiv.org/html/2610.20814#S4.E11)\), with no additional exterior factor ofβ\\beta\.

For logging, the implementation retains the clipped log\-mean\-exp statistic

𝒥^m,cclipg=β​log⁡\(1m​∑ieR¯g,i/β\)−1m​∑iR¯g,i\.\\widehat\{\\mathcal\{J\}\}\_\{m,c\_\{\\mathrm\{clip\}\}\}^\{\\,g\}=\\beta\\log\\\!\\left\(\\frac\{1\}\{m\}\\sum\_\{i\}e^\{\\bar\{R\}\_\{g,i\}/\\beta\}\\right\)\-\\frac\{1\}\{m\}\\sum\_\{i\}\\bar\{R\}\_\{g,i\}\.\(66\)The update is the gradient of Eq\. \([65](https://arxiv.org/html/2610.20814#A5.E65)\) atθ=θ0\\theta=\\theta\_\{0\}, rather than ordinary differentiation through Eq\. \([66](https://arxiv.org/html/2610.20814#A5.E66)\)\. The latter would introduce a clamp derivative that suppresses gradients at saturated tokens\. Removing clipping recovers the normalized original GGPO gradient on the same minibatch\.